---
title: 'Test Scenario Modeling'
source: 'https://academia.sh/en/courses/non-functional-testing/test-scenario-modeling'
course: 'Non-Functional Testing'
language: en
updated: '2026-08-23T14:25:16+00:00'
license: 'CC BY-SA 4.0'
---

# Test Scenario Modeling

The shape of the load, not its size: measuring at what share of the workload mix a defect becomes visible, a search-heavy unrealistic scenario producing a false pass at the same target rate, a closed-loop generator never reaching the target rate because of its virtual-user count and think time, and the same nominal rate giving three separate p95 values.

Every run in the previous lesson sent requests to a single endpoint, at a fixed rate,
without a pause. A real circulation-desk day is not like that: the clerk searches first,
then checks a book out, and pauses between transactions. This lesson takes on the
**shape** of the load rather than its **size**, and measures this: the same target rate
produces a different decision on the same system once the scenario changes.

## The Scenario's Two Parameters

**Workload mix** is how the total load is distributed across request types. A system's
endpoints do not cost the same; the mix determines which path the load falls on, and so
which defect class comes close to saturation.

**Think time** is the pause a virtual user takes between two transactions, and it only has
meaning in a closed-loop generator. In a closed loop there are $N$ virtual users; each
waits for the response, pauses for $Z$, then sends the next request; the rate reached is
$N / (Z + R)$. Here $R$ is the system's own response time — in other words, **the load is a
function of the system under test**. An open-loop generator carries no such feedback.

## Threshold and Assumptions

The threshold carries over from the previous lesson unchanged. **NF1 (assumption +
calculation) — the circulation-desk flow's 60 ms server-side budget split evenly across
three services: p95 ≤ 20 ms.** **NF2 (assumption) — the peak-hour target is 1,000 req/s.**
This lesson writes two more assumptions.

**NF3 (assumption) — the circulation desk's peak-hour workload mix: half the requests are
loans, half are searches.** Rationale: the person who comes to the desk comes to check
a book out; catalog searches also happen through other endpoints but never exceed half of
desk traffic. **NF4 (assumption) — in the closed-loop model, the virtual-user count is
derived from the target rate: $N = 1000 \cdot (Z + R_0)$, $R_0 \approx 2$ ms.** Two pairs
are tested: 22 users / 20 ms and 202 users / 200 ms. Both have a nominal rate of
1,000 req/s.

The decision's owner in this lesson is not the release, it is the test itself: the choice
of scenario determines which defect class gets measured, and a poorly chosen scenario
reports green.

## Harness

The service has two endpoints. `/search` looks a book up by number in the catalog and is
indexed in both structures; `/loan` validates the member's open loan count and
writes five records, and the index exists only in the sound structure. The catalog starts
with 110,000 records, so a full scan is six times more expensive than in the previous
lesson.

```js
// library/server.mjs — circulation-desk service: /search looks up a book, /loan
// validates the member's open loan count and writes five records. Arguments:
// <port> <structure 0|1> — 0 = no member index on the loan path (full scan), 1 = indexed.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 110000, BOOK = 5;  // 220 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_book ON loan(book_no)");      // the search path is indexed in both structures
  if (structure === "1") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const search = db.prepare("SELECT id, member_no, day FROM loan WHERE book_no = ? LIMIT 5");
  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let searches = 0, loans = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ searches, loans, records, scan })); return; }
    if (u.pathname === "/search") {
      searches += 1;
      res.end(JSON.stringify({ found: search.all(u.searchParams.get("book") ?? "K-1").length }));
      return;
    }
    loans += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(loans + j) % 9000}`, loans % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}
```

The generator carries both loop styles. A scenario is just three numbers: the loan
share, the virtual-user count, and the think time.

```js
// library/scenario.mjs — open- and closed-loop generator; the scenario sets the request
// mix and the think time. Arguments: <port> <structure 0|1> <scenario>
import { Agent, request } from "node:http";

const TARGET = 1000, THRESHOLD = 20, DURATION = 3;  // NF2: target rate, NF1: p95 threshold (ms), run length
const SCENARIO = {                          // share: loan-request ratio, n/z: closed loop
  easy: { share: 0.10 }, "share-25": { share: 0.25 }, realistic: { share: 0.50 }, "share-100": { share: 1.00 },
  "closed-22": { share: 0.50, n: 22, z: 20 }, "closed-202": { share: 0.50, n: 202, z: 200 },
};
const [port, structure, name] = [Number(process.argv[2]), process.argv[3], process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const percentile = (d, p) => { const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; };
let seq = 0;

function oneRequest(samples, share) {       // a `share` fraction of requests goes to the loan path
  const i = seq++;
  const path = (i * 9973) % 1000 < share * 1000 ? `/loan?member=U-${i % 500}` : `/search?book=K-${i % 9000}`;
  const start = performance.now();
  return new Promise((done) => {
    const r = request({ port, path, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push(performance.now() - start); done(); });
    });
    r.on("error", () => { samples.push(performance.now() - start); done(); });
    r.end();
  });
}

async function open(duration, rate, share) {   // sends at the target rate, does not wait for the response
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples, share)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

async function closed(duration, n, z, share) {   // n virtual users; each pauses z ms after a request
  const samples = [];
  const start = performance.now();
  const worker = async () => {
    while (performance.now() - start < duration * 1000) {
      await oneRequest(samples, share);
      await new Promise((c) => setTimeout(c, z));
    }
  };
  await Promise.all(Array.from({ length: n }, worker));
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

const sc = SCENARIO[name];
if (Number.isInteger(port) === false || sc === undefined) {
  console.log(`usage: node library/scenario.mjs <port> <structure 0|1> ${Object.keys(SCENARIO).join("|")}`);
} else {
  await oneRequest([], 1);                 // warm-up request
  const r = sc.n === undefined ? await open(DURATION, TARGET, sc.share) : await closed(DURATION, sc.n, sc.z, sc.share);
  const o = await metrics();
  console.log(JSON.stringify({ name, structure: structure === "0" ? "unindexed" : "indexed", share: sc.share,
    loop: sc.n === undefined ? "open" : `closed N=${sc.n} Z=${sc.z}`, p95: percentile(r.samples, 95),
    rate: r.samples.length / r.elapsed, loans: o.loans, scan: o.scan }));
}
agent.destroy();
```

```js
// library/report.mjs — turns ten runs into a table and scans the p95 threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, SCENARIO = ["easy", "realistic", "closed-22", "closed-202"];
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, a) => s.find((r) => r.structure === y && r.name === a);
const fmt2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const verdict = (x) => (x <= THRESHOLD ? "pass" : "fail");

console.log(`threshold p95 <= ${THRESHOLD} ms (NF1), target rate 1000 req/s (NF2), run 3 s`);
console.log("scenario     loop               loan share  structure   reached/s  p95(ms)  verdict");
for (const a of SCENARIO) for (const y of ["unindexed", "indexed"]) {
  const r = find(y, a);
  console.log(a.padEnd(13) + r.loop.padEnd(19) + pad(fmt2(r.share), 10) +
    "  " + y.padEnd(11) + pad(fmt2(r.rate), 10) + pad(fmt2(r.p95), 9) + pad(verdict(r.p95), 9));
}

console.log("\nloan-share scan (unindexed structure, open loop, same 1000 req/s target)");
console.log("loan share      loan req/s  p95(ms)  verdict   scan steps");
for (const a of ["easy", "share-25", "realistic", "share-100"]) {
  const r = find("unindexed", a);
  console.log(pad(fmt2(r.share), 10) + pad(fmt2(r.loans / 3), 16) + pad(fmt2(r.p95), 9) +
    pad(verdict(r.p95), 9) + pad(r.scan.toLocaleString("en-US"), 13));
}

console.log("\nthreshold scan: how many of the four scenarios give a wrong verdict");
console.log("p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones");
for (const e of [2, 5, 15, 20, 30, 100]) {
  const pass = SCENARIO.filter((a) => find("unindexed", a).p95 <= e);
  const fail = SCENARIO.filter((a) => find("indexed", a).p95 > e);
  const dist = SCENARIO.filter((a) => find("indexed", a).p95 <= e && find("unindexed", a).p95 > e);
  console.log(String(e).padEnd(20) + pad(pass.length, 14) + pad(fail.length, 15) + pad(dist.length, 27) +
    "  " + (dist.join(", ") || "-"));
}
```

```bash
# measure.sh — ten runs: four scenarios x two structures, plus two extra loan shares.
# Each run works with a freshly started server; the catalog always starts with 110,000 records.
rm -f result.jsonl
for T in 0 1; do
  for SC in easy realistic closed-22 closed-202; do
    node library/server.mjs 8852 $T & SERVER=$!
    sleep 1.2
    node library/scenario.mjs 8852 $T $SC >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
for SC in share-25 share-100; do
  node library/server.mjs 8852 0 & SERVER=$!
  sleep 1.2
  node library/scenario.mjs 8852 0 $SC >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/report.mjs result.jsonl
```

```
threshold p95 <= 20 ms (NF1), target rate 1000 req/s (NF2), run 3 s
scenario     loop               loan share  structure   reached/s  p95(ms)  verdict
easy         open                     0.10  unindexed      997.48     8.21     pass
easy         open                     0.10  indexed        997.52     1.69     pass
realistic    open                     0.50  unindexed      916.07   760.33     fail
realistic    open                     0.50  indexed        997.51     1.83     pass
closed-22    closed N=22 Z=20         0.50  unindexed      692.28    29.06     fail
closed-22    closed N=22 Z=20         0.50  indexed        970.94     2.55     pass
closed-202   closed N=202 Z=200       0.50  unindexed      855.57    77.97     fail
closed-202   closed N=202 Z=200       0.50  indexed        985.30    11.75     pass

loan-share scan (unindexed structure, open loop, same 1000 req/s target)
loan share      loan req/s  p95(ms)  verdict   scan steps
      0.10           99.67     8.21     pass   33,112,755
      0.25          248.67    13.61     pass   83,449,425
      0.50          500.00   760.33     fail  170,621,250
      1.00          998.67  6106.87     fail  351,992,550

threshold scan: how many of the four scenarios give a wrong verdict
p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones
2                                0              2                          2  easy, realistic
5                                0              1                          3  easy, realistic, closed-22
15                               1              0                          3  realistic, closed-22, closed-202
20                               1              0                          3  realistic, closed-22, closed-202
30                               2              0                          2  realistic, closed-202
100                              3              0                          1  realistic
```

## The Mix Changes the Decision

The target rate is 1,000 req/s in all four runs. The only thing that changes is how that
rate is distributed across the endpoints. At a loan share of 0.10 the flawed structure
stays at 8.21 ms and **passes**; at 0.25 it is still passing at 13.61 ms; at 0.50 it jumps
to 760.33 ms. The difference is not in the system, it is in the scenario.

The reason is in the second column: the loan request rate climbs from 99.67 to 500.00.
In the flawed structure this endpoint does a full scan and its own ceiling is around
400 req/s; the share is under that ceiling at 0.25 and over it at 0.50. The scan step shows
this independently of the run: from 33,112,755 to 170,621,250, **about 5.2 times**. The
name of the defect class this catches is **saturation of a single path**: while the system
as a whole meets the target rate, one endpoint hits its ceiling and corrupts the latency of
every request.

The easy scenario is where this escapes. A search request is the cheapest one to write, so
scenarios tend to skew toward search-heavy. If NF3 writes the realistic mix as 0.50, a test
run at 0.10 produces a **false pass**, and the name of the class it misses is already
known.

## The Closed Loop Throttles Its Own Load

The bottom two scenarios run the same mix (0.50) through the closed loop, and both have a
nominal rate of 1,000 req/s. In the flawed structure the rate reached is 692.28 and
855.57 req/s — that is, 69.2% and 85.6% of the target. **The generator throttled its own
load because the system under test slowed down.** The load NF2 calls for was never
applied; the p95 measured is the p95 of a different rate.

The result is an understatement. The open loop reported 760.33 ms at the same mix; the
closed loop reported 29.06 and 77.97. Same system, same defect, a number **about 26 times
and 10 times smaller**. In this run all three still cross the 20 ms threshold so the
decision stays the same; if the threshold were 30 ms, closed-22 would pass while the open
loop would still fail.

The sound-structure column shows think time's own effect separately. Even with no defect
at all, the same nominal rate gives three separate p95 values: 1.83 ms in the open loop,
2.55 ms with 22 users / 20 ms, 11.75 ms with 202 users / 200 ms — **a spread of about
6.4 times**. The number belongs to the generator's parameters, not the system.

## Threshold Scan and Cost

The scan gives a two-sided result. At NF1's 20 ms, false pass is 1 (the easy scenario),
false fail is 0 — and this still holds at 15 ms in this run. Tightening further to 5 ms
introduces one false fail — the sound structure's closed-202 scenario, whose own
think-time noise alone crosses 5 ms — and tightening to 2 ms doubles that to two, adding
closed-22. At 100 ms false pass climbs to three; only the open-loop realistic scenario
still distinguishes.

The class it misses can also be counted: since the two closed-loop scenarios never reach
the rate NF2 calls for, **the saturation that appears at the target rate** was never
measured in them; in the flawed structure the missing load is about 307 and 144 req/s. The
run-dependent side of the cost depends on the environment — ten runs finished in under a
minute on this machine. The run-independent side: every open-loop run sends 3,000
requests and the loan-request count comes directly from the share (300, 750, 1,500,
3,000); the scan step in the unindexed structure ranges from 33,112,755 to 351,992,550,
**about 10.7 times**.

## Summary

- The scenario has two parameters: workload mix decides which endpoint the load falls on,
  think time decides the rate reached in a closed loop.
- The same 1,000 req/s target passed at a loan share of 0.10 (8.21 ms), passed at 0.25
  (13.61 ms), and failed at 0.50 (760.33 ms) in this run; the defect became visible through
  the scenario, not the system.
- The class it catches is saturation of a single path: as the loan request rate climbs
  from about 100 to 500 req/s, the endpoint crosses its own ceiling and the scan step rises
  about 5.2 times.
- The closed loop never reached the target rate (about 69% and 86% of it) and understated
  the degradation by roughly 26 and 10 times; the p95 measured is not the p95 of the
  requested load.
- Even in the sound structure, the generator's own parameters carry p95 from 1.83 to
  11.75 ms; tightening the threshold to 5 ms is where this noise starts producing a false
  fail.

## Next Step

Up to this point every decision was made by looking at a single number: p95. That choice
was never discussed. If the average had been taken, the 760.33-millisecond run would look
completely different; if p99 had been chosen, the decision would change again, and how many
samples a three-second run's p99 rests on was never asked. That throughput and latency need
to be read together was only implied. The next lesson takes on metric choice: what an
average hides, how many samples a percentile needs to be meaningful, and why latency read
apart from throughput is misleading.