---
title: 'Non-Functional Testing'
source: 'https://academia.sh/en/courses/non-functional-testing'
language: en
updated: '2026-08-23T14:25:15+00:00'
license: 'CC BY-SA 4.0'
lessons: 15
outcomes:
    - 'Designs load, soak, and spike tests according to their goal'
    - 'Separates the security testing types by their scope'
    - 'Settles accessibility and compatibility testing into the process'
---

# Non-Functional Testing

Testing the questions that lie outside functional correctness: separating load, stress, soak, and spike tests by their goal, modeling realistic user behavior, choosing percentile latency and throughput as the metrics, attributing the result across the application, database, and network layers, the discipline that avoids a misleading measurement, separating the scopes of static and dynamic security testing and dependency scanning, placing penetration testing where automation ends, settling accessibility and compatibility testing into the process, and verifying recovery by triggering failure scenarios deliberately.

## Performance Testing

Testing behavior under load: separating load, stress, soak and spike testing by the question each one asks, modeling realistic user behavior with think time and workload-mix ratios, choosing percentile latency and throughput metrics that reveal what an average hides, attributing a single result to the application, database and network layers, and a benchmarking discipline that avoids the traps of warm-up, noise and single-run readings.

### Performance Testing Types

Source: https://academia.sh/en/courses/non-functional-testing/performance-testing-types

Splitting behavior under load into four separate questions: load, stress, soak and spike testing reading the same metric in a different decision window, all four reaching four separate decisions on the same two structures, deriving a threshold from a budget split, and counting which type produces a false pass at which threshold range through a threshold scan.

The closing of the Integration, Contract and End-to-End Testing course left three questions
open: how the system behaves under load, what it yields under attack, and how it degrades
under failure were never tested in any lesson there — every test in that course asked
**is the function correct**. This course starts with the first of those and asks the first
decision: which test runs. Testing under load takes four forms, and what separates them is
not duration or severity but **the question each one asks**.

#### Four Questions, Four Decision Windows

**Load testing** asks whether the threshold is met at the targeted load, **soak testing**
asks whether the same load produces degradation over a long duration, **spike testing**
asks how long recovery takes after a surge. All three were set up and measured in the
Performance Anti-Patterns and Monitoring course, where the result fed into a capacity
decision. Here the test itself is designed, and a fourth type is added: **stress testing**
raises the load until the threshold breaks and looks for the **breaking point**.

All four read a single metric — the p95 latency of the loan request — but each reads
it in a different **decision window**: load testing reads the whole run, stress testing
reads the step at twice the target rate, soak testing reads the last quarter, spike testing
reads the first second after the surge.

#### Threshold and the Threshold's Source

A performance test's output is a distribution; saying "passed" means choosing a threshold.
This course's rule: a threshold is written with its source.

**NF1 (assumption + calculation) — the circulation-desk flow's server-side budget is
60 ms; the flow makes three service calls and the budget is split evenly. The loan
service's share: p95 ≤ 20 ms.** The budget is an assumption, the share is a calculation;
the threshold was never read off a measurement.
**NF2 (assumption) — the circulation-desk transaction rate at peak hour is 1,000 req/s.**

The decision's owner is written too: a red result stops the release; the breaking point
that stress testing produces does not stop it — it feeds the capacity plan.

#### Harness

The system under test is the library's loan service: the member's open loan count
is validated, then records are written. A circulation-desk transaction gives a member at
most five books — the loan limit carried over from the Unit Testing and Test-Driven
Development course — and each book adds one row. Three structures run: an unindexed
catalog, an indexed catalog, and a third structure that is indexed but computes the
loan count incorrectly.

```js
// library/server.mjs — loan service: validates the member's open loan count in a
// node:sqlite catalog and writes new records. Arguments: <port> <structure 0|1|2>
// 0 = unindexed (full scan), 1 = indexed, 2 = indexed but the count is wrong.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 18000, BOOK = 5;  // 36 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1", "2"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1|2>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  if (structure !== "0") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const count = db.prepare(structure === "2"
    ? "SELECT COUNT(*) AS n FROM loan WHERE member_no = ? AND day = 0"   // flawed count
    : "SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let requests = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ requests, records, scan })); return; }
    requests += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(requests + j) % 9000}`, requests % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}
```

The load generator is **open-loop**: it sends requests at the target rate and does not wait
for the server to keep up. A **test oracle** also runs during the load-test run — the source
of the expected result, in the term from the Quality and Testing Fundamentals course. Its
rule: since every request writes five records for the member, the returned loan count
must be greater than the previous response's.

```js
// library/types.mjs — open-loop generator and the four test types; all four read the same
// metric (p95) in a different window. Arguments: <port> <structure 0|1|2> <type>
import { Agent, request } from "node:http";

const TARGET = 1000, THRESHOLD = 20;    // NF2: target rate (req/s), NF1: p95 threshold (ms)
const STEP = [0.5, 1, 2, 3];            // stress ladder: multiples of the target rate, one step per second
const STRUCTURE = { 0: "unindexed", 1: "indexed", 2: "fast-wrong" };
const [port, structure, type] = [Number(process.argv[2]), process.argv[3], process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const last = new Map();
let broken = 0, checks = 0;

const percentile = (d, p) => { const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; };
const window = (k, a, b) => k.filter((r) => r.sec >= a && r.sec < b).map((r) => r.ms);
const fmt1 = (x) => x.toFixed(1);
const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

function oneRequest(i, samples, sec, oracle) {
  const start = performance.now(), member = `U-${i % 500}`;
  return new Promise((done) => {
    const r = request({ port, path: `/loan?member=${member}&book=K-${i % 900}`, agent }, (y) => {
      let g = ""; y.on("data", (p) => { g += p; });
      y.on("end", () => {
        if (oracle) {                   // every request writes five records for the member: count must rise
          const n = JSON.parse(g).loan;
          if (last.has(member)) { checks += 1; if (n <= last.get(member)) broken += 1; }
          last.set(member, n);
        }
        samples.push({ ms: performance.now() - start, sec }); done();
      });
    });
    r.on("error", () => { samples.push({ ms: performance.now() - start, sec, error: 1 }); done(); });
    r.end();
  });
}

async function run(duration, rate, oracle = false) {   // rate(t): target rate at second t
  const samples = [], inFlight = [];
  let debt = 0, i = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate(t) * (t - lastT);     // scaled by real elapsed time: absorbs timer drift
    lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(i++, samples, t, oracle)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

if (Number.isInteger(port) === false || ["load", "stress", "soak", "spike"].includes(type) === false) {
  console.log("usage: node library/types.mjs <port> <structure 0|1|2> load|stress|soak|spike");
} else {
  await oneRequest(0, [], 0, false);                // warm-up request
  let r, decision, extra;
  if (type === "load") {
    r = await run(2, () => TARGET, true);           // the oracle is meaningful only in an unsaturated run
    decision = percentile(r.samples.map((k) => k.ms), 95);
    extra = `oracle ${broken}/${checks} broken`;
  } else if (type === "stress") {
    r = await run(4, (t) => TARGET * STEP[Math.min(3, Math.floor(t))]);
    const p = STEP.map((_, i) => percentile(window(r.samples, i, i + 1), 95));
    const k = p.findIndex((x) => x > THRESHOLD);
    decision = p[2];                                // p95 of the 2x step
    extra = `break ${k < 0 ? "> 3000" : TARGET * STEP[k]} req/s`;
  } else if (type === "soak") {
    r = await run(10, () => TARGET);
    const c = [0, 1, 2, 3].map((i) => percentile(window(r.samples, i * 2.5, i * 2.5 + 2.5), 95));
    decision = c[3];                                // p95 of the last quarter
    extra = `quarter p95 ${c.map(fmt1).join(" / ")}`;
  } else {
    r = await run(5, (t) => (t >= 1.5 && t < 3 ? TARGET * 3 : TARGET));
    decision = percentile(window(r.samples, 3, 4), 95);   // first second after the spike
    const d = [3, 4].find((sec) => percentile(window(r.samples, sec, sec + 1), 95) <= THRESHOLD);
    extra = `recovery ${d === undefined ? "2.0+" : fmt1(d - 2)} s`;
  }
  const o = await metrics();
  console.log(JSON.stringify({ structure: STRUCTURE[structure], type, decision, extra,
    rate: r.samples.length / r.elapsed, scan: o.scan }));
}
agent.destroy();
```

The report reads the nine runs; the scan moves the threshold away from the 20 milliseconds
NF1 gives and counts, at each value, how many types make the wrong decision.

```js
// library/report.mjs — turns nine runs into a table and scans the p95 threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, TYPE = ["load", "stress", "soak", "spike"];
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, t) => s.find((r) => r.structure === y && r.type === t);
const fmt2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const verdict = (x) => (x <= THRESHOLD ? "pass" : "fail");

console.log(`threshold p95 <= ${THRESHOLD} ms (NF1), target rate 1000 req/s (NF2)`);
console.log("type    structure     reached/s  decision p95  verdict  scan steps    type-specific figure");
for (const t of TYPE) for (const y of ["unindexed", "indexed", "fast-wrong"]) {
  const r = find(y, t);
  if (r !== undefined) console.log(t.padEnd(8) + y.padEnd(14) + pad(fmt2(r.rate), 9) +
    pad(fmt2(r.decision), 14) + pad(verdict(r.decision), 9) + pad(r.scan.toLocaleString("en-US"), 13) + "  " + r.extra);
}

console.log("\nthreshold scan: how many of the four types give a wrong verdict (unindexed flawed, indexed sound)");
console.log("p95 threshold (ms)  false pass /4  false fail /4  distinguishing type /4  distinguishing types");
for (const e of [1, 3, 5, 20, 50]) {
  const pass = TYPE.filter((t) => find("unindexed", t).decision <= e);
  const fail = TYPE.filter((t) => find("indexed", t).decision > e);
  const dist = TYPE.filter((t) => find("indexed", t).decision <= e && find("unindexed", t).decision > e);
  console.log(String(e).padEnd(20) + pad(pass.length, 14) + pad(fail.length, 15) + pad(dist.length, 22) +
    "  " + (dist.join(", ") || "-"));
}
```

{/* run: skip */}
```bash
# measure.sh — nine runs: four types x two structures, plus a load test for the third structure.
# Each run works with a freshly started server; the catalog always starts at the same size.
rm -f result.jsonl
for S in 0 1 2; do
  for T in load stress soak spike; do
    if [ "$S" = 2 ] && [ "$T" != load ]; then continue; fi
    node library/server.mjs 8851 $S & SERVER=$!
    sleep 0.7
    node library/types.mjs 8851 $S $T >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
node library/report.mjs result.jsonl
```

Output varies from machine to machine; a sample run:

```
threshold p95 <= 20 ms (NF1), target rate 1000 req/s (NF2)
type    structure     reached/s  decision p95  verdict  scan steps    type-specific figure
load    unindexed        996.70          3.18     pass   45,883,050  oracle 0/1495 broken
load    indexed          997.03          0.91     pass       30,000  oracle 0/1499 broken
load    fast-wrong       996.38          1.44     pass       29,955  oracle 1493/1496 broken
stress  unindexed       1107.94       1486.37     fail  222,457,280  break 2000 req/s
stress  indexed         1622.47          1.33     pass      100,918  break > 3000 req/s
soak    unindexed        940.88        991.11     fail  429,907,005  quarter p95 3.2 / 4.0 / 9.3 / 991.1
soak    indexed          999.33          1.10     pass      157,504  quarter p95 1.1 / 1.0 / 0.9 / 1.1
spike   unindexed        903.12         39.91     fail  304,212,030  recovery 2.0 s
spike   indexed         1598.71          1.08     pass      125,030  recovery 1.0 s

threshold scan: how many of the four types give a wrong verdict (unindexed flawed, indexed sound)
p95 threshold (ms)  false pass /4  false fail /4  distinguishing type /4  distinguishing types
1                                0              3                     1  load
3                                0              0                     4  load, stress, soak, spike
5                                1              0                     3  stress, soak, spike
20                               1              0                     3  stress, soak, spike
50                               2              0                     2  stress, soak
```

The duration columns depend on this machine; the **scan step** does not: a full scan's
step count is the record count at that moment, and an indexed lookup's is its binary
logarithm. The requests sent are arithmetic too — the target rate times the duration,
55,000 across nine runs. The claims rest on this set; values read from a single run are
marked *in this run*.

#### Four Types, Four Decisions

The unindexed structure **passed one of the four types and failed three**. Load testing
found p95 at the target rate at 3.18 ms in this run — about a sixth of the threshold. The
same structure broke at twice the target rate under stress testing, climbed to 991.1 ms in
the last quarter under soak testing, and under spike testing could not drop back below the
threshold within two seconds of the surge. Same metric, same threshold, four separate
decisions; the difference is the window.

The soak test's quarter sequence also gives the reason. The load is constant, the code is
the same; the only thing that changes is the number of records in the catalog. A full scan
touches the entire table on every request, the table grows by five rows on every request,
service time rises, and saturation catches up somewhere along the way. The scan step shows
this independently of the run: the unindexed structure spent 429,907,005 row steps while
the indexed structure did the same work in 157,504 steps — **about 2,730 times**. The name
of the defect class this catches is **degradation tied to data growth**; it is the input to
the anti-pattern diagnoses in the Performance Anti-Patterns and Monitoring course.

The number stress testing gives is of a different kind. The ladder is kept short; a longer
one would mix data growth into the rate. The unindexed structure broke at 2,000 req/s; the
indexed structure was still under the threshold at 3,000 req/s. This is not a pass–fail but
a distance: how many multiples of the target rate the structure holds up to.

#### Threshold Scan

A threshold can only be defended with two numbers: how many real defects it misses and how
many sound runs it turns red.

At the 20 milliseconds NF1 gives, **false pass is 1, false fail is 0**. The escaping type is
load testing: in a two-second run the table only grows by 10,000 rows, so the flawed
structure stays well under the threshold. Lowering the threshold to 5 ms does not change
this; at 3 ms every type classifies correctly in this run. Tightening to 1 ms overcorrects:
the sound structure now fails three of the four types, its own p95 crossing 1 ms under
stress, soak and spike. At 50 ms false pass rises to 2.

Two conclusions follow. **Load testing cannot see this defect class at NF1's threshold**,
because its window gives the defect no time to grow. And a value like 3 ms is derived from
no budget — it is fitted to this run's numbers, and two milliseconds tighter the same fitted
value turns three sound results into false fails. Decoupling the threshold from its source
empties the decision without fixing the scan.

#### The Class It Misses and the Cost

The third structure is indexed, fast and wrong: it computes the loan count from only a
single day's records. Load testing's p95 is 1.44 ms in this run — indistinguishable from the
sound structure's 0.91 ms — and all four types would let this structure pass, since none of
them read the response body. Running the oracle shows 1,493 of 1,496 checks come back
broken. The name of the defect class this misses is **a response-correctness defect**:
performance tests read duration, not the body. What keeps them blind is not the window, it
is the metric.

The run-dependent side of the cost depends on the environment: nine runs finished in under a
minute on this machine. The run-independent side is countable — nine server processes, 55,000
requests, roughly 1.00 billion row steps in the unindexed structure. The most
expensive type is soak testing: close to five times the requests of load testing, and about
9.4 times its scan steps.

#### Summary

- Four types read the same metric (p95) in a different decision window: the whole run, the
  2x step, the last quarter, the first second after the surge.
- A threshold is written with its source: NF1's 20 ms comes from splitting a 60 ms budget
  assumption evenly across three services, never read off a measurement.
- The same defect got four separate decisions: load testing passed, stress testing broke at
  2,000 req/s, soak testing found 991.1 ms in the last quarter in this run, spike testing
  only recovered a window later.
- The threshold scan gives false pass 1, false fail 0 at NF1; at 3 ms every type classifies
  correctly in this run, but two milliseconds tighter the same fitted value turns three sound
  results into false fails — a threshold fitted to one run is fragile.
- The class it catches is degradation tied to data growth (scan steps about 2,730 times
  apart); the class it misses is a response-correctness defect: a fast, wrong structure
  passes all four types.

#### Next Step

Every run here sent requests to one endpoint at a fixed rate without pausing. A real desk day is
not like that: the clerk searches, checks a book out, and pauses between transactions. The next
lesson takes on the scenario — how pauses and transaction ratios produce a different result at
the same target rate, how an unrealistic scenario lets a real defect pass, and at what share of
the mix the defect becomes visible.

### Test Scenario Modeling

Source: https://academia.sh/en/courses/non-functional-testing/test-scenario-modeling

The shape of the load, not its size: measuring at what share of the workload mix a defect becomes visible, a search-heavy unrealistic scenario producing a false pass at the same target rate, a closed-loop generator never reaching the target rate because of its virtual-user count and think time, and the same nominal rate giving three separate p95 values.

Every run in the previous lesson sent requests to a single endpoint, at a fixed rate,
without a pause. A real circulation-desk day is not like that: the clerk searches first,
then checks a book out, and pauses between transactions. This lesson takes on the
**shape** of the load rather than its **size**, and measures this: the same target rate
produces a different decision on the same system once the scenario changes.

#### The Scenario's Two Parameters

**Workload mix** is how the total load is distributed across request types. A system's
endpoints do not cost the same; the mix determines which path the load falls on, and so
which defect class comes close to saturation.

**Think time** is the pause a virtual user takes between two transactions, and it only has
meaning in a closed-loop generator. In a closed loop there are $N$ virtual users; each
waits for the response, pauses for $Z$, then sends the next request; the rate reached is
$N / (Z + R)$. Here $R$ is the system's own response time — in other words, **the load is a
function of the system under test**. An open-loop generator carries no such feedback.

#### Threshold and Assumptions

The threshold carries over from the previous lesson unchanged. **NF1 (assumption +
calculation) — the circulation-desk flow's 60 ms server-side budget split evenly across
three services: p95 ≤ 20 ms.** **NF2 (assumption) — the peak-hour target is 1,000 req/s.**
This lesson writes two more assumptions.

**NF3 (assumption) — the circulation desk's peak-hour workload mix: half the requests are
loans, half are searches.** Rationale: the person who comes to the desk comes to check
a book out; catalog searches also happen through other endpoints but never exceed half of
desk traffic. **NF4 (assumption) — in the closed-loop model, the virtual-user count is
derived from the target rate: $N = 1000 \cdot (Z + R_0)$, $R_0 \approx 2$ ms.** Two pairs
are tested: 22 users / 20 ms and 202 users / 200 ms. Both have a nominal rate of
1,000 req/s.

The decision's owner in this lesson is not the release, it is the test itself: the choice
of scenario determines which defect class gets measured, and a poorly chosen scenario
reports green.

#### Harness

The service has two endpoints. `/search` looks a book up by number in the catalog and is
indexed in both structures; `/loan` validates the member's open loan count and
writes five records, and the index exists only in the sound structure. The catalog starts
with 110,000 records, so a full scan is six times more expensive than in the previous
lesson.

```js
// library/server.mjs — circulation-desk service: /search looks up a book, /loan
// validates the member's open loan count and writes five records. Arguments:
// <port> <structure 0|1> — 0 = no member index on the loan path (full scan), 1 = indexed.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 110000, BOOK = 5;  // 220 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_book ON loan(book_no)");      // the search path is indexed in both structures
  if (structure === "1") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const search = db.prepare("SELECT id, member_no, day FROM loan WHERE book_no = ? LIMIT 5");
  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let searches = 0, loans = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ searches, loans, records, scan })); return; }
    if (u.pathname === "/search") {
      searches += 1;
      res.end(JSON.stringify({ found: search.all(u.searchParams.get("book") ?? "K-1").length }));
      return;
    }
    loans += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(loans + j) % 9000}`, loans % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}
```

The generator carries both loop styles. A scenario is just three numbers: the loan
share, the virtual-user count, and the think time.

```js
// library/scenario.mjs — open- and closed-loop generator; the scenario sets the request
// mix and the think time. Arguments: <port> <structure 0|1> <scenario>
import { Agent, request } from "node:http";

const TARGET = 1000, THRESHOLD = 20, DURATION = 3;  // NF2: target rate, NF1: p95 threshold (ms), run length
const SCENARIO = {                          // share: loan-request ratio, n/z: closed loop
  easy: { share: 0.10 }, "share-25": { share: 0.25 }, realistic: { share: 0.50 }, "share-100": { share: 1.00 },
  "closed-22": { share: 0.50, n: 22, z: 20 }, "closed-202": { share: 0.50, n: 202, z: 200 },
};
const [port, structure, name] = [Number(process.argv[2]), process.argv[3], process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const percentile = (d, p) => { const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; };
let seq = 0;

function oneRequest(samples, share) {       // a `share` fraction of requests goes to the loan path
  const i = seq++;
  const path = (i * 9973) % 1000 < share * 1000 ? `/loan?member=U-${i % 500}` : `/search?book=K-${i % 9000}`;
  const start = performance.now();
  return new Promise((done) => {
    const r = request({ port, path, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push(performance.now() - start); done(); });
    });
    r.on("error", () => { samples.push(performance.now() - start); done(); });
    r.end();
  });
}

async function open(duration, rate, share) {   // sends at the target rate, does not wait for the response
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples, share)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

async function closed(duration, n, z, share) {   // n virtual users; each pauses z ms after a request
  const samples = [];
  const start = performance.now();
  const worker = async () => {
    while (performance.now() - start < duration * 1000) {
      await oneRequest(samples, share);
      await new Promise((c) => setTimeout(c, z));
    }
  };
  await Promise.all(Array.from({ length: n }, worker));
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

const sc = SCENARIO[name];
if (Number.isInteger(port) === false || sc === undefined) {
  console.log(`usage: node library/scenario.mjs <port> <structure 0|1> ${Object.keys(SCENARIO).join("|")}`);
} else {
  await oneRequest([], 1);                 // warm-up request
  const r = sc.n === undefined ? await open(DURATION, TARGET, sc.share) : await closed(DURATION, sc.n, sc.z, sc.share);
  const o = await metrics();
  console.log(JSON.stringify({ name, structure: structure === "0" ? "unindexed" : "indexed", share: sc.share,
    loop: sc.n === undefined ? "open" : `closed N=${sc.n} Z=${sc.z}`, p95: percentile(r.samples, 95),
    rate: r.samples.length / r.elapsed, loans: o.loans, scan: o.scan }));
}
agent.destroy();
```

```js
// library/report.mjs — turns ten runs into a table and scans the p95 threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, SCENARIO = ["easy", "realistic", "closed-22", "closed-202"];
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, a) => s.find((r) => r.structure === y && r.name === a);
const fmt2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const verdict = (x) => (x <= THRESHOLD ? "pass" : "fail");

console.log(`threshold p95 <= ${THRESHOLD} ms (NF1), target rate 1000 req/s (NF2), run 3 s`);
console.log("scenario     loop               loan share  structure   reached/s  p95(ms)  verdict");
for (const a of SCENARIO) for (const y of ["unindexed", "indexed"]) {
  const r = find(y, a);
  console.log(a.padEnd(13) + r.loop.padEnd(19) + pad(fmt2(r.share), 10) +
    "  " + y.padEnd(11) + pad(fmt2(r.rate), 10) + pad(fmt2(r.p95), 9) + pad(verdict(r.p95), 9));
}

console.log("\nloan-share scan (unindexed structure, open loop, same 1000 req/s target)");
console.log("loan share      loan req/s  p95(ms)  verdict   scan steps");
for (const a of ["easy", "share-25", "realistic", "share-100"]) {
  const r = find("unindexed", a);
  console.log(pad(fmt2(r.share), 10) + pad(fmt2(r.loans / 3), 16) + pad(fmt2(r.p95), 9) +
    pad(verdict(r.p95), 9) + pad(r.scan.toLocaleString("en-US"), 13));
}

console.log("\nthreshold scan: how many of the four scenarios give a wrong verdict");
console.log("p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones");
for (const e of [2, 5, 15, 20, 30, 100]) {
  const pass = SCENARIO.filter((a) => find("unindexed", a).p95 <= e);
  const fail = SCENARIO.filter((a) => find("indexed", a).p95 > e);
  const dist = SCENARIO.filter((a) => find("indexed", a).p95 <= e && find("unindexed", a).p95 > e);
  console.log(String(e).padEnd(20) + pad(pass.length, 14) + pad(fail.length, 15) + pad(dist.length, 27) +
    "  " + (dist.join(", ") || "-"));
}
```

{/* run: skip */}
```bash
# measure.sh — ten runs: four scenarios x two structures, plus two extra loan shares.
# Each run works with a freshly started server; the catalog always starts with 110,000 records.
rm -f result.jsonl
for T in 0 1; do
  for SC in easy realistic closed-22 closed-202; do
    node library/server.mjs 8852 $T & SERVER=$!
    sleep 1.2
    node library/scenario.mjs 8852 $T $SC >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
for SC in share-25 share-100; do
  node library/server.mjs 8852 0 & SERVER=$!
  sleep 1.2
  node library/scenario.mjs 8852 0 $SC >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/report.mjs result.jsonl
```

Output varies from machine to machine; a sample run:

```
threshold p95 <= 20 ms (NF1), target rate 1000 req/s (NF2), run 3 s
scenario     loop               loan share  structure   reached/s  p95(ms)  verdict
easy         open                     0.10  unindexed      997.48     8.21     pass
easy         open                     0.10  indexed        997.52     1.69     pass
realistic    open                     0.50  unindexed      916.07   760.33     fail
realistic    open                     0.50  indexed        997.51     1.83     pass
closed-22    closed N=22 Z=20         0.50  unindexed      692.28    29.06     fail
closed-22    closed N=22 Z=20         0.50  indexed        970.94     2.55     pass
closed-202   closed N=202 Z=200       0.50  unindexed      855.57    77.97     fail
closed-202   closed N=202 Z=200       0.50  indexed        985.30    11.75     pass

loan-share scan (unindexed structure, open loop, same 1000 req/s target)
loan share      loan req/s  p95(ms)  verdict   scan steps
      0.10           99.67     8.21     pass   33,112,755
      0.25          248.67    13.61     pass   83,449,425
      0.50          500.00   760.33     fail  170,621,250
      1.00          998.67  6106.87     fail  351,992,550

threshold scan: how many of the four scenarios give a wrong verdict
p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones
2                                0              2                          2  easy, realistic
5                                0              1                          3  easy, realistic, closed-22
15                               1              0                          3  realistic, closed-22, closed-202
20                               1              0                          3  realistic, closed-22, closed-202
30                               2              0                          2  realistic, closed-202
100                              3              0                          1  realistic
```

#### The Mix Changes the Decision

The target rate is 1,000 req/s in all four runs. The only thing that changes is how that
rate is distributed across the endpoints. At a loan share of 0.10 the flawed structure
stays at 8.21 ms and **passes**; at 0.25 it is still passing at 13.61 ms; at 0.50 it jumps
to 760.33 ms. The difference is not in the system, it is in the scenario.

The reason is in the second column: the loan request rate climbs from 99.67 to 500.00.
In the flawed structure this endpoint does a full scan and its own ceiling is around
400 req/s; the share is under that ceiling at 0.25 and over it at 0.50. The scan step shows
this independently of the run: from 33,112,755 to 170,621,250, **about 5.2 times**. The
name of the defect class this catches is **saturation of a single path**: while the system
as a whole meets the target rate, one endpoint hits its ceiling and corrupts the latency of
every request.

The easy scenario is where this escapes. A search request is the cheapest one to write, so
scenarios tend to skew toward search-heavy. If NF3 writes the realistic mix as 0.50, a test
run at 0.10 produces a **false pass**, and the name of the class it misses is already
known.

#### The Closed Loop Throttles Its Own Load

The bottom two scenarios run the same mix (0.50) through the closed loop, and both have a
nominal rate of 1,000 req/s. In the flawed structure the rate reached is 692.28 and
855.57 req/s — that is, 69.2% and 85.6% of the target. **The generator throttled its own
load because the system under test slowed down.** The load NF2 calls for was never
applied; the p95 measured is the p95 of a different rate.

The result is an understatement. The open loop reported 760.33 ms at the same mix; the
closed loop reported 29.06 and 77.97. Same system, same defect, a number **about 26 times
and 10 times smaller**. In this run all three still cross the 20 ms threshold so the
decision stays the same; if the threshold were 30 ms, closed-22 would pass while the open
loop would still fail.

The sound-structure column shows think time's own effect separately. Even with no defect
at all, the same nominal rate gives three separate p95 values: 1.83 ms in the open loop,
2.55 ms with 22 users / 20 ms, 11.75 ms with 202 users / 200 ms — **a spread of about
6.4 times**. The number belongs to the generator's parameters, not the system.

#### Threshold Scan and Cost

The scan gives a two-sided result. At NF1's 20 ms, false pass is 1 (the easy scenario),
false fail is 0 — and this still holds at 15 ms in this run. Tightening further to 5 ms
introduces one false fail — the sound structure's closed-202 scenario, whose own
think-time noise alone crosses 5 ms — and tightening to 2 ms doubles that to two, adding
closed-22. At 100 ms false pass climbs to three; only the open-loop realistic scenario
still distinguishes.

The class it misses can also be counted: since the two closed-loop scenarios never reach
the rate NF2 calls for, **the saturation that appears at the target rate** was never
measured in them; in the flawed structure the missing load is about 307 and 144 req/s. The
run-dependent side of the cost depends on the environment — ten runs finished in under a
minute on this machine. The run-independent side: every open-loop run sends 3,000
requests and the loan-request count comes directly from the share (300, 750, 1,500,
3,000); the scan step in the unindexed structure ranges from 33,112,755 to 351,992,550,
**about 10.7 times**.

#### Summary

- The scenario has two parameters: workload mix decides which endpoint the load falls on,
  think time decides the rate reached in a closed loop.
- The same 1,000 req/s target passed at a loan share of 0.10 (8.21 ms), passed at 0.25
  (13.61 ms), and failed at 0.50 (760.33 ms) in this run; the defect became visible through
  the scenario, not the system.
- The class it catches is saturation of a single path: as the loan request rate climbs
  from about 100 to 500 req/s, the endpoint crosses its own ceiling and the scan step rises
  about 5.2 times.
- The closed loop never reached the target rate (about 69% and 86% of it) and understated
  the degradation by roughly 26 and 10 times; the p95 measured is not the p95 of the
  requested load.
- Even in the sound structure, the generator's own parameters carry p95 from 1.83 to
  11.75 ms; tightening the threshold to 5 ms is where this noise starts producing a false
  fail.

#### Next Step

Up to this point every decision was made by looking at a single number: p95. That choice
was never discussed. If the average had been taken, the 760.33-millisecond run would look
completely different; if p99 had been chosen, the decision would change again, and how many
samples a three-second run's p99 rests on was never asked. That throughput and latency need
to be read together was only implied. The next lesson takes on metric choice: what an
average hides, how many samples a percentile needs to be meaningful, and why latency read
apart from throughput is misleading.

### Metric Selection

Source: https://academia.sh/en/courses/non-functional-testing/metric-selection

Drawing seven separate results from the same run: the mean and median passing the threshold while the tail percentiles fail, counting the minimum sample size a percentile needs to be readable through a subsample scan, and showing through a target-rate ladder that a latency figure cannot be read without the reached rate reported alongside it.

The previous two lessons tied every decision to a single number: p95. That choice was
never discussed. If the average had been taken, the same runs would look completely
different; if p99 had been chosen, the decision would change again, and how many samples
a three-second run's p99 rests on was never asked. This lesson treats the metric itself as
a decision.

#### A Metric Is a Choice

What a run produces is not a single number but thousands of latency values. A metric is the
function that draws a single number out of that distribution, and the choice changes what
the threshold means. The **mean** sums every sample with equal weight; one very slow
request moves it a little, a thousand moderately slow requests move it a lot. **Percentile
latency** instead reads one position in the sorted sample: p99 is the value below which
99 percent of requests fall, and it says nothing about how much worse the remaining one
percent is.

The threshold carries over from the previous lessons unchanged: **NF1 (assumption +
calculation) — p95 ≤ 20 ms.** This lesson adds a sample-size condition to the threshold.

**NF5 (calculation) — for a percentile to carry a decision, the run needs at least
$1/(1-p)$ samples: 20 for p95, 100 for p99, 1,000 for p99.9.** Rationale: if the share the
percentile excludes does not reach at least one sample, the value read belongs not to that
share but to a lower point in the distribution. This number is not an assumption but a
lower bound derived from the sample count, and it is tested below with a scan.

The decision's owner is the choice of metric itself: a threshold tied to p99 also places a
lower bound on the run's duration. Choosing a metric is also choosing how long the test has
to run.

#### Harness

The service carries over from the previous lesson unchanged: `/search` is indexed in both
structures, `/loan` only in the sound one. This time the generator accumulates every
latency and writes it to a file; the analysis runs without a server.

```js
// library/server.mjs — circulation-desk service: /search looks up a book, /loan
// validates the member's open loan count and writes five records. Arguments:
// <port> <structure 0|1> — 0 = no member index on the loan path (full scan), 1 = indexed.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 110000, BOOK = 5;  // 220 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_book ON loan(book_no)");      // the search path is indexed in both structures
  if (structure === "1") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const search = db.prepare("SELECT id, member_no, day FROM loan WHERE book_no = ? LIMIT 5");
  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let searches = 0, loans = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ searches, loans, records, scan })); return; }
    if (u.pathname === "/search") {
      searches += 1;
      res.end(JSON.stringify({ found: search.all(u.searchParams.get("book") ?? "K-1").length }));
      return;
    }
    loans += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(loans + j) % 9000}`, loans % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}
```

```js
// library/measure.mjs — open-loop generator; accumulates a run's full set of latencies
// and prints a JSON line. Arguments: <port> <structure 0|1> <target rate> <loan share> <duration s>
import { Agent, request } from "node:http";

const [port, structure, rate, share, duration] = process.argv.slice(2).map((x, i) => (i === 1 ? x : Number(x)));
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
let seq = 0;

function oneRequest(samples) {
  const i = seq++;
  const path = (i * 9973) % 1000 < share * 1000 ? `/loan?member=U-${i % 500}` : `/search?book=K-${i % 9000}`;
  const start = performance.now();
  return new Promise((done) => {
    const r = request({ port, path, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push(performance.now() - start); done(); });
    });
    r.on("error", () => { samples.push(performance.now() - start); done(); });
    r.end();
  });
}

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/measure.mjs <port> <structure 0|1> <target rate> <loan share> <duration>");
} else {
  await oneRequest([]);                             // warm-up request
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  const elapsed = (performance.now() - start) / 1000;
  console.log(JSON.stringify({ structure: structure === "0" ? "unindexed" : "indexed", rate, share,
    reached: samples.length / elapsed, ms: samples.map((x) => Math.round(x * 1000) / 1000) }));
}
agent.destroy();
```

The subsample scan draws samples with replacement from the recorded latencies; the
generator is a linear congruential one and its seed is visible.

```js
// library/analysis.mjs — metric selection, sample-size scan and a throughput-latency
// reading over the recorded latencies. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, SEED = 20260731, TRIALS = 200;      // NF1 threshold, visible seed, subsample trial count
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, h, p) => s.find((r) => r.structure === y && r.rate === h && r.share === p);
const fmt = (x, n = 2) => x.toFixed(n);
const pad = (x, n) => String(x).padStart(n);
const mean = (d) => d.reduce((a, b) => a + b, 0) / d.length;
const percentile = (d, p) => { const t = [...d].sort((a, b) => a - b);
  return t[Math.min(t.length - 1, Math.floor((p / 100) * t.length))]; };

const A = find("unindexed", 1700, 0.25), B = find("indexed", 1700, 0.25);
const METRIC = [["mean", (d) => mean(d)], ["p50", (d) => percentile(d, 50)], ["p90", (d) => percentile(d, 90)],
  ["p95", (d) => percentile(d, 95)], ["p99", (d) => percentile(d, 99)], ["p99.9", (d) => percentile(d, 99.9)],
  ["max", (d) => Math.max(...d)]];
console.log(`same two runs, seven metrics; threshold ${THRESHOLD} ms (NF1), samples ${A.ms.length} / ${B.ms.length}`);
console.log("metric    unindexed(ms) verdict indexed(ms)  verdict unindexed/indexed");
for (const [name, f] of METRIC) {
  const a = f(A.ms), b = f(B.ms);
  console.log(name.padEnd(10) + pad(fmt(a), 12) + pad(a <= THRESHOLD ? "pass" : "fail", 7) +
    pad(fmt(b), 13) + pad(b <= THRESHOLD ? "pass" : "fail", 7) + pad(fmt(a / b), 18));
}

let t = SEED;
const rand = () => { t = (1103515245 * t + 12345) % 2147483648; return t / 2147483648; };
const subsample = (d, n) => Array.from({ length: n }, () => d[Math.floor(rand() * d.length)]);
console.log(`\nsample-size scan: ${TRIALS} subsamples, seed ${SEED}; full-sample p99 ` +
  `unindexed ${fmt(percentile(A.ms, 99))} ms, indexed ${fmt(percentile(B.ms, 99))} ms`);
console.log("samples  p99 min  p99 max  spread factor  false pass/200  false fail/200");
for (const n of [20, 50, 100, 500, 2000]) {
  const a = Array.from({ length: TRIALS }, () => percentile(subsample(A.ms, n), 99));
  const b = Array.from({ length: TRIALS }, () => percentile(subsample(B.ms, n), 99));
  console.log(pad(n, 5) + pad(fmt(Math.min(...a)), 13) + pad(fmt(Math.max(...a)), 14) +
    pad(fmt(Math.max(...a) / Math.min(...a)), 13) + pad(a.filter((x) => x <= THRESHOLD).length, 18) +
    pad(b.filter((x) => x > THRESHOLD).length, 18));
}

console.log("\nthroughput-latency: unindexed structure, same mix, four target rates");
console.log("target rate  reached/s  reached/target  p95(ms)  verdict");
for (const h of [900, 1300, 1700, 2100]) {
  const r = find("unindexed", h, 0.25);
  console.log(pad(h, 9) + pad(fmt(r.reached), 12) + pad(fmt(r.reached / h, 3), 16) +
    pad(fmt(percentile(r.ms, 95)), 9) + pad(percentile(r.ms, 95) <= THRESHOLD ? "pass" : "fail", 7));
}
```

{/* run: skip */}
```bash
# measure.sh — five runs: the same scenario on two structures (5 s), plus three extra target rates on the unindexed structure (2 s).
rm -f result.jsonl
for T in 0 1; do
  node library/server.mjs 8853 $T & SERVER=$!
  sleep 1.2
  node library/measure.mjs 8853 $T 1700 0.25 5 >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
for H in 900 1300 2100; do
  node library/server.mjs 8853 0 & SERVER=$!
  sleep 1.2
  node library/measure.mjs 8853 0 $H 0.25 2 >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/analysis.mjs result.jsonl
```

Output varies from machine to machine; a sample run:

```
same two runs, seven metrics; threshold 20 ms (NF1), samples 8496 / 8494
metric    unindexed(ms) verdict indexed(ms)  verdict unindexed/indexed
mean             10.79   pass         0.79   pass             13.60
p50               9.40   pass         0.64   pass             14.80
p90              19.97   pass         1.47   pass             13.58
p95              24.80   fail         1.74   pass             14.29
p99              37.59   fail         2.42   pass             15.53
p99.9            96.32   fail         6.41   pass             15.02
max             151.32   fail         8.53   pass             17.75

sample-size scan: 200 subsamples, seed 20260731; full-sample p99 unindexed 37.59 ms, indexed 2.42 ms
samples  p99 min  p99 max  spread factor  false pass/200  false fail/200
   20        18.85         45.63         2.42                22                 0
   50        28.60         45.63         1.60                 0                 0
  100        35.80         45.63         1.27                 0                 0
  500        40.07         45.62         1.14                 0                 0
 2000        40.07         40.07         1.00                 0                 0

throughput-latency: unindexed structure, same mix, four target rates
target rate  reached/s  reached/target  p95(ms)  verdict
      900      896.57           0.996    12.97   pass
     1300     1290.84           0.993    15.14   pass
     1700     1694.54           0.997    24.80   fail
     2100     1404.40           0.669  1388.52   fail
```

#### What the Average Hides

The first table draws seven numbers from a single run, and all seven give the same
structure **two separate decisions**. In this run the mean is 10.79, the median 9.40 and
p90 19.97 — all under the threshold; p95 is 24.80, p99 37.59, p99.9 96.32 and the max
151.32 — all over it. The code is the same, the run is the same, the sample is the same;
the only thing that changes is which point of the distribution gets read. The numbers
belong to the measurement class and depend on this machine; **which pair of metrics the
cutoff falls between shifts from run to run** — in this run it is p90 and p95 that split,
in another it could just as well be p95 and p99. What stays independent of the run is the
ordering itself: moving toward the tail, the value read grows, and so does the unindexed
structure's ratio to the sound one.

Why the average passes shows up in the last column. In this run a quarter of requests go
to the loan path and do a full scan; the remaining three-quarters are cheap searches.
The average blends the two populations and cuts the expensive one's weight to a quarter.
Tail percentiles do not blend: p99 by definition sits only in the slowest one percent, and
there the unindexed structure is 15.53 times the sound one in this run, and 17.75 times at
the max. The exact multiple depends on the run; what stays independent of it is that the
ratio in the tail is larger than at the mean.

The name of the defect class this catches is **tail latency**. The sound structure passes
all seven metrics, so it is also visible in this run that tightening the metric toward p99
produces no false fail: the sound structure never turns red.

#### How Many Samples a Percentile Rests On

The second table draws subsamples from the same latencies and recomputes p99 at each size.
At the full sample p99 is 37.59 ms; in twenty-sample subsamples the same figure ranges from
18.85 to 45.63, a **spread of about 2.42 times**. This is the swing that breaks the
decision: 22 of two hundred subsamples put p99 under the threshold and the run reports a
**false pass**. At fifty samples this number already drops to zero.

That the false-pass count reaches zero this early does not overturn NF5's bound, which is a
worst-case guarantee, not a promise that every smaller sample must fail. The share p99
excludes is one percent, so under a hundred samples that share can go unrepresented in the
draw, and the value read then comes from a lower point in the distribution; this run's zero
at fifty samples is a case where the draws still happened to land near the true tail. The
spread keeps narrowing regardless: 1.27 at a hundred samples, 1.14 at five hundred, 1.00 at
two thousand. The false-fail column stays at zero across every size: the sound structure's
p99 sits so far from the threshold that subsampling cannot push it back over. **Sample
count only distorts the false-pass side** — too few samples always bias toward an
optimistic result.

The rule that follows looks at run duration: at a target rate of 1,700 req/s, the hundred samples
p99 needs are gathered in 0.06 seconds, but in an environment running at 10 req/s outside peak hour
the same count needs 10 seconds. For p99.9 that duration grows tenfold.

#### Latency Cannot Be Read Without Throughput

The third table runs the same structure at four target rates. In the first three rows the
reached rate is between 0.993 and 0.997 of the target; p95 is 12.97, 15.14 and 24.80 ms.
These three numbers are comparable, because all three were measured at essentially the
requested load — the last of them already crosses the threshold, but that is a real result,
not an artifact of an unmet rate.

The fourth row is not comparable. The target is 2,100 req/s, the reached rate is 1,404.40
— **0.669 of the target**. p95 shows as 1,388.52 ms, but that number is not the p95 of
2,100 req/s; it is the p95 of the 1,404 req/s the system could actually accept. Roughly
696 requests per second were never served. A report that carries only the latency column
reads this row as "slow," but what it says is something else: **the system never reached
the requested load.** Latency and throughput are read as a single pair; a latency figure
with no reached rate reported alongside it does not say which load it belongs to.

#### Cost and the Class It Misses

The run-dependent side of the cost depends on the environment: five runs finished in under
half a minute on this machine. The run-independent side is countable — the two analysis
runs accumulated 8,496 and 8,494 latency values, and the subsample scan ran two hundred
trials at each of five sizes for both structures, drawing 1,068,000 samples in total. The
class it misses was also measured: the mean and median let this defect pass through three
of the seven metrics in this run (mean, median, p90), meaning a metric set's own limits are
written in its false-pass count too.

#### Summary

- A metric is a choice: in the same run the mean (10.79) and median (9.40) pass the
  threshold while p95 (24.80) and p99 (37.59) already fail.
- The average blends expensive and cheap requests; a tail percentile does not, and it
  separates the two structures by 15.53 times at p99 and 17.75 times at the max.
- A percentile is sensitive to sample count: at twenty samples the p99 estimate spreads
  about 2.42 times and 22 of 200 subsamples give a false pass; this already drops to zero
  by fifty samples.
- NF5 ties this to a lower bound: a percentile needs at least $1/(1-p)$ samples, meaning
  100 for p99 and 1,000 for p99.9 — a worst-case guarantee, so an early zero does not
  overturn it.
- Latency is read together with throughput: at a 2,100 req/s target the reached rate drops
  to 0.669, so a p95 of 1,388.52 ms is the figure for the accepted load, not the requested
  one.

#### Next Step

The metric is chosen and the threshold has turned red. The report now says: p99 37.59 ms,
threshold 20, fail. That sentence reports a defect but does not say **where** it is. The
same number could come from a computation in the application layer, a scan in the database,
or the size of the response, and the fix for each of the three is different. The next
lesson takes on this attribution: which measurement ties a load-test result to a layer, how
two layers producing the same symptom are told apart, and when attribution is wrong.

### Bottleneck Analysis

Source: https://academia.sh/en/courses/non-functional-testing/bottleneck-analysis

Tying a load-test result to a layer: measuring application, database and transfer duration separately within a request, attributing three modes that give the same client-side symptom to three separate layers, scanning the attribution rule against a share threshold, and showing with numbers a case where the time genuinely is spent in the right layer but the cause is a different decision.

The previous lesson's report reduced the result to a single sentence: p99 37.59 ms,
threshold 20, fail. That sentence reports a defect but does not say **where** it is. The
same number could come from a computation in the application layer, a query in the
database, or the size of the response, and the fix for each of the three is different.
This lesson takes on tying the result to a layer.

#### Layer Share

A request's total duration is measured at the client and is a single number. The
**bottleneck** is wherever consumes most of that duration; finding it requires splitting
the duration **inside** the request. The server reads three clocks on every request:
before the query, after the query, after the computation, and after the response is
written. The gaps between them are the **database**, **application** and **transfer**
layers' durations.

These three numbers alone are not enough, because every layer does some work on every
request. What matters is the **difference**: a **baseline** run carrying no defect is
taken on the same harness, and each layer's duration is read as its difference from the
baseline. **Layer share** is a layer's difference divided by the client's difference.

**NF6 (assumption) — to attribute a result to one layer, that layer's share must be at
least 0.50; otherwise the result counts as unattributable.** Rationale: below a share of
0.50, even completely eliminating that layer removes less than half the symptom, and the
fix decision cannot be defended. The threshold is scanned below.

The decision's owner is a budget: attribution decides which team and which work the fix
goes to. A wrong attribution does not just waste time, it leaves the defect in place. The
Performance Anti-Patterns and Monitoring course built ten diagnoses running from symptom to
cause; the question asked here comes before that: which layer the result **belongs to**.

#### Harness

The loan endpoint runs in five modes. **baseline** does an indexed query and a small
response; **db** does the same query unindexed; **app** runs a computation round in the
application layer; **net** serializes and writes a 4,000-row page fetched once at startup
on every response; **mixed** re-fetches the same page from the database on every request.
All four are tuned to give the same symptom.

```js
// library/server.mjs — loan endpoint, five modes. Every request measures three layers'
// duration separately: database (query), application (computation), transfer (serialization + writing the response).
// Arguments: <port> <mode baseline|db|app|net|mixed>
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 55000, ROUNDS = 29000, ROW = 4000;   // catalog size, computation rounds, page rows
const port = Number(process.argv[2]), mode = process.argv[3];

if (Number.isInteger(port) === false || ["baseline", "db", "app", "net", "mixed"].includes(mode) === false) {
  console.log("usage: node library/server.mjs <port> baseline|db|app|net|mixed");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  if (mode !== "db") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const fetch = db.prepare("SELECT id, book_no, day FROM loan LIMIT ?");
  const cache = mode === "net" ? fetch.all(ROW) : [];   // in net mode the page is fetched once at startup
  let requests = 0, dbTime = 0, appTime = 0, netTime = 0, bytes = 0, scan = 0, computed = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    if (u.pathname === "/metrics") {
      res.writeHead(200, { "content-type": "application/json" });
      res.end(JSON.stringify({ requests, dbTime, appTime, netTime, bytes, scan, computed })); return;
    }
    requests += 1;
    const member = u.searchParams.get("member") ?? "U-1";
    const t0 = performance.now();
    const n = count.get(member).n;                    // database layer
    const page = mode === "mixed" ? fetch.all(ROW) : cache;
    scan += mode === "db" ? START : Math.ceil(Math.log2(START));
    const t1 = performance.now();
    let summary = "";                                 // application layer
    if (mode === "app") for (let k = 0; k < ROUNDS; k += 1) { summary = `${member}/${(n + k) % 97}`; computed += 1; }
    const t2 = performance.now();
    const body = JSON.stringify({ member, loan: n, summary, page });   // transfer layer
    dbTime += t1 - t0; appTime += t2 - t1; bytes += body.length;
    res.writeHead(200, { "content-type": "application/json" });
    res.on("finish", () => { netTime += performance.now() - t2; });
    res.end(body);
  }).listen(port);
}
```

The generator excludes the first second from the measurement and reads the server counters
once more at the start of that window, so that both the client and the server side describe
the same interval.

```js
// library/measure.mjs — open-loop generator; reads the client duration and the server's
// layer counters. Arguments: <port> <mode> <target rate> <duration s>
import { Agent, request } from "node:http";

const WARMUP = 1;   // the first second is not counted: JIT and cache warm-up
const [port, mode, rate, duration] = process.argv.slice(2).map((x, i) => (i === 1 ? x : Number(x)));
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const percentile = (d, p) => { const t = [...d].sort((a, b) => a - b);
  return t[Math.min(t.length - 1, Math.floor((p / 100) * t.length))]; };
let seq = 0;

function oneRequest(samples, sec) {
  const start = performance.now(), i = seq++;
  return new Promise((done) => {
    const r = request({ port, path: `/loan?member=U-${i % 500}`, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push({ ms: performance.now() - start, sec }); done(); });
    });
    r.on("error", () => { samples.push({ ms: performance.now() - start, sec }); done(); });
    r.end();
  });
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

if (Number.isInteger(port) === false) {
  console.log("usage: node library/measure.mjs <port> <mode> <target rate> <duration>");
} else {
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0, m1 = null;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    if (m1 === null && t >= WARMUP) m1 = await metrics();     // the warm-up window closes here
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples, t)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  const m2 = await metrics(), n = m2.requests - m1.requests;
  const d = samples.filter((r) => r.sec >= WARMUP).map((r) => r.ms);
  console.log(JSON.stringify({ mode, rate, samples: d.length,
    client: d.reduce((a, b) => a + b, 0) / d.length, p95: percentile(d, 95),
    db: (m2.dbTime - m1.dbTime) / n, app: (m2.appTime - m1.appTime) / n, net: (m2.netTime - m1.netTime) / n,
    bytes: (m2.bytes - m1.bytes) / n, scan: (m2.scan - m1.scan) / n }));
}
agent.destroy();
```

```js
// library/attribution.mjs — computes layer shares against the baseline, ties attribution to
// a rule, and scans the share threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const SHARE = 0.5;                                  // NF6: minimum share to attribute to one layer
const ACTUAL = { db: "database", app: "application", net: "transfer", mixed: "transfer" };
const LAYER = ["db", "app", "net"], NAME = { db: "database", app: "application", net: "transfer" };
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (k) => s.find((r) => r.mode === k);
const BASE = find("baseline"), MODE = ["db", "app", "net", "mixed"];
const fmt = (x, n = 3) => x.toFixed(n);
const pad = (x, n) => String(x).padStart(n);
const diff = (r) => ({ client: r.client - BASE.client, db: r.db - BASE.db, app: r.app - BASE.app, net: r.net - BASE.net });
const attribute = (r, threshold) => {               // if the largest share crosses the threshold, that layer
  const f = diff(r);
  const top = LAYER.reduce((a, k) => (f[k] > f[a] ? k : a), "db");
  return f[top] / f.client >= threshold ? NAME[top] : "unattributable";
};

console.log(`baseline: client ${fmt(BASE.client)} ms, db ${fmt(BASE.db)}, app ${fmt(BASE.app)}, transfer ${fmt(BASE.net)}, ` +
  `bytes ${Math.round(BASE.bytes)}`);
console.log("mode    client   p95   diff   db share  app share  transfer share  bytes    attribution");
for (const k of MODE) {
  const r = find(k), f = diff(r);
  console.log(k.padEnd(8) + pad(fmt(r.client, 2), 7) + pad(fmt(r.p95, 2), 6) + pad(fmt(f.client, 2), 7) +
    pad(fmt(f.db / f.client, 2), 9) + pad(fmt(f.app / f.client, 2), 10) + pad(fmt(f.net / f.client, 2), 14) +
    pad(Math.round(r.bytes), 9) + "  " + attribute(r, SHARE));
}

console.log("\nshare-threshold scan: the four modes' attribution (actual layer: db=database, " +
  "app=application, net and mixed=transfer)");
console.log("share threshold  correct /4  wrong /4  unattributable /4");
for (const e of [0.3, 0.5, 0.7, 0.8, 1.0]) {
  const a = MODE.map((k) => attribute(find(k), e));
  const correct = a.filter((x, i) => x === ACTUAL[MODE[i]]).length;
  const none = a.filter((x) => x === "unattributable").length;
  console.log(fmt(e, 1).padEnd(17) + pad(correct, 11) + pad(4 - correct - none, 10) + pad(none, 18));
}
```

{/* run: skip */}
```bash
# measure.sh — five runs: baseline and four modes, at the same target rate, each with a fresh server.
rm -f result.jsonl
for M in baseline db app net mixed; do
  node library/server.mjs 8854 $M & SERVER=$!
  sleep 1.2
  node library/measure.mjs 8854 $M 250 6 >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/attribution.mjs result.jsonl
```

Output varies from machine to machine; a sample run:

```
baseline: client 0.629 ms, db 0.032, app 0.000, transfer 0.108, bytes 56
mode    client   p95   diff   db share  app share  transfer share  bytes    attribution
db         2.11  3.45   1.48     0.98     -0.00         -0.06       56  database
app        1.49  2.60   0.86    -0.01      0.90         -0.03       63  application
net        1.51  2.74   0.88    -0.03     -0.00          0.98   160738  transfer
mixed      2.16  3.62   1.53     0.55     -0.00          0.35   160738  database

share-threshold scan: the four modes' attribution (actual layer: db=database, app=application, net and mixed=transfer)
share threshold  correct /4  wrong /4  unattributable /4
0.3                        3         1                 0
0.5                        3         1                 0
0.7                        3         0                 1
0.8                        3         0                 1
1.0                        0         0                 4
```

#### Same Symptom, Three Separate Layers

The first column is the decision itself, and it is the same across all four modes. In this
run the client average ranges from 1.49 to 2.16 ms — about 45% higher at the top than at
the bottom; p95 ranges from 2.60 to 3.62 ms across the four modes. A report that looks only
at the client-side number **cannot tell these four structures apart**.

The share columns tell them apart. In `db` mode the database share is 0.98, in `app` mode
the application share is 0.90, in `net` mode the transfer share is 0.98. Three modes give
the same symptom and get attributed to three separate layers; their fixes are different too
— adding an index, thinning the computation, shrinking the response.

None of the shares in this run exceed 1, but the measurement allows it: if the database
difference came out larger than the client difference, that would show the baseline's own
noise growing relative to a small difference between two runs. The rule only looks at the
largest share, so the attribution would not change; a share over 1 just means the reading
is "the database explains the entire symptom," nothing more.

#### When Attribution Is Wrong

The fourth row is where the rule breaks. In `mixed` mode the database share is 0.55, the
transfer share is 0.35; the rule blames the database. The measurement is correct: time
really is spent in the query, because the query reads 4,000 rows. **The result is still
wrong.**

The evidence is in the last column, and it is independent of the run: `mixed` and `net`
send the **same 160,738 bytes** per response. In `net` mode that page was fetched once at
startup and cached, so the database share is near zero and the rule blames transfer. The
difference between the two modes is not in the query, it is in **the decision about the
size of the response** — and that decision is the same in both. Adding an index to the
database in `mixed` mode changes nothing; the query does not need an index, it reads
4,000 rows regardless.

The rule is this: **layer share says where the time went, not which decision produced it.**
The measurement that answers the second question is a run-independent quantity — bytes
sent, rows read, records scanned. In this run `db` mode scans 55,000 rows per request, the
others scan 16; `net` and `mixed` send 160,738 bytes per request, the others send 56 and
63. Wherever two layers give the same symptom, this is the measurement that tells them
apart.

#### Share Threshold Scan

The second table shifts NF6's 0.50. The table's individual numbers are run-dependent — a
share is drawn from measured duration differences and moves from run to run on this machine
— but **its direction is run-independent** and the decision is made in that direction. At a
low threshold all four modes get attributed to a layer and `mixed` gets a **wrong
attribution**; as the threshold rises, the wrong attribution clears first, then the correct
ones start falling under the threshold too and turn **unattributable**. In this run the
turn starts at 0.7, where `mixed` drops out; the other three hold their attribution up to
just under 1.0, since their own shares (0.98, 0.90, 0.98) sit close to the ceiling, and at
exactly 1.0 none of them clears it either — in another run the turn could start a step
earlier or later, because what crosses the threshold is the share itself.

This is the trade-off between the two error types. A low threshold blames an innocent
layer; a high threshold cannot pin the defect on anyone, and the finding closes without
becoming a record. NF6's 0.50 does not clear the wrong attribution by itself — what clears
it is not the threshold but reading the run-independent quantity.

#### Cost and the Class It Misses

The run-dependent side of the cost depends on the environment: five runs finished in under
forty seconds on this machine. The run-independent side is countable — five server
processes, five of every six seconds measured per run (the first second is set aside for
warm-up), and every run reads four clocks per request, meaning instrumentation adds four
extra measurements per request.

The class it misses can also be counted: this harness only splits duration **inside the
server**. The residual between the client's difference and the sum of the measured layers
— about 14% of the difference in `app` mode — is written to no layer. Network round trip,
event-loop wait and the operating system's socket work stay there. In a system approaching
saturation this residual grows and share-based attribution stops being usable; the metric
to read then is not layer share but the previous lesson's throughput–latency pair.

#### Summary

- Layer share is a layer's difference from the baseline run divided by the client's
  difference; the attribution rule requires the largest share to cross NF6's 0.50.
- The same client-side symptom (average 1.49–2.16 ms) got attributed to three separate
  layers: database at a share of 0.98, application at 0.90, transfer at 0.98.
- Attribution is wrong in `mixed` mode: time really is spent in the query, but it sends
  the same 160,738 bytes as `net`; the cause is not the query, it is the decision about
  response size.
- The share-threshold scan shows the trade-off: as the threshold rises, the wrong
  attribution clears first, then correct attributions fall too and unattributable runs
  climb; which step the turn happens at is run-dependent.
- Layer share says where the time went; the run-independent quantity says which decision
  produced it (55,000 rows scanned, 160,738 bytes sent).

#### Next Step

Every number up to this point was read from **a single run**. This lesson's baseline was
taken once and every share was divided by it; the layer-share residual not fully summing
to the client's own difference — about 14% left over in `app` mode — shows that two runs
of the same harness do not give the same number. And the harness excludes the first second
from measurement altogether — the existence of a warm-up effect was assumed but its size
was never measured. The next lesson puts the measurement itself to the test: how many runs
warm-up lasts, how much variability exists between runs of the same structure, and how many
runs a difference between two structures needs before it can be called meaningful.

### Benchmarking Discipline

Source: https://academia.sh/en/courses/non-functional-testing/benchmarking-discipline

The condition for a performance difference to count as meaningful: measuring the warm-up window with request blocks, counting a harness's own noise by running the same code as two arms, showing a round where a single-round reading reads a regression backwards, and balancing the false-fail–false-pass trade-off by scanning threshold and round count together.

The previous lesson's shares were divided by a single baseline run, and one share crossed
1. The cause was not a defect: the same harness gives two different numbers when run
twice. The harness also excluded the first second from measurement — the existence of a
warm-up effect was accepted, its size never measured. This lesson puts the measurement
itself to the test.

The question is a release decision. The loan service's **candidate** version adds the
member's overdue book count to the circulation-desk screen, and this runs one more query
per request. The release gate has to ask: when the candidate version's measurement comes
back high, does the difference come from the version, or from the measurement?

#### When a Difference Counts as Meaningful

Two runs of the same harness not giving the same number is called **noise**; numbers
spreading from run to run is called **run-to-run variance**. A version difference can only
be read to the extent it can be told apart from that spread.

**NF9 (assumption) — if the candidate version's measurement exceeds the baseline's by more
than five percent, it counts as a regression and the release stops.** Its source is not a
requirement or a measurement but a number chosen for gate sensitivity; it is scanned below.
The decision's owner is the release gate: when it turns red, the candidate version does not
ship.

To count the threshold's two error types, the harness runs three arms. The **baseline** and
**control** arms run **the same code**; the **candidate** arm runs the new version. The
control arm's real difference is zero; if the gate turns it red, that is a **false fail**.
The candidate arm's difference is real; if the gate lets it pass, that is a **false pass**.
Running two identical arms is the cheapest way to read the harness's own error.

#### Harness

The measurement runs ten **rounds**; each round is three runs, and each run starts with its
own server process. Running the three arms back to back keeps a shift in machine load from
loading onto a single arm.

```js
// library/server.mjs — loan endpoint. The candidate version (b) also adds the overdue
// book count to the response, meaning it runs one more query per request. Arguments: <port> <version a|b>
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 80000, RECENT = 40, OVERDUE = 30;   // catalog size, list length, overdue-day cutoff
const port = Number(process.argv[2]), version = process.argv[3];

if (Number.isInteger(port) === false || ["a", "b"].includes(version) === false) {
  console.log("usage: node library/server.mjs <port> <version a|b>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const recentStmt = db.prepare("SELECT id, book_no, day FROM loan WHERE member_no = ? LIMIT ?");
  const overdueStmt = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ? AND day > ?");
  let requests = 0, queries = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ requests, queries })); return; }
    requests += 1;
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n, list = recentStmt.all(member, RECENT); queries += 2;
    let overdue = -1;
    if (version === "b") { overdue = overdueStmt.get(member, OVERDUE).n; queries += 1; }   // the candidate's extra field
    res.end(JSON.stringify({ member, loan: n, overdue, list }));
  }).listen(port);
}
```

The generator carries over from the previous lessons, with two differences: it does
**not** exclude warm-up itself, and it counts failed requests. The second is required — a
request whose connection is refused looks like it finished very fast, so if it is not
counted, a dead server looks like a good measurement.

```js
// library/measure.mjs — open-loop generator. Records latencies in send order and does not
// exclude warm-up. Arguments: <arm name> <port> <round> <target rate> <duration s>
import { Agent, request } from "node:http";

const [arm, port, round, rate, duration] = process.argv.slice(2).map((x, i) => (i === 0 ? x : Number(x)));
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
let errors = 0;

function oneRequest(ms, i) {
  const start = performance.now();
  return new Promise((done) => {
    const finish = () => { ms[i] = Math.round((performance.now() - start) * 1000) / 1000; done(); };
    const r = request({ port, path: `/loan?member=U-${i % 500}`, agent }, (y) => { y.resume(); y.on("end", finish); });
    r.on("error", () => { errors += 1; finish(); });   // a failed request looks fast, so it is counted
    r.end();
  });
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let body = "";
    y.on("data", (p) => { body += p; }); y.on("end", () => done(JSON.parse(body))); }).end();
});

if (Number.isInteger(port) === false) {
  console.log("usage: node library/measure.mjs <arm> <port> <round> <target rate> <duration>");
} else {
  const ms = [], inFlight = [];
  let debt = 0, lastT = 0, seq = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(ms, seq++)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  const m = await metrics(), elapsed = (performance.now() - start) / 1000;
  console.log(JSON.stringify({ arm, round, reached: ms.length / elapsed, errors, queries: m.queries / m.requests, ms }));
}
agent.destroy();
```

The scan picks random sets from the ten rounds and re-makes the decision; the generator is
a linear congruential one and its seed is visible.

```js
// library/compare.mjs — warm-up, variability and decision scan. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 0.05, SEED = 20260731, TRIALS = 500, WARMUP = 100, BLOCK = 100;   // NF9, visible seed
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const fmt = (x, n = 2) => x.toFixed(n);
const pad = (x, n) => String(x).padStart(n);
const percentile = (d, p) => { const t = [...d].sort((a, b) => a - b);
  return t[Math.min(t.length - 1, Math.floor((p / 100) * t.length))]; };
const ROUND = [...new Set(s.map((r) => r.round))];
const find = (t, k) => s.find((r) => r.round === t && r.arm === k);
const pool = (k, start, end) => s.filter((r) => r.arm === k).flatMap((r) => r.ms.slice(start, end));
const P = (k) => percentile(pool(k, WARMUP, 1e9), 95);

console.log(`${ROUND.length} rounds, ${Math.round(pool("baseline", 0, 1e9).length / ROUND.length)} ` +
  `requests per arm per round, ${s.reduce((a, r) => a + r.errors, 0)} failed requests. Pooled p95: baseline ${fmt(P("baseline"))} ms, control ` +
  `${fmt(P("control"))}, candidate ${fmt(P("candidate"))}; candidate/baseline ${fmt(P("candidate") / P("baseline"))}x. Queries/request: ` +
  `baseline ${find(ROUND[0], "baseline").queries}, candidate ${find(ROUND[0], "candidate").queries}.`);

console.log(`\nwarm-up: every round's matching request blocks pooled, block size ${BLOCK} requests`);
console.log("request block  baseline median  baseline p95(ms)  candidate p95(ms)");
for (let i = 0; i < 5; i += 1) {
  const x = i * BLOCK, y = i < 4 ? x + BLOCK : 1e9, d = pool("baseline", x, y);
  console.log((i < 4 ? `${x}-${x + BLOCK - 1}` : `${x}+`).padEnd(13) + pad(fmt(percentile(d, 50), 3), 17) +
    pad(fmt(percentile(d, 95)), 18) + pad(fmt(percentile(pool("candidate", x, y), 95)), 19));
}

const p95 = (t, k) => percentile(find(t, k).ms.slice(WARMUP), 95);
console.log("\nper-round reading: one number per round, first 100 requests dropped");
console.log("measure           min      median    max       spread factor");
for (const name of ["baseline p95(ms)", "control/baseline", "candidate/baseline"]) {
  const d = ROUND.map((t) => (name.includes("/") ? p95(t, name.split("/")[0]) / p95(t, "baseline") : p95(t, "baseline")))
    .sort((m, n) => m - n);
  console.log(name.padEnd(18) + pad(fmt(d[0]), 8) + pad(fmt(percentile(d, 50)), 9) + pad(fmt(d[d.length - 1]), 10) +
    pad(fmt(d[d.length - 1] / d[0]), 13));
}

let t = SEED;
const rand = () => { t = (1103515245 * t + 12345) % 2147483648; return t / 2147483648; };
const shuffle = (n) => { const a = [...Array(n).keys()];
  for (let i = n - 1; i > 0; i -= 1) { const j = Math.floor(rand() * (i + 1)); [a[i], a[j]] = [a[j], a[i]]; }
  return a; };
const median = (d, ix) => percentile(ix.map((i) => d[i]), 50);
const scan = (n, threshold, x) => {
  const series = (k) => ROUND.map((tt) => percentile(find(tt, k).ms.slice(x), 95));
  const BASE = series("baseline"), CTRL = series("control"), CAND = series("candidate");
  let falseFail = 0, falsePass = 0;
  for (let i = 0; i < TRIALS; i += 1) {
    const ix = shuffle(ROUND.length).slice(0, n), gate = median(BASE, ix) * (1 + threshold);
    if (median(CTRL, ix) > gate) falseFail += 1;      // control runs the same code as baseline: no regression
    if (median(CAND, ix) <= gate) falsePass += 1;     // the candidate runs one more query per request
  }
  return `${pad(falseFail, 3)}/${pad(falsePass, 3)}`;
};

const THRESHOLDS = [0.02, THRESHOLD, 0.10, 0.20];
console.log(`\ndecision scan: ${TRIALS} trials, seed ${SEED}; cell = false fail / false pass`);
console.log("rounds/side  window                " + THRESHOLDS.map((e) => pad(`threshold ${Math.round(e * 100)}%`, 15)).join(""));
for (const [n, x] of [[1, WARMUP], [2, WARMUP], [3, WARMUP], [5, WARMUP], [8, WARMUP], [1, 0], [8, 0]]) {
  console.log(pad(n, 7) + "  " + (x === 0 ? "with warm-up" : "first 100 dropped").padEnd(20) +
    THRESHOLDS.map((e) => pad(scan(n, e, x), 15)).join(""));
}
```

{/* run: skip */}
```bash
# measure.sh — ten rounds; each round runs three arms back to back: baseline, control
# (same code as baseline) and candidate. Each run starts its own server process.
rm -f result.jsonl
for R in $(seq 1 10); do
  for ARM in "baseline a 8855" "control a 8856" "candidate b 8857"; do
    set -- $ARM
    node library/server.mjs $3 $2 & SERVER=$!
    sleep 0.8
    node library/measure.mjs $1 $3 $R 500 1.4 >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
node library/compare.mjs result.jsonl
```

Output varies from machine to machine; a sample run:

```
10 rounds, 698 requests per arm per round, 0 failed requests. Pooled p95: baseline 2.12 ms, control 2.13, candidate 2.27; candidate/baseline 1.07x. Queries/request: baseline 2, candidate 3.

warm-up: every round's matching request blocks pooled, block size 100 requests
request block  baseline median  baseline p95(ms)  candidate p95(ms)
0-99                     0.496              1.65               1.84
100-199                  0.784              2.24               2.54
200-299                  1.022              2.27               2.45
300-399                  0.947              1.94               2.17
400+                     0.882              2.08               2.17

per-round reading: one number per round, first 100 requests dropped
measure           min      median    max       spread factor
baseline p95(ms)      1.69     2.16      2.27         1.34
control/baseline      0.92     1.00      1.31         1.43
candidate/baseline    0.71     1.10      1.43         2.03

decision scan: 500 trials, seed 20260731; cell = false fail / false pass
rounds/side  window                   threshold 2%   threshold 5%  threshold 10%  threshold 20%
      1  first 100 dropped           179/155        180/188        125/277         82/375
      2  first 100 dropped           212/ 49        168/144         90/314          0/424
      3  first 100 dropped           140/ 80         58/120          0/340          0/497
      5  first 100 dropped            79/ 20          6/ 56          0/415          0/500
      8  first 100 dropped             0/  0          0/  0          0/500          0/500
      1  with warm-up                181/150        125/191        124/279         81/376
      8  with warm-up                  0/  0          0/  0          0/500          0/500
```

#### The Warm-Up Window

**Warm-up** is a process's first requests being served at a different cost than the settled
state. The first table reads the first hundred requests' median at 0.496 ms and the
requests past the four-hundredth at 0.882 ms: the first block is **about 1.78 times
optimistic** on the median. The same block's p95 does not follow: at 1.65 ms it is close to
the last block's 2.08 ms, and the two middle blocks (2.24 and 2.27 ms) read highest — in the
tail the effect is real but has no single direction. Warm-up does two things at once: compilation and cache
fill make a handful of requests very slow, and because the generator's concurrency has not
yet built up, the remaining requests find an empty queue.

The last column matters most. In the first block the candidate/baseline ratio is
1.84/1.65, or **1.12**; in the settled window it is 2.17/2.08, or 1.04. In this run warm-up
does not flip the sign of the regression, but it does inflate it, and the two middle blocks
inflate it further (1.13 and 1.08). The window is read off the measurement for exactly this
reason — the harness drops the first hundred requests, because the table only settles from
the second block on.

#### Noise and the Single-Round Decision

The second table reads the same number ten times. The baseline's p95 spreads **1.34
times**, from 1.69 to 2.27 ms. Nothing changed in the code; the only thing that changed is
the moment the round ran.

The second row gives the harness's own error. The control arm runs **the same code** as the
baseline; the ratio is expected to be 1.00 in every round, but it ranges from 0.92 to 1.31.
The median is 1.00, so there is no systematic bias; but the spread is 1.43 times, meaning
two identical versions can show what looks like a 31-percent "difference" in a single
round.

The third row carries the real difference: the rounds' candidate/baseline median is 1.10,
spread 0.71–1.43. The pooled ratio is 1.07; the rounds' median treats every round equally,
the pool weights the slower rounds more. **The smallest value is below 1:** in one round,
the candidate version — despite running one more query per request — measured **about 29
percent faster** than the baseline. A decision that looks at a single round does not just
miss the regression there, it reads it backwards.

#### How Many Rounds, Which Threshold

The third table scans two axes together: the number of rounds pooled per side, and the
threshold. Each cell counts the control arm's false fails and the candidate arm's false
passes.

The two-percent threshold needs all eight rounds to clear: 179 false fails at one round,
zero only at eight. The reason is spread, not median: the control arm's readings climbing as
high as 1.31 push even a pooled median over the gate. A threshold cannot go below the spread
the same code produces against itself.

NF9's five percent sits above this lower bound and is defensible — but not at a single
round. At one round there are 180 false fails and 188 false passes; at three rounds
58/120, at five rounds 6/56, at eight rounds **0/0**. This is the number the lesson is
after: the five-percent gate carries no decision until eight rounds are pooled per side.

The upper bound comes from the last column. At twenty percent, false fail is zero, but
false pass is 500 out of 500 even with eight rounds pooled: since the real difference is
only about 1.07 to 1.24 times, the gate sits well above it and the regression always
ships. The threshold has to stay clearly below the size of the difference it is meant to
catch.

The warm-up rows give the table a second time. At eight rounds and five percent both
windows read 0/0; at a single round, taking warm-up in raises the false pass from 188 to
191. The warm-up decision only turns the outcome while the round count is low, and it
always turns it the same way: it hides the regression.

#### Cost and the Class It Misses

The run-dependent side of the cost depends on the environment: thirty runs finished in under
seventy seconds on this machine. The run-independent side is countable — thirty server processes,
an 80,000-row fill per process (2.4 million rows total), zero failed requests, and 28,000 re-made
decisions in the scan table. The number of requests sent (20,940 in this run) is run-dependent. The
version difference is also tied to a run-independent quantity: two queries per request against
three.

This measurement misses two classes. First, the scan counts **random** noise, not
**systematic bias**. The control arm's median came out at 1.00, so no bias is visible here;
if it had appeared, adding rounds would not have shrunk it, because averaging only removes
the random component. What would tell the two apart is not this harness but rotating the
arm order between rounds.

Second is the threshold's shape. A proportional gate lets large absolute regressions
through on an endpoint whose baseline is slow: five percent here is about 0.11 ms, but if
the baseline were 200 ms it would be 10 ms. The gate therefore has to carry the first
lesson's absolute threshold too, not just a ratio.

#### Summary

- Noise is measured, not assumed: the baseline's p95 spread 1.34 times across ten rounds,
  and two identical arms' ratio ranged from 0.92 to 1.31.
- The warm-up window is read with request blocks: the first hundred requests are about
  1.78 times optimistic on the median; its net effect on the candidate/baseline ratio is
  to read 1.12 in the first block against 1.04 once settled.
- A decision that looks at a single round can mislead: in one of ten rounds
  candidate/baseline measured 0.71, and a version running one extra query per request
  looked faster than the baseline.
- The threshold's lower bound is the harness's own spread: the two-percent gate needs all
  eight rounds pooled to reach zero false fails; NF9's five percent also only reaches 0/0
  at eight rounds.
- The threshold's upper bound is the real difference: the twenty-percent gate lets the
  roughly 1.07–1.24 times regression through in every trial, even with eight rounds
  pooled.

#### Next Step

Performance testing's five lessons built a threshold, modeled the scenario, chose the metric,
attributed the result to a layer, and put the comparison under discipline. The same assumption
sits under all five: the requests were valid, the intent good. The load generator quickly did
what a real reader could; no run ever asked what a client that deliberately abuses the system
would find. The next lesson takes on that question where it is cheapest, in the program's source
text, and again measures the test itself: how many defects are found, how many warnings turn out
empty, which defect never shows up at any threshold.

## Security and Accessibility Testing

Testing against attack and unusability: the classes source-code scanning catches and produces false positives on, the coverage limit of dynamic testing that exercises the running application, the scan that searches for known vulnerabilities in the dependency tree, penetration testing's place where automation ends, the criterion share automatic auditing covers in accessibility against the area manual verification demands, and turning the rendering-engine, device, and platform matrix into compatibility testing.

### Static Application Security Testing

Source: https://academia.sh/en/courses/non-functional-testing/static-application-security-testing

Measuring an audit that scans source code without running it: the false positive and false negative counts a three-rule scanner produces on a known set of flawed files, how threshold scanning picks a different threshold under two cost weights, and the defect that remains in the codebase after the gate turns green.

Every measurement up to this point assumed the system's own legitimate user. The load
generator sent valid requests, the scenario imitated a realistic reader, and bottleneck
analysis asked where a correctly working path slowed down. No measurement asked what a
client that deliberately misuses the system would find.

The cheapest place to ask this question is the source code. **Static application security
testing** looks at a program's text without running it: which calls it makes, how it builds
its queries, which constant values sit inside the code. This lesson's subject is not the
vulnerabilities themselves — their client-side counterparts were built in the Frontend
Quality course, and the risk of a concatenated query in the Advanced SQL course — it is
**measuring the scan itself**: how many defects it finds, how many warnings turn out empty,
and which defect never shows up at any threshold.

#### The Codebase and Known Defects

Six files of the lending system will be scanned. The location of each defect has been
manually verified; the scanner's result will be compared against this list.

**NF7 (assumption):** the manually verified defect list is complete. A line absent from the
list is counted as clean. In a real codebase this assumption does not hold; the measured
ratios are ratios relative to this list.

```js
// source.mjs — the codebase to scan for the lending system and the manually verified defect list
export const CODE = {
  "catalog.mjs": [
    `const SEARCH = "SELECT isbn, title FROM book WHERE title LIKE ?";`,
    `export const search = (db, q) => db.prepare(SEARCH).all("%" + q + "%");`,
    `export const sort = (db, field) =>`,
    `  db.prepare("SELECT isbn FROM book ORDER BY " + field).all();`,
  ],
  "loan.mjs": [
    `const DAYS = 21;`,
    `export function lend(db, memberId, isbn) {`,
    `  const s = "INSERT INTO loan(memberId, isbn, days) VALUES(?, ?, " + DAYS + ")";`,
    `  return db.prepare(s).run(memberId, isbn);`,
    `}`,
    `export const history = (db, memberId) =>`,
    "  db.prepare(`SELECT * FROM loan WHERE memberId = '${memberId}'`).all();",
  ],
  "member.mjs": [
    `const token_key = "k9TdQ2vRm7XzP4bH";`,
    `export const sign = (memberId) => memberId + ":" + token_key.length;`,
    `export const password_field = "user-password";`,
  ],
  "notification.mjs": [
    `export function format(template, data) {`,
    "  return new Function(\"v\", \"return `\" + template + \"`\")(data);",
    `}`,
    `// eval( ) call removed from this file`,
  ],
  "report.mjs": [
    `const rule = { late: "days > 21", penalty: "days * 2" };`,
    `export const apply = (name, v) => eval(rule[name]);`,
    `export const headers = ["memberId", "isbn", "days"];`,
  ],
  "config.mjs": [
    `export const secret_path = process.env.LOAN_KEY;`,
    `export const channel_password = "password123";`,
    `export const key_name = "LOAN_KEY";`,
    `export const server = "127.0.0.1";`,
  ],
};

// Manually verified defects: file, line (from 1), class.
export const ACTUAL = [
  { file: "catalog.mjs", line: 4, class: "concatenated-query" },
  { file: "loan.mjs", line: 7, class: "concatenated-query" },
  { file: "member.mjs", line: 1, class: "hardcoded-secret" },
  { file: "notification.mjs", line: 2, class: "forbidden-call" },
  { file: "report.mjs", line: 2, class: "forbidden-call" },
  { file: "config.mjs", line: 2, class: "hardcoded-secret" },
];

export const lineCount = Object.values(CODE).reduce((t, s) => t + s.length, 0);
```

The scanner consists of three rules. Each rule carries a pattern and a **score**; the score
is the scanner's own estimate of the probability that a finding is a real defect.

```js
// scanner.mjs — three-rule static scanner; every finding carries a score
const secret = /\b(\w*(?:key|password|token|secret)\w*)\s*=\s*["']([^"']{8,})["']/i;
const query = /(SELECT|INSERT|UPDATE|DELETE)[^"'`]*["'`]\s*\+\s*(\w+)/i;

export const RULE = [
  {
    name: "forbidden-call", pattern: /\beval\s*\(|\bnew\s+Function\s*\(/,
    score: (s) => (s.trimStart().startsWith("//") ? 20 : 90),
  },
  {
    name: "concatenated-query", pattern: query,
    // An all-uppercase name is a constant; a lowercase name could be an outside value.
    score: (s) => (/^[A-Z0-9_]+$/.test(s.match(query)[2]) ? 30 : 80),
  },
  {
    name: "hardcoded-secret", pattern: secret,
    // A long, mixed value is likely a generated secret; a short or uniform one is usually a name.
    score: (s) => {
      const d = s.match(secret)[2];
      return d.length >= 16 && /[a-z]/.test(d) && /[A-Z]/.test(d) && /\d/.test(d) ? 85 : 40;
    },
  },
];

export function scan(code) {
  const findings = [];
  for (const [file, lines] of Object.entries(code)) {
    lines.forEach((s, i) => {
      for (const r of RULE) if (r.pattern.test(s)) findings.push({ file, line: i + 1, class: r.name, score: r.score(s) });
    });
  }
  return findings;
}

export const comparisons = (code) =>
  Object.values(code).reduce((t, s) => t + s.length, 0) * RULE.length;
```

#### Findings

```js
// findings.mjs — the scanner's findings and each one's match against the actual defect list
import { CODE, ACTUAL, lineCount } from "./source.mjs";
import { scan, comparisons, RULE } from "./scanner.mjs";

const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));
const findings = scan(CODE).sort((a, b) => b.score - a.score);

console.log(`${Object.keys(CODE).length} files, ${lineCount} lines, ${RULE.length} rules -> ${comparisons(CODE)} comparisons`);
console.log(`${findings.length} findings, ${ACTUAL.length} manually verified defects\n`);
console.log(`${"location".padEnd(21)}${"class".padEnd(23)}score  actual`);
for (const b of findings) {
  console.log(`${key(b).padEnd(21)}${b.class.padEnd(23)}${String(b.score).padStart(5)}  ${actualSet.has(key(b)) ? "yes" : "no"}`);
}

const found = new Set(findings.map(key));
const missed = ACTUAL.filter((g) => !found.has(key(g)));
console.log(`\ndefect that appears at no score: ${missed.map((g) => `${key(g)} ${g.class}`).join(", ")}`);
```

```
6 files, 25 lines, 3 rules -> 75 comparisons
9 findings, 6 manually verified defects

location             class                  score  actual
notification.mjs:2   forbidden-call            90  yes
report.mjs:2         forbidden-call            90  yes
member.mjs:1         hardcoded-secret          85  yes
catalog.mjs:4        concatenated-query        80  yes
member.mjs:3         hardcoded-secret          40  no
config.mjs:2         hardcoded-secret          40  yes
config.mjs:3         hardcoded-secret          40  no
loan.mjs:3           concatenated-query        30  no
notification.mjs:4   forbidden-call            20  no

defect that appears at no score: loan.mjs:7 concatenated-query
```

Five of the nine findings are real, four are **false positives**. What the false positives
share is that the pattern matched but the context was innocent: a call name inside a
comment line, query text concatenated with a constant, and two variables carrying a field
name rather than a secret value. The scanner sees the line; it does not see what the line
means.

The sixth defect never appeared at any score. The query inside `loan.mjs` does not use a
concatenation operator; it places the value inside a template literal. The pattern looks
for `" +`, and no such sequence exists there. **This is the missed class: code that writes
the same risk in a different syntax.** Extending the rule is possible, but every extension
brings a new false positive; the next section counts that trade-off.

#### Threshold Scanning

Not every score is carried to the gate. A threshold is chosen, and findings below it are
listed but do not stop the release. Where does the threshold come from?

**NF8 (assumption):** the cost of a missed defect is five times the cost of reviewing a
false positive. This figure comes not from a measurement but from an acceptance, and
throughout the lesson the weight is called `w`. The threshold is derived directly from this
weight.

```js
// threshold.mjs — threshold scan: false positives, false negatives and cost under two weights at every threshold
import { CODE, ACTUAL } from "./source.mjs";
import { scan } from "./scanner.mjs";

const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));
const findings = scan(CODE);

// NF8: the cost of a missed defect is WEIGHT times the cost of reviewing a false positive.
const measure = (threshold) => {
  const remaining = findings.filter((b) => b.score >= threshold);
  const falsePositive = remaining.filter((b) => !actualSet.has(key(b))).length;
  const seen = new Set(remaining.filter((b) => actualSet.has(key(b))).map(key)).size;
  return { threshold, remaining: remaining.length, falsePositive, falseNegative: ACTUAL.length - seen };
};

const THRESHOLDS = [20, 30, 40, 50, 60, 70, 80, 90];
const rows = THRESHOLDS.map(measure);
console.log(`${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=1".padStart(10)}${"cost w=5".padStart(10)}`);
for (const r of rows) {
  console.log(`${String(r.threshold).padStart(9)}${String(r.remaining).padStart(11)}${String(r.falsePositive).padStart(16)}${String(r.falseNegative).padStart(16)}${String(r.falsePositive + r.falseNegative).padStart(10)}${String(r.falsePositive + 5 * r.falseNegative).padStart(10)}`);
}

for (const w of [1, 5]) {
  const best = rows.reduce((a, b) => (b.falsePositive + w * b.falseNegative < a.falsePositive + w * a.falseNegative ? b : a));
  console.log(`w=${w} -> lowest-cost threshold ${best.threshold} (false positive ${best.falsePositive}, false negative ${best.falseNegative})`);
}
```

```
threshold  remaining  false positive  false negative  cost w=1  cost w=5
       20          9               4               1         5         9
       30          8               3               1         4         8
       40          7               2               1         3         7
       50          4               0               2         2        10
       60          4               0               2         2        10
       70          4               0               2         2        10
       80          4               0               2         2        10
       90          2               0               4         4        20
w=1 -> lowest-cost threshold 50 (false positive 0, false negative 2)
w=5 -> lowest-cost threshold 40 (false positive 2, false negative 1)
```

In the table, the **false positive** column is a clean line turning the gate red.
**False pass** is the false-negative column: a real defect passes through the gate. The two
columns move in opposite directions, and there is no outright winner between them.

The weight decides. A team that prefers two reviews over one miss keeps the threshold at
40; a team for which review is expensive raises it to 50 and knowingly lets two defects
pass through the gate. Same scanner, same code, same finding set — a different threshold.
**The threshold's source is NF8**, not a measurement; what gets measured is the threshold's
result.

#### Gate and Remediation

Threshold 40 is chosen, and the scan runs as the **quality gate** defined in the Quality
and Testing Fundamentals course. The gate turns red, four defects are fixed, and the code
is scanned again.

```js
// gate.mjs — rescanning after the fix and the quality gate running at threshold 40
import { CODE, ACTUAL } from "./source.mjs";
import { scan } from "./scanner.mjs";

const THRESHOLD = 40;
const key = (b) => `${b.file}:${b.line}`;
const actualSet = new Set(ACTUAL.map(key));

// The fixed form of four defects; loan.mjs:7 was deliberately left untouched.
const FIX = {
  "catalog.mjs": { 4: `  db.prepare("SELECT isbn FROM book " + ORDER[field]).all();` },
  "member.mjs": { 1: `const token_key = process.env.LOAN_KEY;` },
  "notification.mjs": { 2: "  return template.replace(/{(\\w+)}/g, (_, a) => data[a]);" },
  "report.mjs": { 2: `export const apply = (name, v) => RULES[name](v);` },
  "config.mjs": { 2: `export const channel_password = process.env.CHANNEL_PASSWORD;` },
};

const fixed = Object.fromEntries(Object.entries(CODE).map(([d, s]) =>
  [d, s.map((line, i) => FIX[d]?.[i + 1] ?? line)]));

const gate = (code, suppressed = []) => {
  const remaining = scan(code).filter((b) => b.score >= THRESHOLD && !suppressed.includes(key(b)));
  return { status: remaining.length === 0 ? "green" : "red", remaining };
};

for (const [label, code] of [["before the fix", CODE], ["after the fix", fixed]]) {
  const { status, remaining } = gate(code);
  console.log(`${label.padEnd(17)} threshold ${THRESHOLD} -> ${status}, ${remaining.length} findings` +
    `, ${remaining.filter((b) => !actualSet.has(key(b))).length} of them false positive`);
  for (const b of remaining) console.log(`  ${key(b).padEnd(21)}${b.class.padEnd(23)}${b.score}`);
}

const SUPPRESSED = ["member.mjs:3", "config.mjs:3"];   // reviewed, a field name; a ticket was opened
console.log(`\nsuppression list ${SUPPRESSED.length} lines -> gate ${gate(fixed, SUPPRESSED).status}`);

const remainingActual = ACTUAL.filter((g) =>
  !Object.keys(FIX[g.file] ?? {}).includes(String(g.line)));
console.log(`actual defect still in the codebase while the gate is green: ${remainingActual.length}` +
  ` (${remainingActual.map((g) => `${key(g)} ${g.class}`).join(", ")})`);
```

```
before the fix    threshold 40 -> red, 7 findings, 2 of them false positive
  catalog.mjs:4        concatenated-query     80
  member.mjs:1         hardcoded-secret       85
  member.mjs:3         hardcoded-secret       40
  notification.mjs:2   forbidden-call         90
  report.mjs:2         forbidden-call         90
  config.mjs:2         hardcoded-secret       40
  config.mjs:3         hardcoded-secret       40
after the fix     threshold 40 -> red, 2 findings, 2 of them false positive
  member.mjs:3         hardcoded-secret       40
  config.mjs:3         hardcoded-secret       40

suppression list 2 lines -> gate green
actual defect still in the codebase while the gate is green: 1 (loan.mjs:7 concatenated-query)
```

The catalog fix reads the sort field from a fixed table; the rule still matches, but
because its operand is now an uppercase constant the score drops from 80 to 30 and falls
below the threshold. After the fix the gate is still red, and both of the two remaining
findings are false positives. This is where static scanning spends the most time: to turn
the gate green, either the rule is narrowed or each finding is reviewed one by one and
written to the **suppression list**. Once the list grows by two lines the gate turns green
— and a real defect still sits in the codebase.

**Who owns the decision:** a red gate stops the release, and every suppressed line opens a
review record. The suppression list grows over time, and when it is not revisited it
silently narrows the area the scan actually watches.

#### The Cost of Scanning

The run-independent cost lives in three numbers. The first is the scan itself: 25 lines, 3
rules, 75 comparisons; zero processes start, zero requests go out, zero test data is
prepared. This is why the scan can run on every change.

The second is review: at threshold 40, 7 findings need a human eye, and 2 of them turn out
empty. As the codebase grows this number grows with the line count, not with the defect
count.

The third is maintenance: the three rules and the two-line suppression list were written by
hand. When a new syntax comes into use, the rule stays silent — `loan.mjs:7` is the proof
of this — and a rule that stays silent is indistinguishable from a rule that does not
exist.

#### Summary

- Static application security testing scans the source without running it; in this lesson
  a three-rule scanner produced 9 findings over 6 files and 25 lines, 5 of them real
  defects.
- The false positives share a common cause, context blindness: a comment line, a query
concatenated with a constant, and variables carrying a field name rather than a secret value all
  tripped the pattern.
- Threshold scanning yields two columns together: at threshold 40, 2 false positives and 1 false negative; at threshold 50, 0 false positives and 2 false negatives. The choice derives from
  NF8's weight.
- The caught class is a flawed line that matches a pattern; the missed class is code that
  writes the same risk in a different syntax — the query built with a template literal
  never appeared at any threshold.
- The cost: 75 comparisons, zero processes, zero requests; in exchange, 7 findings manually
  reviewed at threshold 40 and the upkeep of a growing suppression list.
- When the gate turned green, a real defect still sat in the codebase; a green gate does
  not mean flawless code, only that the rules have gone quiet.

#### Next Step

Everything this scan saw was read from text: which call was written, how the query was
built, which value was embedded in the code. Text, however, states what the program
writes, not what it does. The call inside `report.mjs` could sit on a branch that is never
reached; the value inside `config.mjs` could be overridden in production by an environment
variable. The reverse also holds: in a system where two files are each clean on their own,
the combination of their endpoints can meet a request with a completely unexpected
response. The next lesson actually brings the lending service up, sends requests to its
endpoints, and measures a single question: how much of the endpoints can a test that
exercises the running application reach, and where do the defect classes the two methods
see diverge.

### Dynamic Application Security Testing

Source: https://academia.sh/en/courses/non-functional-testing/dynamic-application-security-testing

Measuring a test that sends requests to a running service: the share of endpoints reached through link discovery and a route table, threshold scanning applied to the scored findings the probes produce, and the empty intersection between the defect classes static and dynamic testing see.

Static scanning read what the code writes, not what it does. A call sitting on a line
could be on a branch that is never reached; a value embedded in the code could be
overridden in production by an environment variable. The reverse also holds: in a system
where two files are each clean on their own, the combination of their endpoints can meet a
request with a completely unexpected response.

**Dynamic application security testing** therefore looks at the running process rather
than the source: it sends requests to endpoints, reads the responses, and treats the
response itself as evidence. This lesson measures three things: how much of the endpoints
this test can reach, where the threshold that turns evidence into a finding sits, and where
the defect classes the two methods see diverge.

#### The Service Under Test

The lending service has eight endpoints. Five defects have been manually verified; the
results will be compared against this list.

```js
// service.mjs — the lending service's eight endpoints; start() returns a listening server
import { createServer } from "node:http";

const BOOK = { "978-1": "The Hourglass", "978-2": "Still on the Shelf" };
const MEMBER = { 1: { name: "A. Fields", penalty: 0 }, 2: { name: "B. Marsh", penalty: 12 } };

export const ENDPOINTS = ["/", "/catalog", "/book", "/status", "/member", "/loan", "/report", "/admin/log"];

// Manually verified defects (endpoint | class).
export const ACTUAL = [
  "/book|input-echo", "/book|empty-input-detail", "/member|unauthenticated-access",
  "/loan|unauthenticated-accept", "/admin/log|internal-detail",
];

export function start() {
  const s = createServer((req, res) => {
    const u = new URL(req.url, "http://x");
    const send = (code, body) => res.writeHead(code, { "content-type": "application/json" })
      .end(JSON.stringify(body));

    if (u.pathname === "/") return send(200, { links: ["/catalog", "/status"] });
    if (u.pathname === "/catalog")
      return send(200, { book: Object.keys(BOOK).map((i) => `/book?isbn=${i}`) });
    if (u.pathname === "/book") {
      const isbn = u.searchParams.get("isbn") ?? "";
      if (isbn === "")                                          // branch that opens only on empty input
        return send(400, { error: "isbn empty", query: "SELECT title FROM book WHERE isbn = ''" });
      return BOOK[isbn] ? send(200, { isbn, title: BOOK[isbn] })
        : send(404, { error: `no such record: ${isbn}` });        // input is echoed back as-is
    }
    if (u.pathname === "/status") return send(200, { status: "open", queue: 0 });
    if (u.pathname.startsWith("/member/")) {
      const no = u.pathname.slice(8);
      return MEMBER[no] ? send(200, { no, ...MEMBER[no] }) : send(404, { error: "no such member" });
    }
    if (u.pathname === "/loan" && req.method === "POST") return send(202, { accepted: true });
    if (u.pathname === "/report") return send(200, { rows: Object.keys(MEMBER).length });
    if (u.pathname === "/admin/log")
      return send(500, { error: "could not read log", path: "/srv/loan/var/log/loan.log" });
    return send(404, { error: "no such path" });
  });
  return s;
}

export const listen = (s) => new Promise((c) => s.listen(0, "127.0.0.1", () => c(s.address().port)));
```

#### Endpoint Coverage

Dynamic testing's first limit is the surface it can reach. A discovery run that starts
only from the root endpoint and follows the links found in responses gives a different
result on the same service than a run given a route table extracted from the source.

```js
// discovery.mjs — the share of endpoints dynamic testing can reach: by links alone, and with a route table
import { start, listen, ENDPOINTS, ACTUAL } from "./service.mjs";

const s = start();
const p = await listen(s);
const base = `http://127.0.0.1:${p}`;
const path = (u) => (u.split("?")[0].startsWith("/member/") ? "/member" : u.split("?")[0]);

// (a) Discovery that starts only from "/" and follows the links found in responses.
const seen = new Set(), queue = ["/"];
let requests = 0;
while (queue.length) {
  const u = queue.shift();
  if (seen.has(path(u))) continue;
  seen.add(path(u));
  const body = await (await fetch(base + u)).text();
  requests += 1;
  for (const m of body.matchAll(/"(\/[^"]*)"/g)) if (!seen.has(path(m[1]))) queue.push(m[1]);
}
console.log(`link-following discovery : ${requests} requests, ${seen.size}/${ENDPOINTS.length} endpoints` +
  ` = ${((100 * seen.size) / ENDPOINTS.length).toFixed(0)}%`);
const outside = ENDPOINTS.filter((u) => !seen.has(u));
console.log(`  unreachable            : ${outside.join(", ")}`);
console.log(`  defects sitting on these endpoints: ${ACTUAL.filter((g) => outside.includes(g.split("|")[0])).length}/${ACTUAL.length}`);

// (b) Given a route table extracted from the source, the same run touches every endpoint.
const routes = new Set();
for (const u of ENDPOINTS) {
  const target = u === "/member" ? "/member/1" : u;
  const y = await fetch(base + target, u === "/loan" ? { method: "POST", body: "{}" } : undefined);
  if (y.status !== 404 || u === "/book") routes.add(u);
  await y.text();
}
console.log(`route-table discovery    : ${ENDPOINTS.length} requests, ${routes.size}/${ENDPOINTS.length} endpoints` +
  ` = ${((100 * routes.size) / ENDPOINTS.length).toFixed(0)}%`);
s.close();
```

```
link-following discovery : 4 requests, 4/8 endpoints = 50%
  unreachable            : /member, /loan, /report, /admin/log
  defects sitting on these endpoints: 3/5
route-table discovery    : 8 requests, 8/8 endpoints = 100%
```

Discovery that follows links saw half the endpoints, and the half it saw held only two of
the defects. The four it could not reach are mentioned in no response: one requires a
request body, one sits on the administrative side, and two never appear in any link list.
**The coverage threshold's source here is a measurement:** because the defect count sitting
on uncovered endpoints comes out to 3/5, the rule "a run is not accepted unless it touches
every endpoint" becomes defensible.

The way to meet this threshold is telling: the route table is extracted **from the
source**. What pushes dynamic testing's coverage to the ceiling is a static artifact; the
two methods feed each other here.

#### Probes and Threshold Scanning

A single request goes to each endpoint. The response is scored against three rules: does
the harmless marker sent come back verbatim in the body, does a personal field come back
without any identity declared, does an internal detail appear in the body. The scores mean
the same thing as in the previous lesson; the weight comes from **NF8** and continues as
`w = 5`.

```js
// probe.mjs — one-request probes for every endpoint, scored findings and threshold scanning
import { start, listen, ENDPOINTS, ACTUAL } from "./service.mjs";

const MARKER = "ISBN-CHECK-7391";                // harmless marker: searched for in the response
const PERSONAL = ["name", "penalty"], COUNTER = ["queue", "rows"];

const s = start();
const base = `http://127.0.0.1:${await listen(s)}`;
const findings = [];
for (const endpoint of ENDPOINTS) {
  const post = endpoint === "/loan";
  const target = endpoint === "/member" ? "/member/2" : `${endpoint}?isbn=${MARKER}`;
  const y = await fetch(base + target, post ? { method: "POST", body: "{}" } : undefined);
  const body = await y.text();
  if (body.includes(MARKER)) findings.push({ endpoint, class: "input-echo", score: 80 });
  if (y.status < 300 && PERSONAL.some((a) => body.includes(`"${a}"`))) findings.push({ endpoint, class: "unauthenticated-access", score: 90 });
  if (y.status < 300 && post) findings.push({ endpoint, class: "unauthenticated-accept", score: 50 });
  if (/"\/(srv|var|home)\//.test(body)) findings.push({ endpoint, class: "internal-detail", score: 85 });
  else if (COUNTER.some((a) => body.includes(`"${a}"`))) findings.push({ endpoint, class: "internal-counter", score: 40 });
}
s.close();

const key = (b) => `${b.endpoint}|${b.class}`;
console.log(`${ENDPOINTS.length} endpoints, ${ENDPOINTS.length} requests, ${findings.length} findings, ${ACTUAL.length} manually verified defects\n`);
for (const b of findings.sort((a, c) => c.score - a.score)) {
  console.log(`  ${b.endpoint.padEnd(17)}${b.class.padEnd(24)}${String(b.score).padStart(3)}  actual: ${ACTUAL.includes(key(b)) ? "yes" : "no"}`);
}

console.log(`\n${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=5".padStart(10)}`);
for (const e of [30, 40, 50, 60, 80, 90]) {
  const k = findings.filter((b) => b.score >= e);
  const fp = k.filter((b) => !ACTUAL.includes(key(b))).length;
  const falseNegative = ACTUAL.length - k.filter((b) => ACTUAL.includes(key(b))).length;
  console.log(`${String(e).padStart(9)}${String(k.length).padStart(11)}${String(fp).padStart(16)}${String(falseNegative).padStart(16)}${String(fp + 5 * falseNegative).padStart(10)}`);
}
console.log(`never appears at any threshold: ${ACTUAL.filter((g) => !findings.some((b) => key(b) === g)).join(", ")}`);
```

```
8 endpoints, 8 requests, 6 findings, 5 manually verified defects

  /member          unauthenticated-access   90  actual: yes
  /admin/log       internal-detail          85  actual: yes
  /book            input-echo               80  actual: yes
  /loan            unauthenticated-accept   50  actual: yes
  /status          internal-counter         40  actual: no
  /report          internal-counter         40  actual: no

threshold  remaining  false positive  false negative  cost w=5
       30          6               2               1         7
       40          6               2               1         7
       50          4               0               1         5
       60          3               0               2        10
       80          3               0               2        10
       90          1               0               4        20
never appears at any threshold: /book|empty-input-detail
```

The lowest cost falls at threshold 50: two false positives drop out, four real defects
remain. What differs from the previous lesson is that the findings cluster **discretely**
along the score axis. In static scanning, real and innocent findings sat in the same band;
here the two false positives sit at 40, and the four real defects sit at 50 and above. The
reason is the method's nature: the evidence is the response itself, not a guess at intent.

**The caught classes** are unauthenticated access, unauthenticated accept, input echo, and
internal detail leakage. All four are visible only while running. **The missed class** is
on the last line: the branch that opens only on empty input never appeared at any
threshold, because the probe never sent that input. Dynamic testing sees the paths its
input opens; the input it never sends stays silent in the codebase.

#### Where the Two Methods Diverge

What comes out if the previous lesson's three static rules are applied to the same
service?

```js
// divergence.mjs — applying static rules to the same service and the class set each method sees
import { readFileSync } from "node:fs";

const secret = /\b(\w*(?:key|password|token|secret)\w*)\s*=\s*["'][^"']{8,}["']/i;
const RULE = [
  ["forbidden-call", /\beval\s*\(|\bnew\s+Function\s*\(/],
  ["concatenated-query", /(SELECT|INSERT|UPDATE|DELETE)[^"'`]*["'`]\s*\+\s*\w+/i],
  ["hardcoded-secret", secret],
];

const lines = readFileSync(new URL("./service.mjs", import.meta.url), "utf8").split("\n");
const staticFindings = lines.flatMap((s, i) =>
  RULE.filter(([, d]) => d.test(s)).map(([name]) => `service.mjs:${i + 1} ${name}`));
console.log(`service.mjs: ${lines.length} lines, ${RULE.length} rules -> ${staticFindings.length} static findings`);

// NF9: each defect class's visibility to the two methods was classified by hand.
const CLASS = [                                  // [class, static sees, dynamic sees]
  ["concatenated-query", true, false],
  ["hardcoded-secret", true, false],
  ["forbidden-call", true, false],
  ["input-echo", false, true],
  ["unauthenticated-access", false, true],
  ["unauthenticated-accept", false, true],
  ["internal-detail", false, true],
  ["template-literal-query", false, false],
  ["empty-input-detail", false, false],
];

const count = (s, d) => CLASS.filter(([, a, b]) => a === s && b === d).length;
console.log(`\n${CLASS.length} defect classes:`);
console.log(`  static only  : ${count(true, false)}  (${CLASS.filter(([, a, b]) => a && !b).map(([n]) => n).join(", ")})`);
console.log(`  dynamic only : ${count(false, true)}  (${CLASS.filter(([, a, b]) => !a && b).map(([n]) => n).join(", ")})`);
console.log(`  both         : ${count(true, true)}`);
console.log(`  neither      : ${count(false, false)}  (${CLASS.filter(([, a, b]) => !a && !b).map(([n]) => n).join(", ")})`);
const union = CLASS.filter(([, a, b]) => a || b).length;
console.log(`combined coverage: ${union}/${CLASS.length} = ${((100 * union) / CLASS.length).toFixed(0)}%`);
```

```
service.mjs: 46 lines, 3 rules -> 0 static findings

9 defect classes:
  static only  : 3  (concatenated-query, hardcoded-secret, forbidden-call)
  dynamic only : 4  (input-echo, unauthenticated-access, unauthenticated-accept, internal-detail)
  both         : 0
  neither      : 2  (template-literal-query, empty-input-detail)
combined coverage: 7/9 = 78%
```

The static rules produced not a single finding in this file; the dynamic probes found four
real defects in the same file. The reverse direction is just as sharp: the embedded value
and the concatenated query in the previous lesson's codebase are invisible to any request
made here. In this setup the two methods' **intersection is empty**, and the combined
coverage is seven of nine classes. As the rule sets grow the intersection grows too, but
neither substitutes for the other.

**Who owns the decision:** a finding above threshold 50 stops the release; a run that fails
to meet the coverage threshold reports no result and runs again. A dynamic testing report
that does not state its coverage hides how many endpoints were never tried.

#### The Cost of Testing

The run-independent cost lives in three places. The first is request count: 8 endpoints, 8
requests per run, and one process staying up. Static scanning ran with zero processes;
dynamic testing demands the application itself, its data, and its dependencies.

The second is maintaining the route table: what raises coverage from 50% to 100% is an
eight-line list extracted from the source, and when a new endpoint is added and this list
is not updated by hand, coverage silently drops.

The third is the input set: probes see only the branches the value they send opens. The
empty-input branch was missed because a single value was missing. Growing the input set
multiplies the request count by the endpoint count; coverage grows linearly while cost
grows multiplicatively.

#### Summary

- Dynamic application security testing sends requests to the running process and takes its
  evidence from the response itself; in this lesson, 8 requests to 8 endpoints produced 6
  findings, 4 of them real defects.
- Link-following discovery reached 50% of the endpoints, and 3 of the 5 manually verified
defects remained on unreachable endpoints; given a route table, coverage rose to 100%.
- Threshold scanning gave the lowest cost at 50: 0 false positives, 1 false negative. Because the
  findings cluster discretely along the score axis, choosing the threshold is easier here
  than in static scanning.
- The caught classes are the ones visible only while running; the missed class is the
  branch opened by input the probe never sends.
- Three static rules applied to the same service gave zero findings; across nine defect
  classes the two methods' intersection came out empty, and combined coverage stayed at
  78%.
- The cost: one process staying up, 8 requests per run, a hand-maintained route table, and
  a request count that grows multiplicatively as the input set grows.

#### Next Step

Both scans tested code that was **written**: one read its text, the other observed its
behavior while running. Yet most of the lines the lending service executes were not
written by this team. The catalog schema depends on a validator, the notification channel
on a queue client, the log writer on a formatter, and each of these brings its own
dependencies. In this code, searching for defects is not a matter of scanning but of
**searching for known vulnerabilities**: which defect is reported for which version of a
component is on record. The next lesson extracts the lending system's dependency tree,
matches its own advisory set against that tree, and measures three things: the share of
dependencies that were never written directly, the false positives produced by
version-range matching, and how many packages a single fix touches.

### Dependency and Component Scanning

Source: https://academia.sh/en/courses/non-functional-testing/dependency-and-component-scanning

Searching for known vulnerabilities in the dependency tree: the share of transitive dependencies, the false positives version-range matching produces, threshold scanning over severity, and how many packages a single fix touches at once because of version ranges.

Both scans tested code this team wrote. Yet most of the lines the lending service executes
were not written by this team: the catalog schema depends on a validator, the notification
channel on a queue client, the log writer on a formatter, and each of these brings its own
dependencies.

In this code, searching for defects is not a matter of scanning but of searching for
**known vulnerabilities**: which defect a component carries in which version range is on
record, and scanning matches these records against the installed versions. The method is
cheap, but it is wrong in three places — this lesson counts them.

#### The Dependency Tree and the Lock File

The lending system has four direct dependencies. The lock file pins the installed
versions to exact numbers; scanning looks at these versions, not at the declared ranges.

```js
// tree.mjs — the lending system's dependency tree, lock file, and version-range check
export const ROOT = {
  "schema-validator": "^2.1.0", "queue-client": "^1.4.0",
  "log-formatter": "^3.0.0", "identity-signer": "^1.2.0",
};

// Each package version's own declared ranges.
export const REGISTRY = {
  "schema-validator": {
    "2.1.4": { "pattern-compiler": "^1.0.0", "text-normalizer": "^0.9.0" },
    "2.2.0": { "pattern-compiler": "^2.0.0", "text-normalizer": "^0.9.0" },
  },
  "queue-client": {
    "1.4.2": { "frame-decoder": "^4.2.0", "retry": "^1.1.0" },
    "1.5.0": { "frame-decoder": "^4.3.0", "retry": "^1.2.0" },
  },
  "log-formatter": { "3.0.1": { "text-normalizer": "^0.9.0", "timestamp": "^2.0.0" } },
  "identity-signer": { "1.2.0": { "digest": "^3.1.0" } },
  "pattern-compiler": { "1.0.7": {}, "2.0.0": {} },
  "text-normalizer": { "0.9.8": {} },
  "frame-decoder": { "4.2.1": {}, "4.3.0": {} },
  "retry": { "1.1.0": {}, "1.2.0": {} },
  "timestamp": { "2.0.4": {}, "2.1.0": {} },
  "digest": { "3.1.2": {} },
};

// Lock file: installed versions.
export const LOCK = {
  "schema-validator": "2.1.4", "queue-client": "1.4.2", "log-formatter": "3.0.1",
  "identity-signer": "1.2.0", "pattern-compiler": "1.0.7", "text-normalizer": "0.9.8",
  "frame-decoder": "4.2.1", "retry": "1.1.0", "timestamp": "2.0.4",
  "digest": "3.1.2",
};

export const part = (v) => v.split(".").map(Number);
export const less = (a, b) => {
  const [x, y] = [part(a), part(b)];
  for (let i = 0; i < 3; i += 1) if (x[i] !== y[i]) return x[i] < y[i];
  return false;
};
// "^a.b.c": the first non-zero digit stays fixed. "<a.b.c": any version below it.
export function satisfies(version, range) {
  if (range.startsWith("<")) return less(version, range.slice(1));
  const t = part(range.slice(1)), s = part(version);
  if (less(version, range.slice(1))) return false;
  const i = t.findIndex((x) => x > 0);
  return s.slice(0, i + 1).join(".") === t.slice(0, i + 1).join(".");
}

// Walks locked versions to find every package's depth.
export function tree(root = ROOT, lock = LOCK) {
  const depth = {}, queue = Object.keys(root).map((a) => [a, 1]);
  while (queue.length) {
    const [name, d] = queue.shift();
    if (depth[name] !== undefined && depth[name] <= d) continue;
    depth[name] = d;
    for (const child of Object.keys(REGISTRY[name][lock[name]])) queue.push([child, d + 1]);
  }
  return depth;
}
```

#### Matching and False Positives

The advisory set is our own records: each advisory carries a package, an affected version
range, a severity score, and the fixed version. Scanning checks whether the locked version
falls inside the affected range — a version comparison a little more involved than a text
comparison, and it uses no other information.

**NF10 (assumption):** the real risk is the flawed code actually being **called** in this
system. A package can be in the affected range while its flawed function is never called
in this system; the manually verified risk list makes this distinction, scanning cannot.

```js
// scan.mjs — matching the advisory set against the lock file, false positives, and threshold scanning
import { LOCK, satisfies, tree } from "./tree.mjs";

const depth = tree();
const direct = Object.keys(depth).filter((a) => depth[a] === 1);
const n = Object.keys(depth).length;
console.log(`${n} packages, direct ${direct.length}, transitive ${n - direct.length}` +
  ` (${((100 * (n - direct.length)) / n).toFixed(0)}%), max depth ${Math.max(...Object.values(depth))}`);

// Our own advisory set: package, affected range, severity, fixed version.
const ADVISORY = [
  { no: "K-01", package: "pattern-compiler", affected: "<2.0.0", severity: 80, fixed: "2.0.0" },
  { no: "K-02", package: "text-normalizer", affected: "<0.9.8", severity: 60, fixed: "0.9.8" },
  { no: "K-03", package: "frame-decoder", affected: "<4.3.0", severity: 70, fixed: "4.3.0" },
  { no: "K-04", package: "queue-client", affected: "<1.5.0", severity: 90, fixed: "1.5.0" },
  { no: "K-05", package: "timestamp", affected: "<2.1.0", severity: 30, fixed: "2.1.0" },
];

// NF10: the manually verified real risk is the flawed code actually being called in this system.
const ACTUAL = ["pattern-compiler", "queue-client", "retry"];

const matches = ADVISORY.filter((k) => satisfies(LOCK[k.package], k.affected));
console.log(`${ADVISORY.length} advisories, ${matches.length} matches, ${ACTUAL.length} manually verified risks\n`);
console.log(`${"advisory".padEnd(10)}${"package".padEnd(19)}${"installed".padEnd(11)}${"affected".padEnd(11)}severity  depth  actual`);
for (const k of ADVISORY) {
  const e = satisfies(LOCK[k.package], k.affected);
  console.log(`${k.no.padEnd(10)}${k.package.padEnd(19)}${LOCK[k.package].padEnd(11)}${k.affected.padEnd(11)}` +
    `${String(k.severity).padStart(8)}${String(depth[k.package]).padStart(7)}  ${e ? (ACTUAL.includes(k.package) ? "yes" : "no") : "-"}`);
}

console.log(`\n${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=5".padStart(10)}`);
for (const e of [20, 30, 50, 70, 80, 90]) {
  const k = matches.filter((b) => b.severity >= e);
  const fp = k.filter((b) => !ACTUAL.includes(b.package)).length;
  const fn = ACTUAL.length - k.filter((b) => ACTUAL.includes(b.package)).length;
  console.log(`${String(e).padStart(9)}${String(k.length).padStart(11)}${String(fp).padStart(16)}${String(fn).padStart(16)}${String(fp + 5 * fn).padStart(10)}`);
}
console.log(`actual risk that never appears in the advisory set: ${ACTUAL.filter((g) => !ADVISORY.some((k) => k.package === g)).join(", ")}`);
```

```
10 packages, direct 4, transitive 6 (60%), max depth 2
5 advisories, 4 matches, 3 manually verified risks

advisory  package            installed  affected   severity  depth  actual
K-01      pattern-compiler   1.0.7      <2.0.0           80      2  yes
K-02      text-normalizer    0.9.8      <0.9.8           60      2  -
K-03      frame-decoder      4.2.1      <4.3.0           70      2  no
K-04      queue-client       1.4.2      <1.5.0           90      1  yes
K-05      timestamp          2.0.4      <2.1.0           30      2  no

threshold  remaining  false positive  false negative  cost w=5
       20          4               2               1         7
       30          4               2               1         7
       50          3               1               1         6
       70          3               1               1         6
       80          2               0               1         5
       90          1               0               2        10
actual risk that never appears in the advisory set: retry
```

Three numbers are worth reading. The first is **the share of transitive dependencies**:
six of the ten packages were declared nowhere; they came in underneath the four that were
declared. Without the lock file, it would not even be known which version these six
packages are at, and scanning cannot work without knowing the version.

The second is **false positives**: two of the four matches are not real risk. Both are
correct from a version-comparison standpoint — the installed version is in the affected
range — but the flawed code path does not run in this system. Version-range matching
measures a package's presence, not its use.

The third is **false negative**: the defect inside `retry` is on the manually verified
risk list, but not in the advisory set. This is dependency scanning's structural limit — it
finds only **reported** defects, and no threshold can see a defect that has not been
reported.

**The threshold's source is a measurement:** threshold scanning shows that both false
positives drop out at severity 80, and both of the two remaining findings turn out real;
cost falls to its lowest value there. The weight again comes from NF8.

#### How a Fix Propagates

The two advisories that clear the threshold will be fixed. A fix looks like changing a
single version number; version ranges get in the way of that.

```js
// fix.mjs — how many packages fixing one advisory touches
import { REGISTRY, ROOT, LOCK, satisfies, less, tree } from "./tree.mjs";

const versions = (name) => Object.keys(REGISTRY[name]).sort((a, b) => (less(a, b) ? -1 : 1));
const lowestSatisfying = (name, range) => versions(name).find((v) => satisfies(v, range));
const parents = (name, lock) => Object.keys(REGISTRY).filter((u) => REGISTRY[u][lock[u]]?.[name]);

function upgrade(lock, package_, target) {
  const next = { ...lock }, touched = new Set(), blocked = [];
  const queue = [[package_, target]];
  while (queue.length) {
    const [name, s] = queue.shift();
    if (next[name] === s) continue;
    next[name] = s;
    touched.add(name);
    for (const [child, range] of Object.entries(REGISTRY[name][s]))
      if (!satisfies(next[child], range)) queue.push([child, lowestSatisfying(child, range)]);
    for (const u of parents(name, next)) {
      if (satisfies(s, REGISTRY[u][next[u]][name])) continue;
      const suitable = versions(u).find((v) => REGISTRY[u][v][name] && satisfies(s, REGISTRY[u][v][name]));
      suitable ? queue.push([u, suitable]) : blocked.push(`no suitable version for ${u}`);
    }
    if (ROOT[name] && !satisfies(s, ROOT[name])) blocked.push(`root range ${ROOT[name]} does not accept ${name}@${s}`);
  }
  return { next, touched: [...touched], blocked };
}

let combined = LOCK;
for (const [no, package_, target] of [["K-01", "pattern-compiler", "2.0.0"], ["K-04", "queue-client", "1.5.0"]]) {
  const { next, touched, blocked } = upgrade(LOCK, package_, target);
  console.log(`${no}: ${package_} ${LOCK[package_]} -> ${target}; touches ${touched.length} packages`);
  for (const a of touched) console.log(`   ${a.padEnd(23)}${LOCK[a]} -> ${next[a]}`);
  if (blocked.length) console.log(`   blocked: ${blocked.join("; ")}`);
  combined = upgrade(combined, package_, target).next;
}

const changed = Object.keys(LOCK).filter((a) => combined[a] !== LOCK[a]);
const n = Object.keys(LOCK).length;
console.log(`\nboth fixes together: ${changed.length}/${n} packages (${((100 * changed.length) / n).toFixed(0)}%) moved to a new version`);
console.log(`tree depth unchanged: ${Math.max(...Object.values(tree(ROOT, combined)))}`);
console.log(`side effect: K-03's target frame-decoder ${LOCK["frame-decoder"]} -> ${combined["frame-decoder"]}`);
```

```
K-01: pattern-compiler 1.0.7 -> 2.0.0; touches 2 packages
   pattern-compiler       1.0.7 -> 2.0.0
   schema-validator       2.1.4 -> 2.2.0
K-04: queue-client 1.4.2 -> 1.5.0; touches 3 packages
   queue-client           1.4.2 -> 1.5.0
   frame-decoder          4.2.1 -> 4.3.0
   retry                  1.1.0 -> 1.2.0

both fixes together: 5/10 packages (50%) moved to a new version
tree depth unchanged: 2
side effect: K-03's target frame-decoder 4.2.1 -> 4.3.0
```

The first fix propagated upward. Because `pattern-compiler`'s fixed version raises the
major digit, the range the package above it declares no longer accepts it, so the
intermediate package has to be upgraded too. The second fix propagated downward: the new
version wants newer versions of its own dependencies. The two fixes together moved five of
the ten packages.

This number is the scan's real cost. The new version of five packages also brings behavior
changes that have nothing to do with the security advisory; the regression testing that
follows an upgrade is unavoidable, and its cost is measured not by the finding count but by
**the number of packages touched**. The last line also shows the reverse: an advisory
counted as a false positive closed for free, as the side effect of a different fix.

**Who owns the decision:** a match above severity 80 stops the release and opens an
upgrade task; matches below it accumulate on a list. Because the advisory set can change
every day, the same lock file can be green today and red tomorrow — this is the one check
whose result changes without the code changing.

#### Summary

- Dependency scanning matches the versions in the lock file against the affected ranges in
  the advisory set; in this tree, six of ten packages (60%) are transitive dependencies
  declared nowhere.
- Two of the four matches turned out to be false positives: the version matched correctly,
  but the flawed code path is not called in this system. Version-range matching measures
  presence, not use.
- The caught class is a reported defect found in the installed version; the missed class
is a defect not yet reported — one of the three manually verified risks appeared in no advisory.
- Threshold scanning gave severity 80: 0 false positives, 1 false negative. The threshold's source
  is this measurement; the weight's source is NF8.
- The cost is the number of packages touched: the two fixes propagated up and down through
  the version ranges, moved five of the ten packages to a new version, and each one
  demands a regression test.
- This scan is the one check whose result can change without the code ever changing; as
  the advisory set grows, the same lock file turns red again.

#### Next Step

All three methods looked at a rule set: static scanning at patterns, dynamic testing at
response signatures, dependency scanning at advisories. What the three share is the
assumption that what is being searched for is **named in advance**. The lending system's
most expensive defect, however, is on no list: a reader whose membership has lapsed being
able to move a penalty record onto another member is a sequence of requests, each
individually authorized, that only becomes visible when viewed together. No pattern names
this, because the defect lives not in the code but in the combination of rules. The next
lesson draws this boundary: on a threat model, it separates the defect classes automation
can cover from the ones that require human judgment, and counts the share that falls
between the two.

### The Role of Penetration Testing

Source: https://academia.sh/en/courses/non-functional-testing/role-of-penetration-testing

Measuring the point where automation ends: the share of entries a threat model can decide automatically, the coverage gap between single-step and multi-step scenarios, and how a chain of individually authorized operations can only be counted as a violation by a hand-written invariant.

All three methods looked at a rule set: static scanning at patterns, dynamic testing at
response signatures, dependency scanning at advisories. What the three share is the
assumption that what is being searched for is **named in advance**. The lending system's
most expensive defect, however, is on no list, because the defect does not live in a
single line of code but in the combination of rules.

**Penetration testing** is the name for this gap: a person reading the system's rules and
combining individually legitimate operations in a sequence the system's owner never
considered. This lesson deals not with technique but with **place** — it separates the
defect classes automation can cover from the ones that require judgment, and counts the
share that falls between them.

#### Counting Coverage with a Threat Model

Asking the coverage question requires a list. A **threat model** enumerates assets, entry
points, and the threats directed at them; the list here was extracted for the lending
system, and every entry carries two fields: which source can make the decision, and how
many steps the scenario has.

**NF11 (assumption):** the threat model is complete. A threat absent from the list is
counted as nonexistent; in reality this assumption does not hold, and every coverage ratio
is relative to this list.

```js
// threat.mjs — the lending system's threat model and how coverage is spread across methods
// [threat, decision source, scenario step count]
// decision source: pattern (static), response (dynamic), advisory (dependency), judgment (human)
const THREAT = [
  ["the catalog query built by concatenation", "pattern", 1],
  ["a signing key embedded in the code", "pattern", 1],
  ["a formatter that generates code from a string", "pattern", 1],
  ["a member record returning without authentication", "response", 1],
  ["an internal path disclosed in an error response", "response", 1],
  ["a loan request accepted without authentication", "response", 1],
  ["input echoed back verbatim in the response", "response", 1],
  ["an accepted request turning into a record", "response", 2],
  ["the queue client's reported defect", "advisory", 1],
  ["the pattern compiler's reported defect", "advisory", 1],
  ["a single member derivable from the report's total", "judgment", 1],
  ["a penalty transferred through a partnership", "judgment", 4],
  ["a reader with a lapsed membership keeping their loan queue position", "judgment", 3],
  ["an extension resetting the penalty threshold", "judgment", 2],
];

const AUTOMATIC = ["pattern", "response", "advisory"];
const count = (f) => THREAT.filter(f).length;
const automatic = (t) => AUTOMATIC.includes(t[1]);

console.log(`${THREAT.length} threat entries`);
for (const k of [...AUTOMATIC, "judgment"]) {
  const n = count((t) => t[1] === k);
  console.log(`  ${k.padEnd(10)}${String(n).padStart(3)}  ${((100 * n) / THREAT.length).toFixed(0)}%`);
}
console.log(`automatically covered: ${count(automatic)}/${THREAT.length} = ${((100 * count(automatic)) / THREAT.length).toFixed(0)}%`);

for (const [name, f] of [["single-step", (t) => t[2] === 1], ["multi-step", (t) => t[2] > 1]]) {
  const group = THREAT.filter(f);
  const k = group.filter(automatic).length;
  console.log(`${name}: ${group.length} threats, ${k} automatic = ${((100 * k) / group.length).toFixed(0)}%`);
}

const manual = THREAT.filter((t) => !automatic(t));
console.log(`\n${manual.length} entries require human judgment, ${manual.reduce((s, t) => s + t[2], 0)} steps total:`);
for (const [name, , n] of manual) console.log(`  ${String(n)} steps  ${name}`);
```

```
14 threat entries
  pattern     3  21%
  response    5  36%
  advisory    2  14%
  judgment    4  29%
automatically covered: 10/14 = 71%
single-step: 10 threats, 9 automatic = 90%
multi-step: 4 threats, 1 automatic = 25%

4 entries require human judgment, 10 steps total:
  1 steps  a single member derivable from the report's total
  4 steps  a penalty transferred through a partnership
  3 steps  a reader with a lapsed membership keeping their loan queue position
  2 steps  an extension resetting the penalty threshold
```

The gap between the two ratios is this lesson's main finding. 90% of single-step threats
are settled automatically; only 25% of multi-step threats are. Automation decides **per
request**: a line, a response, a version number. The relationship between steps is never
the input to any rule.

The distinction does not end with the step count, either. The list's one single-step
judgment entry is a single request to the report endpoint; the response is flawlessly
formed, passes the authorization check, and leaks no internal detail. The defect is that
the returned total can be narrowed to a single member with a specific query — making that
call requires knowing what the data **means**.

#### Individually Authorized Steps

The most expensive judgment entry was four steps. The lending system has five operations,
and each carries its own auth condition: adding a partnership is open only to members,
transferring a penalty only while a partnership exists, leaving a partnership only to a
partner. Automatic checking tests this condition at every step, and no step is out of
bounds.

```js
// chain.mjs — the violation born from combining individually authorized steps, and the chain-length threshold
const START = { member: true, partner: false, onLoan: true, penalty: 0, partnerPenalty: 0 };

// Each operation carries its own auth condition; none is out of bounds on its own.
const OPERATION = {
  "partner-add": { auth: (d) => d.member && !d.partner, effect: (d) => ({ ...d, partner: true }) },
  "partner-remove": { auth: (d) => d.partner, effect: (d) => ({ ...d, partner: false }) },
  "penalty-transfer": { auth: (d) => d.partner && d.penalty > 0, effect: (d) => ({ ...d, penalty: 0, partnerPenalty: d.partnerPenalty + d.penalty }) },
  "return": { auth: (d) => d.onLoan, effect: (d) => ({ ...d, onLoan: false }) },
  "report-lost": { auth: (d) => d.onLoan, effect: (d) => ({ ...d, onLoan: false, penalty: d.penalty + 30 }) },
};

// Two hand-written invariants. The correct one ties partnerPenalty to the partnership being open.
const INVARIANT = {
  correct: (d) => !(d.partnerPenalty > 0 && !d.partner),
  narrow: (d) => !(d.partnerPenalty > 0),
};

// Generates every authorized operation sequence up to length k.
function sequences(k) {
  const output = [];
  const walk = (d, path) => {
    if (path.length) output.push({ path, end: d });
    if (path.length === k) return;
    for (const [name, op] of Object.entries(OPERATION)) if (op.auth(d)) walk(op.effect(d), [...path, name]);
  };
  walk(START, []);
  return output;
}

console.log(`${Object.keys(OPERATION).length} operations, the auth condition is checked at every step`);
console.log(`${"k".padStart(2)}${"sequences".padStart(11)}${"steps".padStart(8)}${"auth passed".padStart(13)}${"correct violation".padStart(19)}${"narrow violation".padStart(18)}${"narrow false positive".padStart(23)}`);
for (const k of [1, 2, 3, 4, 5]) {
  const d = sequences(k);
  const steps = d.reduce((t, x) => t + x.path.length, 0);
  const correct = d.filter((x) => !INVARIANT.correct(x.end));
  const narrow = d.filter((x) => !INVARIANT.narrow(x.end));
  console.log(`${String(k).padStart(2)}${String(d.length).padStart(11)}${String(steps).padStart(8)}` +
    `${String(steps).padStart(13)}${String(correct.length).padStart(19)}${String(narrow.length).padStart(18)}` +
    `${String(narrow.length - correct.length).padStart(23)}`);
}

const first = sequences(5).filter((x) => !INVARIANT.correct(x.end)).sort((a, b) => a.path.length - b.path.length)[0];
console.log(`\nshortest violation (${first.path.length} steps): ${first.path.join(" -> ")}`);
console.log(`end state: penalty ${first.end.penalty}, penalty on the partner ${first.end.partnerPenalty}, partner ${first.end.partner}`);
```

```
5 operations, the auth condition is checked at every step
 k  sequences   steps  auth passed  correct violation  narrow violation  narrow false positive
 1          3       3            3                  0                 0                      0
 2          8      13           13                  0                 0                      0
 3         17      40           40                  0                 2                      2
 4         28      84           84                  2                 4                      2
 5         45     169          169                  2                10                      8

shortest violation (4 steps): partner-add -> report-lost -> penalty-transfer -> partner-remove
end state: penalty 0, penalty on the partner 30, partner false
```

The fourth column matches the third on every row: **the auth condition passed on every
step taken.** Not one of the hundred sixty-nine steps was out of bounds. The fifth column
rises to two at four steps: a partnership is formed, a book is reported lost, the penalty
is transferred to the partner, the partnership is closed. The result is a member who
carries a penalty but is no longer a partner.

What finds this violation is not a rule but a **hand-written invariant**. Nowhere in the
lending system was the sentence "when a partnership closes, a transferred penalty reverts
too" written down; writing it is the job of a person who knows the domain. This is exactly
where automation ends: a machine can generate chains, it cannot say what the violation is.

The sixth and seventh columns show judgment's own margin of error. The narrow invariant
says "a penalty can never be transferred" and counts legitimate transfers as violations
too: at k=4, two of the four findings are false positives; at k=5, eight of the ten are. A
poorly written invariant is no different from a poorly written scan rule — it is only more
expensive, because a person sifts through its findings.

#### The Chain-Length Threshold

**NF12 (assumption):** the longest chain to examine is four steps. The threshold's source
is not a requirement but a budget; the table's second and third columns give what this
budget corresponds to.

**A false negative** is a violation that stays below the threshold: at k=3 both of the two real
violations were missed, at k=4 both were caught. **A false positive** is the narrow invariant
turning legitimate chains red, and it grows fast as the chain lengthens. Because the two
columns grow in the same direction, raising the threshold costs double: more sequences get
examined, and more false positives get sifted out.

The run-independent measure of the cost is the step count: 40 at k=3, 84 at k=4, 169 at
k=5. Even in a five-operation domain the count grows to nearly double each time; in a real
system the operation count is in the dozens, and raising the chain length by one
multiplies the sequences to examine many times over. This is why penetration testing is
bounded by a budget — coverage is not everything that could be found, it is everything
**up to a chosen length**.

**Who owns the decision:** these findings do not stop the release, because they are not
inside the run cycle. A penetration test finding opens a record; the record's priority
comes from the value of the asset in the threat model, and the fix is usually a rule
change — after which that rule becomes automatable. Judgment entries turn into automatic
entries over time, but the list never empties.

#### Summary

- A threat model is the list required for coverage to be countable; ten of fourteen
  entries were settled by automatic methods, four required human judgment.
- The coverage gap sits in the step count: 90% of single-step threats are automatic, 25%
  of multi-step threats are. Automation decides per request; it does not look at the
  relationship between steps.
- In the five-operation lending domain, all hundred sixty-nine steps passed the auth
  condition; the violation became visible only with a hand-written invariant and a
  four-step chain.
- The caught class is a business-logic defect born from combining operations; the
  uncaught class is every rule whose invariant was never written — the machine produces
  chains, it cannot produce the definition of a violation.
- A hand-written invariant has its own false positives too: a narrowly written invariant
  emptied out eight of ten findings at k=5.
- The cost grows with chain length: 40 steps at k=3, 84 at k=4, 169 at k=5. The
  threshold's source is a budget, not a requirement, and coverage is defined by that
  budget.

#### Next Step

The four lessons so far assumed a client that misuses the system. The other face of the
same question is a reader who **cannot use** it: does the lending screen work for someone
who cannot see it, cannot use a mouse, or reads with the text enlarged? This question's
criteria and their interface-side counterparts were built in the Frontend Quality course;
the next lesson's question is different, and it is this course's question: how many of
those criteria can be handed to a program, how many must be manually verified, and what is
the automatic check's own false positive and false negative rate.

### Accessibility Testing

Source: https://academia.sh/en/courses/non-functional-testing/accessibility-testing

Measuring an automatic audit's automatable share: separating criteria by decision source, the false positive and false negative counts an auditor that actually runs on a small element tree produces, and the area where manual verification is mandatory in a check where raising the threshold brings no gain.

Four lessons assumed a client that misuses the system. The other face of the same question
is a reader who **cannot use** it: does the lending screen work for someone who cannot see
the screen, cannot use a mouse, or reads with the text enlarged?

This question's criteria, their interface-side counterparts, and the automatic checker's
rule classes were built in the Frontend Quality course; they are not repeated here. This
lesson's question is a testing question, and it measures **the accessibility audit
itself**: the **automatable share**. How many of the criteria can be handed to a program,
how many must be manually verified, and what is the automatic check's own false positive
and false negative rate? The measurement will be made not over browser output but over a
small document model.

#### The Document Model and Known Defects

The lending screen's element tree carries eight defects; the location of each has been
manually verified.

```js
// document.mjs — the element tree of the lending screen and the manually verified defect list
export const DOCUMENT = {
  type: "page", title: "Loan Operations", lang: null, children: [
    { type: "section", name: "navigation", children: [
      { type: "link", href: "/catalog", text: "Catalog" },
      { type: "link", href: "/loan", text: "Loan" },
    ] },
    { type: "section", name: "main", children: [
      { type: "heading", level: 1, text: "Loan Operations" },
      { type: "heading", level: 3, text: "Waiting on the Shelf" },
      { type: "list", children: [
        { type: "row", children: [
          { type: "text", text: "The Hourglass" },
          { type: "action", id: "get-1", text: "", icon: "cart", size: 40, focusable: true },
        ] },
        { type: "row", children: [
          { type: "text", text: "Still on the Shelf" },
          { type: "action", id: "get-2", text: "Get", size: 40, focusable: true },
        ] },
      ] },
      { type: "field", id: "member-no", kind: "text", label: null },
      { type: "text", text: "Penalty: 30", color: "#8a8a8a", background: "#ffffff" },
      { type: "action", id: "history", text: "History", size: 40, focusable: true,
        disabled: true, color: "#9b9b9b", background: "#ffffff" },
    ] },
  ],
};

// Manually verified defects: criterion code, location, and a short reason.
// The disabled "history" action is exempt from the contrast threshold; it is not on the list.
export const ACTUAL = [
  ["D1", "get-1", "the cart-icon action's name is empty"],
  ["D2", "member-no", "the member-number field has no label"],
  ["D3", "heading 3", "jumps from heading level one to level three"],
  ["D5", "page", "the document carries no language declaration"],
  ["D7", "Penalty: 30", "the penalty text's contrast is below the threshold"],
  ["D9", "-", "the delay is signaled only by a color change"],
  ["D10", "-", "an action that performs two different operations carries the same name"],
  ["D11", "-", "the focus order does not match the visual order"],
];

export function* walk(d = DOCUMENT) {
  yield d;
  for (const c of d.children ?? []) yield* walk(c);
}
```

#### The Criteria's Decision Source

What decides whether a criterion can be automated is not the criterion's importance but
**where the decision is read from**. There are four sources: a declaration read directly
from the tree, a number computed from declared values, an intent that depends on what the
content means, and a sequence of interaction over time. The first two can be handed to a
program, the last two cannot.

```js
// auditor.mjs — the criterion set, decision sources, and the checks decidable from the tree
import { walk } from "./document.mjs";

// decision source: tree | calc (both automatic) | intent | sequence (both manual)
export const CRITERION = [
  ["D1", "action name is empty", "tree", 90], ["D2", "field has no label", "tree", 90],
  ["D3", "heading level is skipped", "tree", 70], ["D4", "id is duplicated", "tree", 90],
  ["D5", "no language declaration", "tree", 90], ["D6", "action does not take focus", "tree", 90],
  ["D7", "contrast is below the threshold", "calc", 60], ["D8", "touch target is small", "calc", 50],
  ["D9", "signaled by color alone", "intent", 0], ["D10", "action name is ambiguous", "intent", 0],
  ["D11", "focus order does not match the visual order", "intent", 0],
  ["D12", "the dialog cannot be exited with the keyboard", "sequence", 0],
  ["D13", "the timing of the change announcement", "sequence", 0],
  ["D14", "content is lost when text is enlarged", "sequence", 0],
];

const channel = (c) => (c <= 0.04045 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4);
const luminance = (h) => {
  const [r, g, b] = [1, 3, 5].map((i) => channel(parseInt(h.slice(i, i + 2), 16) / 255));
  return 0.2126 * r + 0.7152 * g + 0.0722 * b;
};
export const contrast = (fg, bg) => {
  const [a, b] = [luminance(fg), luminance(bg)].sort((x, y) => y - x);
  return (a + 0.05) / (b + 0.05);
};

export function audit(document) {
  const findings = [], ids = [];
  let previous = 0;
  for (const d of walk(document)) {
    if (d.type === "page" && !d.lang) findings.push({ criterion: "D5", location: "page" });
    if (d.id) ids.push(d.id);
    if (d.type === "heading") {
      if (previous && d.level > previous + 1) findings.push({ criterion: "D3", location: `heading ${d.level}` });
      previous = d.level;
    }
    if (d.type === "action") {
      if (!d.text) findings.push({ criterion: "D1", location: d.id });
      if (!d.focusable) findings.push({ criterion: "D6", location: d.id });
      if (d.size < 24) findings.push({ criterion: "D8", location: d.id });
    }
    if (d.type === "field" && !d.label) findings.push({ criterion: "D2", location: d.id });
    if (d.color && d.background && contrast(d.color, d.background) < 4.5)
      findings.push({ criterion: "D7", location: d.id ?? d.text, ratio: contrast(d.color, d.background).toFixed(2) });
  }
  const duplicated = ids.filter((k, i) => ids.indexOf(k) !== i);
  for (const k of new Set(duplicated)) findings.push({ criterion: "D4", location: k });
  return findings;
}
```

**NF13 (assumption):** the decision-source classification and the manually verified
defect list are correct. The automatable share is relative to these two lists; the share
changes when the criterion set is expanded.

#### Automatable Share and Findings

```js
// measurement.mjs — automatable share, findings, and threshold scanning
import { DOCUMENT, ACTUAL } from "./document.mjs";
import { CRITERION, audit } from "./auditor.mjs";

const AUTOMATIC = ["tree", "calc"];
const automatic = CRITERION.filter(([, , k]) => AUTOMATIC.includes(k));
console.log(`${CRITERION.length} criteria; breakdown by decision source:`);
for (const k of ["tree", "calc", "intent", "sequence"]) {
  const n = CRITERION.filter(([, , x]) => x === k).length;
  console.log(`  ${k.padEnd(10)}${String(n).padStart(3)}  ${((100 * n) / CRITERION.length).toFixed(0)}%`);
}
console.log(`automatable share: ${automatic.length}/${CRITERION.length} = ${((100 * automatic.length) / CRITERION.length).toFixed(0)}%`);

const score = Object.fromEntries(CRITERION.map(([k, , , p]) => [k, p]));
const label = Object.fromEntries(CRITERION.map(([k, a]) => [k, a]));
const actualSet = new Set(ACTUAL.map(([k, y]) => `${k}|${y}`));
const key = (b) => `${b.criterion}|${b.location}`;
const findings = audit(DOCUMENT);
const correct = findings.filter((b) => actualSet.has(key(b))).length;
console.log(`\n${findings.length} findings, ${ACTUAL.length} manually verified defects, automatically found ${correct}` +
  ` = ${((100 * correct) / ACTUAL.length).toFixed(0)}%`);
for (const b of findings) {
  console.log(`  ${b.criterion}  ${label[b.criterion].padEnd(33)}${String(b.location).padEnd(13)}` +
    `${String(score[b.criterion]).padStart(3)}  actual: ${actualSet.has(key(b)) ? "yes" : "no"}` +
    `${b.ratio ? `  (ratio ${b.ratio})` : ""}`);
}

console.log(`\n${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=5".padStart(10)}`);
for (const e of [50, 60, 70, 90]) {
  const k = findings.filter((b) => score[b.criterion] >= e);
  const fp = k.filter((b) => !actualSet.has(key(b))).length;
  const fn = ACTUAL.length - k.filter((b) => actualSet.has(key(b))).length;
  console.log(`${String(e).padStart(9)}${String(k.length).padStart(11)}${String(fp).padStart(16)}${String(fn).padStart(16)}${String(fp + 5 * fn).padStart(10)}`);
}
const automaticCode = automatic.map(([k]) => k);
const manual = ACTUAL.filter(([k]) => !automaticCode.includes(k));
console.log(`\n${manual.length} defects that never appear at any threshold (the manual verification area):`);
for (const [k, , g] of manual) console.log(`  ${k}  ${g}`);
```

```
14 criteria; breakdown by decision source:
  tree        6  43%
  calc        2  14%
  intent      3  21%
  sequence    3  21%
automatable share: 8/14 = 57%

6 findings, 8 manually verified defects, automatically found 5 = 63%
  D5  no language declaration          page          90  actual: yes
  D3  heading level is skipped         heading 3     70  actual: yes
  D1  action name is empty             get-1         90  actual: yes
  D2  field has no label               member-no     90  actual: yes
  D7  contrast is below the threshold  Penalty: 30   60  actual: yes  (ratio 3.45)
  D7  contrast is below the threshold  history       60  actual: no  (ratio 2.78)

threshold  remaining  false positive  false negative  cost w=5
       50          6               1               3        16
       60          6               1               3        16
       70          4               0               4        20
       90          3               0               5        25

3 defects that never appear at any threshold (the manual verification area):
  D9  the delay is signaled only by a color change
  D10  an action that performs two different operations carries the same name
  D11  the focus order does not match the visual order
```

The two ratios say different things. **The automatable share, 57%**, is the portion of
the criterion set that can be handed to a program, and it is independent of the document.
**The found-defect rate, 63%**, says that five of the eight defects in this document were
caught by the automatic audit, and it depends on the document: had the defects fallen into
a different distribution, this rate would change. The second number cannot be derived from
the first.

The one false positive came from the contrast calculation. A disabled action's text
contrast is below the threshold, but disabled controls are exempt from this threshold, and
the auditor does not know that. This is the common problem with calculated criteria: the
number is correct, the rule is correct, **the context is missing**.

#### Why Raising the Threshold Does Not Help

Threshold scanning gives a different result than the previous three lessons. There,
raising the threshold dropped false positives while missing some real defects, and there
was a sweet spot in between. Here, raising the threshold drops a single false positive
and, in exchange, misses two real defects; the cost rises monotonically. The lowest cost
sits at the **bottom** end of the scan.

The table's false-positive column is the **false-fail** count, its false-negative column
is the **false-pass** count; here false pass never drops below three at any threshold. The
reason is in the table's last three rows: the three missed defects never appear at any
threshold, because the criteria they correspond to carry no score — they cannot be handed
to a program. **Here, missing is not the result of the threshold choice; it is the
method's structural limit.** A delay being signaled only by a color change, two actions
with the same name doing different things, and a focus order that does not match the
visual order: none of the three leaves any missing declaration in the tree.

So this audit's threshold is not a score. **The threshold and its source** are written as
follows: release is tied to the condition that the automatic audit reports zero findings
**and** all six criteria on the manual verification list have been checked off. The
threshold's source is this measurement — three of the eight defects never appearing in the
automatic audit at all shows that a gate relying only on the automatic audit would turn
green with three real defects still present.

#### The Cost of the Audit

The run-independent cost lives in three places. The automatic side is cheap: one tree
traversal, fourteen elements, zero processes, zero requests; it can run on every change.

The manual side is not cheap, and its cost is measured by **the criterion count per
screen**: six criteria, separately for every screen. As the lending system's screen count
grows this number grows linearly, and unlike the automatic audit it is paid again with
every release.

The third cost is maintaining the classification. A criterion's decision source can
change: a check that requires intent today can become readable from the tree once a new
declaration is added to the document model. If the classification is not updated, the
automatable share appears lower than it is, and the manual verification list stays longer
than it needs to.

**Who owns the decision:** an automatic finding stops the release; the manual verification
list is a checklist, and an unchecked criterion does not stop the release, it only records
which criteria the release went out without testing.

#### Summary

- What determines a criterion's automatability is the decision's source: a declaration
  read from the tree and a computed value are automatic, intent and an interaction
  sequence are not.
- Eight of fourteen criteria can be checked automatically (automatable share 57%); five of
  the eight defects in this document were found automatically (63%). The two ratios
  measure different things, and one cannot be derived from the other.
- The one false positive came from a calculated criterion: a disabled control's contrast
  was below the threshold, but that control was exempt from the threshold — the number
  was correct, the context was missing.
- The three missed defects never appeared at any threshold; here, missing is the result
  of the method, not the threshold choice, and raising the threshold only increases the
  cost in one direction.
- The threshold is not a score but a condition: zero automatic findings **and** all six
  manual verification criteria checked off. Its source is the measurement that a gate
  relying only on the automatic audit would turn green with three real defects present.
- The cost: the automatic side is as cheap as one tree traversal, the manual side is as
  expensive as six criteria per screen, and it is paid again with every release.

#### Next Step

This audit ran on a single document model and gave a single result. Yet the same lending
screen does not produce the same tree for every reader: different rendering engines
interpret the same declaration differently, different screen widths build different
layouts, different input methods impose different interaction orders. Every measurement so
far was made in a single environment, and the user share that environment represents was
never asked. The next lesson opens the environment axes into a matrix, selects which
subset of the matrix to run with two separate criteria, and counts the share the selection
covers.

### Compatibility Testing

Source: https://academia.sh/en/courses/non-functional-testing/compatibility-testing

Measuring a matrix that grows with the product of the rendering engine, platform, and screen axes: reducing feature support profiles to distinct classes, the defect gap between selecting by share and selecting by profile, and how the selection criterion turns out more decisive than the threshold.

The previous audit ran on a single document model and gave a single result. Yet the same
lending screen does not produce the same tree for every reader: different rendering
engines support the same declaration differently, different screen widths build different
layouts. Every measurement so far was made in a single environment, and the user share
that environment represents was never asked.

**Compatibility testing** opens the product of these axes into a matrix and asks a single
question: which subset of this matrix will run? The Integration, Contract and End-to-End
Testing course's device–version matrix built the same arithmetic over device and operating
system version; the axis of the matrix here is not the device but the **feature support
profile**, and the selection criterion derives not from the user share but from how
distinct the profiles are.

#### The Matrix and Support Profiles

Three rendering engines, two platforms, and two screen classes give twelve environments.
The support for six features in each environment is a vector; environments carrying the
same vector are **indistinguishable** from a feature standpoint.

**NF14 (assumption):** the environment shares have been measured and the matrix is
complete. An environment absent from the list is counted as nonexistent; every coverage
ratio is relative to these twelve rows.

```js
// matrix.mjs — the compatibility matrix, feature support profiles, and known defects
export const FEATURE = ["persistent storage", "transition animation", "grid layout",
  "date formatter", "observer", "input suggestion"];

const ENGINE = { a: [1, 1, 1, 1, 1, 1], b: [1, 0, 0, 1, 1, 0], c: [1, 1, 1, 1, 0, 1] };

// On the mobile platform, the date formatter (feature 3) is absent in every engine.
export const profile = (o) =>
  ENGINE[o.engine].map((v, i) => (i === 3 && o.platform === "mobile" ? 0 : v)).join("");

// Twelve environments with a measured user share.
export const ENVIRONMENT = [
  { engine: "a", platform: "desktop", screen: "wide", share: 42 },
  { engine: "a", platform: "mobile", screen: "narrow", share: 26 },
  { engine: "b", platform: "desktop", screen: "wide", share: 14 },
  { engine: "c", platform: "mobile", screen: "narrow", share: 9 },
  { engine: "b", platform: "mobile", screen: "narrow", share: 3 },
  { engine: "c", platform: "desktop", screen: "wide", share: 2 },
  { engine: "a", platform: "desktop", screen: "narrow", share: 1.5 },
  { engine: "a", platform: "mobile", screen: "wide", share: 1 },
  { engine: "b", platform: "desktop", screen: "narrow", share: 0.7 },
  { engine: "c", platform: "desktop", screen: "narrow", share: 0.4 },
  { engine: "b", platform: "mobile", screen: "wide", share: 0.3 },
  { engine: "c", platform: "mobile", screen: "wide", share: 0.1 },
].map((o) => ({ ...o, name: `${o.engine}-${o.platform}-${o.screen}` }));

// Manually verified defects and the environment condition they appear under.
export const DEFECT = [
  ["K1 grid layout does not render", (o) => o.engine === "b"],
  ["K2 date format returns empty", (o) => o.platform === "mobile"],
  ["K3 list does not refresh without the observer", (o) => o.engine === "c"],
  ["K4 desktop layout overflows on a narrow screen", (o) => o.platform === "desktop" && o.screen === "narrow"],
  ["K5 queue client cannot connect on mobile", (o) => o.engine === "b" && o.platform === "mobile"],
  ["K6 mobile layout scatters on a wide screen", (o) => o.platform === "mobile" && o.screen === "wide"],
];

// Robust behaviors; strict verification would turn these red too.
export const ROBUST = [
  ["Y1 transition animation is skipped", (o) => o.engine === "b"],
  ["Y2 menu collapses on a narrow screen", (o) => o.screen === "narrow"],
];

export const found = (selection, list) => list.filter(([, k]) => selection.some(k)).length;
export const red = (selection, list) => selection.filter((o) => list.some(([, k]) => k(o))).length;
```

#### Three Selection Criteria

```js
// selection.mjs — the matrix's size, distinguishing profiles, and three selection criteria
import { ENVIRONMENT, FEATURE, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs";

const profiles = [...new Set(ENVIRONMENT.map(profile))];
console.log(`${ENVIRONMENT.length} environments, ${FEATURE.length} features, ${profiles.length} distinguishing feature support profiles`);
for (const p of profiles) {
  const matching = ENVIRONMENT.filter((o) => profile(o) === p);
  console.log(`  ${p}  ${String(matching.reduce((t, o) => t + o.share, 0).toFixed(1)).padStart(5)}%  ${matching.map((o) => o.name).join(", ")}`);
}

// (a) By share: from largest to smallest, until the cumulative share reaches the threshold.
const byShare = (threshold) => {
  const s = [];
  for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); }
  return s;
};
// (b) By profile: the environment with the largest share from each support profile.
const byProfile = profiles.map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]);
// (c) Profile and layout: the missing platform-screen pairs are added to (b).
const pair = (o) => `${o.platform}-${o.screen}`;
const profileAndLayout = [...byProfile];
for (const o of ENVIRONMENT) if (!profileAndLayout.some((x) => pair(x) === pair(o))) profileAndLayout.push(o);

console.log(`\n${"selection".padEnd(20)}${"runs".padStart(6)}${"share".padStart(7)}${"profile".padStart(9)}${"defects".padStart(9)}${"red runs".padStart(15)}`);
for (const [name, s] of [["by share 90%", byShare(90)], ["by profile", byProfile],
  ["profile and layout", profileAndLayout], ["full matrix", ENVIRONMENT]]) {
  console.log(`${name.padEnd(20)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(7)}` +
    `${`${new Set(s.map(profile)).size}/${profiles.length}`.padStart(9)}${`${found(s, DEFECT)}/${DEFECT.length}`.padStart(9)}` +
    `${String(red(s, ROBUST)).padStart(15)}`);
}

const missed = DEFECT.filter(([, k]) => !byShare(90).some(k));
console.log(`\nwhat the by-share selection misses:`);
for (const [name] of missed) console.log(`  ${name}`);
```

```
12 environments, 6 features, 6 distinguishing feature support profiles
  111111   43.5%  a-desktop-wide, a-desktop-narrow
  111011   27.0%  a-mobile-narrow, a-mobile-wide
  100110   14.7%  b-desktop-wide, b-desktop-narrow
  111001    9.1%  c-mobile-narrow, c-mobile-wide
  100010    3.3%  b-mobile-narrow, b-mobile-wide
  111101    2.4%  c-desktop-wide, c-desktop-narrow

selection             runs  share  profile  defects       red runs
by share 90%             4  91.0%      4/6      3/6              3
by profile               6  96.0%      6/6      4/6              4
profile and layout       8  98.5%      6/6      6/6              5
full matrix             12 100.0%      6/6      6/6              8

what the by-share selection misses:
  K4 desktop layout overflows on a narrow screen
  K5 queue client cannot connect on mobile
  K6 mobile layout scatters on a wide screen
```

Twelve environments come down to six profiles: the matrix is half the size from a
feature-support standpoint. Selecting by share covers 91% of users with four runs, but it
sees only four of the six profiles and misses three of the six defects. What the missed
ones share is telling: all three arise from **the combination of axes**. The desktop
layout overflowing on a narrow screen has nothing to do with any engine's support; it has
to do with a value the two axes take together.

Selecting by profile is therefore not enough: it covers every support profile with six
runs, but it still cannot see the layout defects, because the support vector never carries
the screen class. The third selection, eight runs, adds the missing platform–screen pairs
and finds all six of the six defects — with as many runs as two-thirds of the full matrix.

#### Threshold Scanning

The common rule is written as "cover this much of the users." What this threshold
actually buys can be scanned.

```js
// threshold.mjs — scanning the covered-share threshold; counts false pass and false fail
import { ENVIRONMENT, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs";

const byShare = (threshold) => {
  const s = [];
  for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); }
  return s;
};
const profiles = new Set(ENVIRONMENT.map(profile)).size;

console.log(`${"share threshold".padEnd(17)}${"runs".padStart(6)}${"covered".padStart(10)}${"profile".padStart(9)}` +
  `${"false pass".padStart(14)}${"false fail".padStart(14)}${"cost w=5".padStart(12)}`);
for (const e of [80, 90, 95, 97, 98, 99, 100]) {
  const s = byShare(e);
  const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST);
  console.log(`${`${e}%`.padEnd(17)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(10)}` +
    `${`${new Set(s.map(profile)).size}/${profiles}`.padStart(9)}${String(pass).padStart(14)}${String(fail).padStart(14)}` +
    `${String(fail + 5 * pass).padStart(12)}`);
}

// The set selected by the profile-and-layout criterion is placed in the same table.
const pair = (o) => `${o.platform}-${o.screen}`;
const s = [...new Set(ENVIRONMENT.map(profile))].map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]);
for (const o of ENVIRONMENT) if (!s.some((x) => pair(x) === pair(o))) s.push(o);
const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST);
console.log(`\nprofile-and-layout criterion: ${s.length} runs, false pass ${pass}, false fail ${fail},` +
  ` cost ${fail + 5 * pass}`);
console.log(`runs needed to catch the same cost by share: ${byShare(99).length}`);
```

```
share threshold    runs   covered  profile    false pass    false fail    cost w=5
80%                   3     82.0%      3/6             4             2          22
90%                   4     91.0%      4/6             3             3          18
95%                   6     96.0%      6/6             2             4          14
97%                   7     97.5%      6/6             1             5          10
98%                   8     98.5%      6/6             0             5           5
99%                   9     99.2%      6/6             0             6           6
100%                 12    100.0%      6/6             0             8           8

profile-and-layout criterion: 8 runs, false pass 0, false fail 5, cost 5
runs needed to catch the same cost by share: 9
```

**False pass** is a real defect sitting in an environment left outside the selection;
**false fail** is strict verification turning red in an environment that is robust but
behaves differently. The two move in opposite directions: as environments are added,
missed defects decrease, and runs that turn red for nothing increase. Even the full matrix
is not the best option — two of the eight red runs are unavoidable noise anyway.

The real finding is in the threshold itself. At 99% selecting by share finds every defect with nine
runs, while the profile-and-layout criterion gives the same result with eight. But one step lower,
at 98%, selecting by share also drops to eight runs and both cost five — the advantage belonged to
the chosen threshold, not to the criterion. **The threshold's source** here is not a measurement
but a misconception: covered user share is an intuitive number, but it has nothing to do with how
defects are distributed. Defects are distributed **by environment class**, not by user share, and
the selection criterion turns out more decisive than the threshold.

**Who owns the decision:** every environment left outside the matrix is a known gap. The
release decision is tied not to the sentence "all runs are green" but to "this many
profiles and this many layout pairs were run"; the classes that were not run go on record.

#### The Cost of Testing

The run-independent cost lives in three numbers. The first is the run count: eight
environments, each demanding its own setup, its own data, and its own time; this number
grows multiplicatively as axes are added — adding one value to one axis does not add to
the matrix, it multiplies it.

The second is maintaining the matrix: the support vectors are written by hand, and the
profile count changes once an engine starts supporting a feature. A support table that is
not updated either runs two environments it treats as distinguishable for nothing, or
never runs one of two environments it fails to treat as distinct.

The third is sifting through red runs: five of the eight runs turned red, and two of them
were not real defects. This is compatibility testing's permanent burden — not every
environment difference is a defect, and what separates a difference from a defect is a
decision, not a rule.

#### Summary

- The compatibility matrix is the product of its axes; twelve environments came down to
  six distinguishing support profiles over six features, and the matrix shrank by half
  from a feature standpoint.
- Selecting by share covered 91% of users with four runs but missed three of the six
  defects; all three of the misses arose not from a single axis but from the combination
  of axes.
- Selecting by profile was not enough either, because the support profile does not carry
  the screen class; the profile-and-layout criterion found all six of the six defects with
  eight runs — as many as two-thirds of the full matrix.
- Threshold scanning moved the two columns in opposite directions: as environments were
  added, false pass decreased and false fail increased. Even the full matrix was not the
  lowest-cost option.
- Defects are distributed by environment class, not by user share; the selection criterion
  is more decisive than the covered-share threshold.
- The cost is multiplicative: every new axis value grows the matrix by a factor, the
  support table demands manual upkeep, and a portion of red runs gets sifted through again
  on every run.

#### Next Step

This topic's six lessons tested the same system with six separate methods, and all six
shared a common assumption: the system's parts are working. Static scanning looked at the
code's text, dynamic testing sent requests to the running service, dependency scanning
matched installed versions against advisories, penetration testing questioned the
combination of rules, the accessibility audit read the rendered tree, compatibility
testing ran the same screen in different environments. Both load and attack were pressure
coming from **outside**, and in both cases the system itself stayed up. Yet what happens
when the database the lending service depends on stops responding, when the queue client
loses a message, or when the catalog service returns a half response, was tested in no
lesson. The next topic moves the pressure from outside to inside: a fault is deliberately
triggered in one of the system's own parts, and how the system meets that fault is
measured.

## Resilience Testing

Producing failure before it is unexpected: deliberately triggering failure scenarios with fault injection, running controlled breakage in a production-like environment with rules and abort conditions, having a backup-and-failover drill verify recovery objectives, and finding the capacity limit by raising load up to the breaking point.

### Fault Injection

Source: https://academia.sh/en/courses/non-functional-testing/fault-injection

Deliberately triggering failure scenarios: injecting delay, error, and loss into the catalog service to measure whether the loan flow's pattern actually engages, scanning the drop-rate threshold to count false passes against false fails, the injection layer's cost in code path and default-closed checks, and the failure class injection cannot see.

The previous topic closed out security testing and left a boundary behind. Both load and attack
were pressure from the **outside**: many valid requests in one case, a single deliberately
malformed request in the other. In both cases, every part of the system kept working. What
happens when one of the system's own components breaks was never tested.

**Fault injection** is deliberately triggering a failure scenario in a running system and
measuring the response it produces. The loan service depends on the catalog service, and that
dependency already carries two patterns built in the Resilience and Reliability course: timeout,
and graceful degradation via a fallback path. What those patterns do was explained there; the
question here is different — **does the pattern actually engage, and with what threshold can we
say so.**

#### The Injection Harness

The harness is a **model** and is written from scratch; no real chaos tool or cluster is set up.
The catalog service is a local `node:http` process with an injection layer placed in front of it.
The layer produces three fault types: **delay** (the response is held back), **error** (returns
503), and **loss** (the response is never written). Which request gets broken is derived from
the request sequence with a seeded hash; the same seed breaks the same set of requests.

**NF14 — 0.03 of healthy catalog requests stall for 150 ms.** This is an assumption: even on a
fault-free day a real service is not perfectly flat, and this stall sets the floor of the
threshold.

```js
// catalog.mjs — the catalog service and the injection layer. The plan comes from outside
// and defaults to closed; which request breaks is derived from the request sequence.
import { createServer } from 'node:http';

export const CLOSED = { type: 'closed', rate: 0, duration: 0 };
export const SEED = { injection: 20260731, noise: 19870415 };
export const NOISE = { rate: 0.03, duration: 150 };      // NF14: background stall

export function hash(seed, i) {                      // seeded, deterministic value per index
  let x = (seed ^ Math.imul(i + 1, 0x9e3779b1)) >>> 0;
  x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
  x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
  return ((x ^ (x >>> 16)) >>> 0) / 4294967296;
}

const RECORD = { 'K-903': { title: 'book K-903', author: 'Sabahattin Ali' } };

export function setupCatalog(plan = CLOSED) {
  const measure = { request: 0, injected: 0, stall: 0 };
  const server = createServer((request, response) => {
    const address = new URL(request.url, 'http://local');
    const index = Number(address.searchParams.get('index') ?? 0);
    const no = address.searchParams.get('no') ?? 'K-903';
    measure.request += 1;
    const injected = plan.type !== 'closed' && hash(SEED.injection, index) < plan.rate;
    if (injected) measure.injected += 1;
    const delayed = injected && plan.type === 'delay';
    const stall = delayed === false && hash(SEED.noise, index) < NOISE.rate;
    if (stall) measure.stall += 1;
    if (injected && plan.type === 'loss') return;             // response never written
    const send = () => {
      if (injected && plan.type === 'error') {
        response.writeHead(503, { 'content-type': 'application/json' });
        return response.end('{"error":"catalog cannot respond"}');
      }
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end(JSON.stringify({ no, ...RECORD[no] }));
    };
    const wait = delayed ? plan.duration : (stall ? NOISE.duration : 0);
    if (wait === 0) send(); else setTimeout(send, wait);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}
```

The loan flow produces one of three outcomes: a **full** response, a **gracefully degraded**
response (record fields empty), or a **dropped** request; the second outcome disappears once the
fallback path is closed.

```js
// loan.mjs — the loan flow requests a record from the catalog. Timeout and the fallback
// path are patterns from the Resilience and Reliability course; here we test if they engage.
export function loanFlow({ base, timeout, fallbackPath }) {
  const s = { full: 0, degraded: 0, dropped: 0, timeoutCount: 0, errorCount: 0 };
  return {
    measure: () => ({ ...s }),
    async lend(index, no = 'K-903') {
      try {
        const y = await fetch(`${base}/record?no=${no}&index=${index}`,
          { signal: AbortSignal.timeout(timeout) });
        if (y.ok === false) { await y.text(); throw new Error(`status ${y.status}`); }
        const g = await y.json();
        s.full += 1;
        return { no, title: g.title, status: 'full' };
      } catch (h) {
        if (h.name === 'TimeoutError' || h.name === 'AbortError') s.timeoutCount += 1;
        else s.errorCount += 1;
        if (fallbackPath === false) { s.dropped += 1; return null; }
        s.degraded += 1;                       // graceful degradation: response without record fields
        return { no, title: null, status: 'degraded' };
      }
    },
  };
}
```

#### The Response Per Fault Type

Injection is run across three configurations. The **healthy** configuration is the system under
test; **loose** and **noFallback** are two deliberately broken versions, and they measure whether
the test can tell them apart.

**NF13 — the run is 200 requests, 8 concurrent clients.** **NF15 — injection rate 0.20, delay
240 ms.** **NF16 — timeout is 100 ms in the healthy configuration, 400 ms in the loose one.** All
three are assumptions; the reasoning behind NF16 is this: 100 ms cuts off both the 150 ms stall
and the 240 ms delay, while 400 ms swallows both.

```js
// run.mjs — three configurations are run through four fault types and the measurement is
// written to measurement.json. Counts are deterministic; only wall-clock time depends on the run.
import { writeFileSync } from 'node:fs';
import { setupCatalog, CLOSED, NOISE } from './catalog.mjs';
import { loanFlow } from './loan.mjs';

const N = 200, CONCURRENCY = 8;                     // NF13
const PLAN = {
  closed: CLOSED,
  delay: { type: 'delay', rate: 0.20, duration: 240 },   // NF15
  error: { type: 'error', rate: 0.20, duration: 0 },
  loss: { type: 'loss', rate: 0.20, duration: 0 },
};
const CONFIG = {
  healthy: { timeout: 100, fallbackPath: true },           // NF16
  loose: { timeout: 400, fallbackPath: true },
  noFallback: { timeout: 100, fallbackPath: false },
};

async function run(configName, planName) {
  const { server, measure } = setupCatalog(PLAN[planName]);
  await new Promise((c) => server.once('listening', c));
  const base = `http://127.0.0.1:${server.address().port}`;
  const flow = loanFlow({ base, ...CONFIG[configName] });
  let index = 0;
  const worker = async () => { while (index < N) { const i = index; index += 1; await flow.lend(i); } };
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return { ...flow.measure(), ...measure };
}

const result = {};
const p = (x, n) => String(x).padStart(n);
console.log(`${N} requests, ${CONCURRENCY} concurrent; injection rate ${PLAN.delay.rate}, delay ` +
  `${PLAN.delay.duration} ms, background stall ${NOISE.rate} / ${NOISE.duration} ms`);
console.log(`\n${'config'.padEnd(13)}${'fault'.padEnd(9)}${'injected'.padStart(9)}` +
  `${'full'.padStart(6)}${'degraded'.padStart(10)}${'dropped'.padStart(8)}` +
  `${'timeout'.padStart(13)}${'drop rate'.padStart(13)}`);
for (const config of Object.keys(CONFIG)) {
  for (const plan of Object.keys(PLAN)) {
    const r = await run(config, plan);
    result[`${config}/${plan}`] = r;
    console.log(`${config.padEnd(13)}${plan.padEnd(9)}${p(r.injected, 9)}${p(r.full, 6)}` +
      `${p(r.degraded, 10)}${p(r.dropped, 8)}${p(r.timeoutCount, 13)}` +
      `${p(((100 * r.dropped) / N).toFixed(2) + '%', 13)}`);
  }
}
writeFileSync('measurement.json', `${JSON.stringify({ N, result }, null, 1)}\n`);
console.log(`\nin the fault-free run the injection check ran on ${result['healthy/closed'].request} requests, ` +
  `injecting on ${result['healthy/closed'].injected}`);
```

```
200 requests, 8 concurrent; injection rate 0.2, delay 240 ms, background stall 0.03 / 150 ms

config       fault     injected  full  degraded dropped      timeout    drop rate
healthy      closed           0   190        10       0           10        0.00%
healthy      delay           45   148        52       0           52        0.00%
healthy      error           45   148        52       0           10        0.00%
healthy      loss            45   148        52       0           52        0.00%
loose        closed           0   200         0       0            0        0.00%
loose        delay           45   200         0       0            0        0.00%
loose        error           45   155        45       0            0        0.00%
loose        loss            45   155        45       0           45        0.00%
noFallback   closed           0   190         0      10           10        5.00%
noFallback   delay           45   148         0      52           52       26.00%
noFallback   error           45   148         0      52           10       26.00%
noFallback   loss            45   148         0      52           52       26.00%

in the fault-free run the injection check ran on 200 requests, injecting on 0
```

The healthy configuration's three rows produce the same result: 45 injected requests, plus 10
stalled ones, turn into a degraded response — 52 in total — and dropped requests are zero. But
the path that engages the pattern is not the same: delay and loss count 52 timeouts, error only
10. The error type never runs the timeout; the other half of the pattern, the status-code check,
engages instead. **The same drop rate can come from different code paths**, which is why the
measurement also counts which path ran.

The loose configuration's second row is the lesson's most important number: all 45 injected
requests turn into full responses, degraded is zero, timeouts are zero. The pattern never
engaged, and the system displayed it as a flawless result — 200 full responses, a 0.00% drop
rate. **The broken configuration looks cleaner than the healthy one.**

#### The Threshold and Its Source

Calling something "successful" under a fault is choosing a threshold; the most natural threshold
is the drop rate. The scan below compares two rules across five thresholds: at each threshold,
how many defective configurations pass (**false pass**), and how many healthy ones fail (**false
fail**).

```js
// threshold.mjs — a threshold scan over the measurement.json written by run.mjs: at each
// threshold, how many defective configurations pass and how many healthy ones fail.
import { readFileSync } from 'node:fs';

const { N, result } = JSON.parse(readFileSync('measurement.json', 'utf8'));
const TRUTH = { healthy: 'healthy', loose: 'defective', noFallback: 'defective' };
const FAULT = ['delay', 'error', 'loss'];

function ruleA(config, e) {                       // drop rate threshold only
  const k = result[`${config}/closed`];
  if ((k.degraded + k.dropped) / N > e) return false;
  return FAULT.every((a) => result[`${config}/${a}`].dropped / N <= e);
}
function ruleB(config, e) {                       // drop rate + how often the pattern engaged
  if (ruleA(config, e) === false) return false;
  return FAULT.every((a) => result[`${config}/${a}`].degraded >= result[`${config}/${a}`].injected);
}

const count = (rule, e) => {
  let falsePass = 0, falseFail = 0;
  for (const [config, g] of Object.entries(TRUTH)) {
    const passes = rule(config, e);
    if (passes && g === 'defective') falsePass += 1;
    if (passes === false && g === 'healthy') falseFail += 1;
  }
  return { falsePass, falseFail };
};

const p = (x, n) => String(x).padStart(n);
console.log(`${'threshold e'.padEnd(12)}${'rule'.padEnd(9)}${'passing config'.padStart(26)}` +
  `${'false pass'.padStart(14)}${'false fail'.padStart(14)}`);
for (const e of [0, 0.02, 0.05, 0.10, 0.30])
  for (const [name, rule] of [['A', ruleA], ['B', ruleB]]) {
    const r = count(rule, e);
    const passing = Object.keys(TRUTH).filter((y) => rule(y, e));
    console.log(`${((100 * e).toFixed(0) + '%').padEnd(12)}${name.padEnd(9)}` +
      `${(passing.join(",") || "-").padStart(26)}${p(r.falsePass, 14)}${p(r.falseFail, 14)}`);
  }
console.log(`\nA: in the fault-free run the share going to the fallback path <= e, drop rate <= e in every fault`);
console.log(`B: A, and in every fault degraded responses >= injected requests ` +
  `(${result['healthy/delay'].injected}; source: the harness's own counter)`);
```

```
threshold e rule                 passing config    false pass    false fail
0%          A                             loose             1             1
0%          B                                 -             0             1
2%          A                             loose             1             1
2%          B                                 -             0             1
5%          A                     healthy,loose             1             0
5%          B                           healthy             0             0
10%         A                     healthy,loose             1             0
10%         B                           healthy             0             0
30%         A          healthy,loose,noFallback             2             0
30%         B                           healthy             0             0

A: in the fault-free run the share going to the fallback path <= e, drop rate <= e in every fault
B: A, and in every fault degraded responses >= injected requests (45; source: the harness's own counter)
```

**Rule A** cannot drive both error types to zero at any threshold. Below a 2% threshold, the
healthy configuration fails — the background stall sends 10 requests to the fallback path even
in the fault-free run; this is a **false fail**, and its source is not a flaw in the system but
noise in the environment. At a 30% threshold the noFallback configuration also passes: two
**false passes**. In the 5–10% range in between, false fail is zero, but the loose configuration
passes at every threshold; the effect of the flaw is not a drop but a turn into delay.

**Rule B** adds a second threshold, and the source of that threshold is the injection itself: the
harness **counts** that it broke 45 requests, so exactly 45 gracefully degraded responses can be
demanded of the pattern — **injection also produces the source of the threshold.** This is the
chosen threshold: e = 5% (its source is the measurement, the 10/200 share going to the fallback
path in the fault-free run), plus one gracefully degraded response per injected request (its
source is the harness's own counter). At this pair, false pass and false fail are both zero.

Rule B's limit was also measured: only the **delay** fault gives away the loose configuration; in
the error and loss rows, degraded is 45 and the rule passes. What makes the flaw visible is not
the threshold but the **fault type injected**. The outcome is a release decision: when rule B
breaks, the resilience configuration has changed, and the release stops.

#### Injection's Own Cost

The injection layer sits inside production code and runs on every request, even on a fault-free
day.

```js
// cost.mjs — the injection harness's own cost: code path, the default check, and how many
// paths can open the plan. measurement.json is produced by run.mjs.
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { setupCatalog } from './catalog.mjs';

const code = readFileSync('catalog.mjs', 'utf8').split('\n')
  .filter((s) => s.trim() !== '' && s.trim().startsWith('//') === false);
const injectionPattern = /plan|injected|delayed|NOISE|stall|hash|SEED/;
const decision = code.filter((s) => /if |\? /.test(s) && injectionPattern.test(s));

const { server, measure } = setupCatalog();               // no plan given: defaults to closed
await new Promise((c) => server.once('listening', c));
const base = `http://127.0.0.1:${server.address().port}`;
for (let i = 0; i < 50; i += 1) await (await fetch(`${base}/record?no=K-903&index=${i}`)).json();
server.closeAllConnections();
await new Promise((c) => server.close(c));
assert.equal(measure.injected, 0, 'the default plan must not inject');

const faultFree = JSON.parse(readFileSync('measurement.json', 'utf8')).result['healthy/closed'];
const injectionLines = code.filter((s) => injectionPattern.test(s)).length;
console.log(`catalog.mjs: ${code.length} lines of code, ${injectionLines} of them the injection layer ` +
  `(${((100 * injectionLines) / code.length).toFixed(1)}%); ${decision.length} decision points on the request path`);
console.log(`in the fault-free run the injection check ran on ${faultFree.request}/${faultFree.request} requests, ` +
  `injecting ${faultFree.injected} times`);
console.log(`paths that can open the plan: 1 (setupCatalog's plan parameter); default check ` +
  `${measure.injected} injections on 50 requests - passed`);
```

```
catalog.mjs: 38 lines of code, 13 of them the injection layer (34.2%); 5 decision points on the request path
in the fault-free run the injection check ran on 200/200 requests, injecting 0 times
paths that can open the plan: 1 (setupCatalog's plan parameter); default check 0 injections on 50 requests - passed
```

The cost is written in quantities independent of the run: 13 of the catalog code's 38 lines
(34.2%) are the injection layer, it adds 5 decision points on the request path, and these checks
run on **all** 200 requests in the fault-free run. That is not the real cost — the real cost is
what it means if the harness is accidentally left open in production: the loan flow breaking
itself. The counterpart to that risk is the **number of paths that can open the plan** — here,
1 — tested against a claim that the default is closed. If the plan were also read from an
environment variable, a header, and a configuration file, the number would be 4, and each would
need its own check.

#### The Failure Class Injection Cannot See

Injection produces the **symptom** of a failure, not its cause. This distinction leaves three
classes outside the harness. The first is a **multi-component failure**: the layer sits at a
single boundary, but the catalog and the notification channel could break at once. The second is
the **layer underneath the symptom**: a full disk produces the same delay but brings secondary
effects with it, such as logging loss. The third is **call sites where the layer is not
placed**: if a dependency has no layer, that path is never tested and the table still comes back
green. The harness's coverage is exactly as wide as the number of boundaries it covers.

#### Summary

- Fault injection deliberately triggers a failure scenario and measures the response it produces;
  three types were injected (delay, error, loss), the harness was written from scratch, and the
  seed is visible.
- In the healthy configuration all three fault types produced the same result (52 gracefully
  degraded responses, 0 dropped) but did not go through the same code path: 52 timeouts for delay
  and loss, 10 for error.
- The broken configuration whose timeout is longer than the injected delay produced 200 full
  responses and a 0.00% drop rate; a drop-rate threshold catches this flaw at no value at all.
- In the threshold scan, rule A produced 1 false fail below 2% and 2 false passes at 30%; rule B,
  which turns the injected request count itself into a threshold, zeroed out both.
- The chosen threshold is e = 5% (its source is the measurement: 10/200 going to the fallback
  path in the fault-free run) plus one gracefully degraded response per injected request (its
  source is the harness's own counter); the release stops when the rule breaks.
- The cost is 13 of 38 lines, 5 decision points on the request path, and a 200/200 check; the
  real risk is the plan being left open in production, and the number of opening paths is kept
  at 1 and tested with a claim about the default.

#### Next Step

In this lesson, injection was a **test harness**: the scope was a single boundary, the
environment was stood up only for this lesson, and nothing broken did anyone any harm. Doing the
same injection where real traffic flows is a different question, and it comes **before** whether
the pattern works: **when is breakage allowed, and when does it stop.** A deviation cannot be
measured without first writing down the system's normal behavior; blast radius cannot be
controlled without bounding in advance how many requests the experiment will affect; if things go
worse than expected, the number that stops the experiment must be clear. The next lesson builds
these three decisions — the steady state hypothesis, bounding the blast radius, and the abort
condition — and counts the cost of each being set late or wrong.

### Chaos Experiments

Source: https://academia.sh/en/courses/non-functional-testing/chaos-experiments

Designing controlled breakage in a production-like environment: measuring the steady state hypothesis from a control run, the blast radius bounded by the experiment tag turning out four times wider because of batching, scanning the share margin to count false fails against late detection, and measuring the damage that accumulates under the abort condition's consecutive window count.

The previous lesson used injection as a test harness: the scope was a single boundary, the
environment was stood up only for that lesson, and nothing that broke harmed anyone. When the
same injection is done in an environment where real traffic flows, three decisions move to the
front: what is the system's normal behavior, how many requests will the experiment affect, and
by what number will the experiment be stopped.

A **chaos experiment** is performing a controlled breakage in a production-like environment and
measuring whether it deviates from the system's normal behavior. This lesson deals with the
experiment's **design**; the patterns that respond to an injected fault were built in the
Resilience and Reliability course and are not retold here. Each of the three design decisions is
given as a number, and the cost of getting each one wrong is measured.

#### The Steady State Hypothesis

The **steady state hypothesis** is a measurable claim about the system's fault-free behavior,
written before the experiment. "The system is healthy" is not a hypothesis; "in every 40-request
window, the full-response share is above this threshold" is. Here the source of the threshold is
not a requirement but a **measurement**: before the experiment opens, the same load is run once,
and the window average is taken as the baseline.

The measurement harness is a **model** and is written from scratch. The catalog service is a
local `node:http` process; injection is applied only to requests carrying the **experiment tag**.
The loan service sends record lookups in batches of four, because a loan list requests several
records on a single screen.

**NF17 — batch size 4, injected delay 240 ms, loan service timeout 100 ms.** **NF18 — 0.03 of
untagged requests stall for 150 ms.** Both are assumptions; choosing a delay larger than the
timeout makes the effect of injection deterministic.

```js
// chaos.mjs — the catalog service scoped to the experiment, and the loan client that sends
// batches. THIS IS A MODEL: no real chaos tool or cluster is set up; injection applies only to
// a batch carrying the experiment tag, and which user is tagged comes from a seeded hash.
import { createServer } from 'node:http';

export const BATCH = 4, DELAY = 240, TIMEOUT = 100;    // NF17
export const NOISE = { rate: 0.03, duration: 150 };     // NF18

export function hash(seed, i) {
  let x = (seed ^ Math.imul(i + 1, 0x9e3779b1)) >>> 0;
  x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
  x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
  return ((x ^ (x >>> 16)) >>> 0) / 4294967296;
}

export function setupCatalog() {
  const measure = { batch: 0, tagged: 0, stall: 0 };
  const server = createServer((request, response) => {
    const a = new URL(request.url, 'http://local');
    const tag = a.searchParams.get('experiment') === '1';
    const no = Number(a.searchParams.get('batch'));
    measure.batch += 1;
    if (tag) measure.tagged += 1;
    const stall = tag === false && hash(31337, no) < NOISE.rate;
    if (stall) measure.stall += 1;
    const wait = tag ? DELAY : (stall ? NOISE.duration : 0);
    const send = () => {
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end('{"record":"full"}');
    };
    if (wait === 0) send(); else setTimeout(send, wait);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}

export function batchSender(base) {
  return async (no, tagged) => {
    try {
      await fetch(`${base}/record?batch=${no}&experiment=${tagged ? 1 : 0}`,
        { signal: AbortSignal.timeout(TIMEOUT) });
      return 'full';
    } catch { return 'degraded'; }
  };
}
```

#### Bounding the Blast Radius

The **blast radius** is the widest area the experiment could affect, and it is written down as a
number before the experiment starts. The plan here is: 0.05 of users are tagged, and injection
is applied only to them. A plan is an **intent**; what needs to be measured is whether the tag
actually covers that many requests.

**NF19 — the run is 400 requests, the window is 40 requests, the experiment opens after request
200, tagged user share is 0.05.** This is an assumption; opening the experiment in the middle of
the run lets the earlier windows serve as the control.

```js
// experiment.mjs — the control run and the experiment run are made with real requests; each
// request's record is written to log.json. Counts are deterministic, only wall-clock time depends on the run.
import { writeFileSync } from 'node:fs';
import { setupCatalog, batchSender, hash, BATCH } from './chaos.mjs';

const N = 400, WINDOW = 40, EXPERIMENT_START = 200, SHARE = 0.05, CONCURRENCY = 4;   // NF19
const taggedUser = (i) => hash(20260731, i) < SHARE;

async function run(experimentOn) {
  const { server, measure } = setupCatalog();
  await new Promise((c) => server.once('listening', c));
  const send = batchSender(`http://127.0.0.1:${server.address().port}`);
  const log = [];
  let index = 0;
  const worker = async () => {
    while (index < N / BATCH) {
      const b = index; index += 1;
      const members = Array.from({ length: BATCH }, (_, k) => b * BATCH + k);
      const tagged = experimentOn && members[0] >= EXPERIMENT_START && members.some(taggedUser);
      const result = await send(b, tagged);
      for (const u of members) log.push({ request: u, tagged, result });
    }
  };
  await Promise.all(Array.from({ length: CONCURRENCY }, worker));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return { log: log.sort((a, b) => a.request - b.request), measure };
}

const control = await run(false);
const experiment = await run(true);
writeFileSync('log.json', `${JSON.stringify({ N, WINDOW, EXPERIMENT_START,
  control: control.log, experiment: experiment.log }, null, 0)}\n`);

const inWindow = experiment.log.filter((k) => k.request >= EXPERIMENT_START);
const planned = inWindow.filter((k) => taggedUser(k.request)).length;
const affected = inWindow.filter((k) => k.tagged).length;
const degraded = inWindow.filter((k) => k.result === 'degraded').length;
const y = (a, b) => `${a}/${b} (${((100 * a) / b).toFixed(2)}%)`;
console.log(`${N} requests, batches of ${BATCH}, experiment opens after request ${EXPERIMENT_START}`);
console.log(`planned blast radius          : ${y(planned, inWindow.length)} tagged users`);
console.log(`measured blast radius         : ${y(affected, inWindow.length)} tagged batch members`);
console.log(`blast radius multiplier       : ${(affected / planned).toFixed(2)}x`);
console.log(`degraded in experiment window : ${y(degraded, inWindow.length)}`);
console.log(`degraded in control run       : ${y(control.log.filter((k) => k.result === 'degraded').length, N)}`);
```

```
400 requests, batches of 4, experiment opens after request 200
planned blast radius          : 11/200 (5.50%) tagged users
measured blast radius         : 44/200 (22.00%) tagged batch members
blast radius multiplier       : 4.00x
degraded in experiment window : 52/200 (26.00%)
degraded in control run       : 16/400 (4.00%)
```

The plan said 11 users; the measurement found 44 requests: a **blast radius multiplier of
4.00x**. Its source is **batching**. The tag is placed on the user, but injection is applied to
the batch; a single tagged member in a batch pulls the whole batch into the experiment. The
multiplier's upper bound is the batch size, and this run hit that upper bound, because all eleven
tagged users happened to fall into separate batches. **The blast radius is only as narrow as the
tag is carried**: every aggregation, queueing, and caching step that does not carry the tag
silently widens the radius.

This is an experiment design flaw, and it is visible only by measuring. The way to actually
narrow the radius is to carry the tag into the batch-forming rule as well; the cost of that is
the number of call sites that must now carry the tag, and that number must be known before the
experiment starts.

#### The Hypothesis, Falsification, and the Abort Condition

Once the experiment run ends, what is left is a request log. The hypothesis's threshold, the
abort condition's sensitivity, and the cost of both are scanned over the same log; the experiment
is not rerun.

```js
// scan.mjs — two scans over log.json: the steady state hypothesis's share margin and the
// abort condition's consecutive window count. The experiment is not rerun, its log is read.
import { readFileSync } from 'node:fs';

const g = JSON.parse(readFileSync('log.json', 'utf8'));
const windows = (log) => {
  const p = [];
  for (let b = 0; b < g.N; b += g.WINDOW) {
    const d = log.slice(b, b + g.WINDOW);
    p.push({ fullShare: d.filter((k) => k.result === 'full').length / d.length,
      damage: d.filter((k) => k.tagged && k.result === 'degraded').length });
  }
  return p;
};
const C = windows(g.control), E = windows(g.experiment);
const BASELINE = C.reduce((a, x) => a + x.fullShare, 0) / C.length;   // source of threshold: control run
const falsifies = (p, m) => p.fullShare < BASELINE - m;
const p = (x, n) => String(x).padStart(n);

console.log(`steady state hypothesis: full-response share in every window >= baseline - m; baseline ` +
  `is the control run's window average = ${(100 * BASELINE).toFixed(2)}% (${C.length} windows, ` +
  `${g.WINDOW} requests per window)`);
console.log(`\n${'share margin m'.padEnd(15)}${'falsified in control'.padStart(22)}` +
  `${'falsified in experiment'.padStart(25)}${'first falsified window'.padStart(24)}`);
for (const m of [0, 0.05, 0.10, 0.20]) {
  const first = E.findIndex((x) => falsifies(x, m));
  console.log(`${((100 * m).toFixed(0) + '%').padEnd(15)}` +
    `${p(`${C.filter((x) => falsifies(x, m)).length}/${C.length}`, 22)}` +
    `${p(`${E.filter((x) => falsifies(x, m)).length}/${E.length}`, 25)}` +
    `${p(first < 0 ? 'none' : first + 1, 24)}`);
}

const M = 0.10;                            // chosen share margin: no false falsification in control
const TOTAL = E.reduce((a, x) => a + x.damage, 0);
const after = (w) => E.slice(w).reduce((a, x) => a + x.damage, 0);
console.log(`\nchosen share margin ${100 * M}%; the experiment's total damage is ${TOTAL} requests`);
console.log(`\n${'consecutive windows k'.padEnd(23)}${'abort window'.padStart(14)}` +
  `${'damage until abort'.padStart(20)}${'cost over k=1'.padStart(16)}`);
let base = null;
for (const k of [1, 2, 3, 4]) {
  let streak = 0, stop = -1;
  for (let i = 0; i < E.length; i += 1) {
    streak = falsifies(E[i], M) ? streak + 1 : 0;
    if (streak >= k) { stop = i + 1; break; }
  }
  const incurred = stop < 0 ? TOTAL : TOTAL - after(stop);
  if (base === null) base = incurred;
  console.log(`${String(k).padEnd(23)}${p(stop < 0 ? 'never' : stop, 14)}` +
    `${p(incurred, 20)}${p(`+${incurred - base}`, 16)}`);
}
```

```
steady state hypothesis: full-response share in every window >= baseline - m; baseline is the control run's window average = 96.00% (10 windows, 40 requests per window)

share margin m   falsified in control  falsified in experiment  first falsified window
0%                               4/10                     7/10                       2
5%                               4/10                     7/10                       2
10%                              0/10                     4/10                       6
20%                              0/10                     2/10                       8

chosen share margin 10%; the experiment's total damage is 44 requests

consecutive windows k    abort window  damage until abort   cost over k=1
1                                   6                   8              +0
2                                   9                  28             +20
3                                  10                  44             +36
4                               never                  44             +36
```

The first table scans the share margin. Below a 5% margin, the hypothesis is falsified in four
of the control run's ten windows — while the experiment was never even open. This is a **false
fail**, and its consequence is serious: if the abort condition is tied to this hypothesis, the
experiment stops in the second window before it has effectively begun. Raised to a 20% margin,
the control clears, but the experiment is falsified in only two windows and the first
falsification shifts to window 8; most of the damage passes unseen. The chosen threshold sits
between the two: **m = 10%**, and its source is the control run's measured distribution — the
control's lowest window sits 6 points below the baseline, and the margin is set above that noise.

The second table scans the **abort condition**. The abort condition is the rule by which the
experiment terminates itself, and here it takes the form "stop if the hypothesis is falsified in
k consecutive windows." k = 1 stops the experiment at the sixth window, by which point 8 requests
have taken damage. k = 2 waits until the ninth window: 28 requests, that is, **20 requests
extra**. k = 3 and k = 4 never stop before the experiment ends, and accept the full damage: 44
requests. The cost of a late abort is written not in run duration but in the number of requests
damaged.

The seventh window shows why the second table can be misleading: even while the experiment was
open, the hypothesis was not falsified in that window, because the window average stayed above
the threshold. The experiment ran for five windows, and the hypothesis was falsified in four of
them; the falsification rate is **4/5, not 5/5**. The window average swallows sparse damage, and
growing k makes it harder to bridge these gaps.

The outcome of this test is not a release decision; it belongs to the experiment itself. When the
abort condition breaks, the experiment is stopped, the tag is withdrawn, and the falsified window
is opened as a record.

#### The Class the Experiment Does Not Catch

The experiment can only falsify what the hypothesis measures. Because the hypothesis is built on
the full-response share, **damage that turns into delay is invisible**: had the timeout been
400 ms, the requests would count as full responses and the hypothesis would stay silent — the
same as the loose configuration in the previous lesson. The second class is **damage that spills
outside the window**: work still queued after the experiment stops, or a retried request, never
enters the log. The third is **dependencies outside the scope**: this experiment only broke the
catalog boundary; the notification channel was never tested and the table would still come back
green.

#### Summary

- A chaos experiment is defined by three decisions: the steady state hypothesis, the blast
  radius, and the abort condition; all three are written down as numbers before the experiment
  starts.
- The hypothesis's threshold comes from measurement: the control run's window average, 96.00%,
  was taken as the baseline, and the share margin is a share set below that baseline.
- The planned blast radius was 11 users; the measured one came out to 44 requests: a blast radius
  multiplier of 4.00x, whose upper bound is the batch size; the radius is only as narrow as the
  tag is carried.
- Below a 5% share margin, the hypothesis was falsified in 4/10 control-run windows (a false
  fail); at 20%, only 2/10 experiment windows were falsified and the first falsification shifted
  to window 8.
- At the chosen 10% margin, the abort condition's damage is 8 requests at k = 1, 28 at k = 2, and
  44 at k = 3 and beyond; the cost of a late abort is measured in the number of requests damaged.
- The experiment ran for five windows and the hypothesis was falsified in four of them; the
  window average swallows sparse damage.

#### Next Step

This experiment broke a part of the system that kept running: the catalog slowed down, the loan
flow degraded gracefully, the experiment stopped, and everything returned to how it was. The
return was automatic, because nothing was lost. **That assumption collapses once what breaks is
data**: when a copy is lost, a record is corrupted, or a unit drops entirely, what brings the
system back is not stopping the experiment but **restoring the backup**. A backup is taken every
day, and every day it is taken is counted a success — but that is a claim, and it has never been
tested anywhere. The next lesson tests that claim: a backup existing does not show that it is
restorable, the recovery steps that were planned and the steps actually required are not the
same, and what a verification set fails to check can be counted just as well as what it does
check.

### Recovery Verification

Source: https://academia.sh/en/courses/non-functional-testing/recovery-verification

Testing that a backup is restorable: the same data actually restored in three backup formats and run through five checks, counting false passes against false fails as the verification set grows, a verification that never tests the write path failing to see a lost constraint, and the cost of the verification set staying fixed while restore work grows with the data.

The previous experiment broke a part of the system that kept running; when the experiment
stopped, everything returned to how it was, because nothing had been lost. **That assumption
collapses when what breaks is data**: when a copy is lost, what brings the system back is not
stopping the experiment but restoring the backup.

A backup is taken every day, and every day it is taken is counted a success. This is a claim.
**Recovery verification** is testing that claim by actually restoring the backup. The question
here is not how long recovery takes — the arithmetic of recovery objectives and the failover
model were built in the Resilience and Reliability course and are not repeated here. The question
here is: **which set of checks says a backup is restorable, and what can that set not see.**

#### A Backup Is a Dump, Its Scope Is a Decision

The library database consists of three tables and one index; the index and the third table were
added by two later migrations. A backup is a **SQL dump**, and the dump's **scope** is determined
by which objects make it in. Three formats will be compared: one that reads the schema from
`sqlite_master`, one whose hand-written table list has no knowledge of the latest migration, and
one that takes the tables but skips the indexes.

**NF20 — the source data is 6 books, 10 loans, and 3 fines; the scale factor grows it tenfold.**
**NF21 — the backup is restored within the same process.** Both are assumptions; the second is
what makes the harness a **model**, because in a real recovery the file moves to another machine,
and the move itself is a separate source of failure.

```js
// library.mjs — the catalog and loan schema, the data, and the three backup formats. A backup
// is a SQL dump; its scope is determined by which objects make it into the dump.
import { DatabaseSync } from 'node:sqlite';

export const SCHEMA = [
  'create table book(no text primary key, title text not null)',
  'create table loan(id integer primary key, book text not null, member text not null, day integer not null)',
  'create unique index loan_unique on loan(book, member)',                // migration 2
  'create table fine(member text primary key, amount integer not null)',   // migration 3
];

export function setup(path, scale = 1) {
  const db = new DatabaseSync(path);
  for (const s of SCHEMA) db.exec(s);
  const book = Array.from({ length: 6 * scale }, (_, i) => `K-${901 + i}`);
  const k = db.prepare('insert into book values (?, ?)');
  for (const no of [...book].reverse()) k.run(no, `book ${no}`);     // insertion order reversed
  const o = db.prepare('insert into loan(book, member, day) values (?, ?, ?)');
  for (let i = 0; i < 10 * scale; i += 1)
    o.run(book[i % book.length], `member-${Math.floor(i / 6) % 7}`, 14 + (i % 14));
  const c = db.prepare('insert into fine values (?, ?)');
  for (let i = 0; i < 3 * scale; i += 1) c.run(`member-${i}`, 5 * (i + 1));
  return db;
}

// 'full' reads the schema from sqlite_master; 'missing-table' uses a hand-written table list;
// 'missing-index' takes the tables but skips the indexes.
export function takeBackup(db, format) {
  const objects = db.prepare(
    'select type, name, sql from sqlite_master where sql is not null order by rowid').all();
  const tables = format === 'missing-table'
    ? ['book', 'loan']                      // hand-written list: does not know about migration 3
    : objects.filter((n) => n.type === 'table').map((n) => n.name);
  const lines = [];
  for (const n of objects) {
    if (n.type === 'index' && format === 'missing-index') continue;
    if (n.type === 'table' && tables.includes(n.name) === false) continue;
    lines.push(`${n.sql};`);
  }
  for (const t of tables) {
    const columns = db.prepare(`select * from pragma_table_info('${t}')`).all().map((s) => s.name);
    const orderBy = t === 'book' ? ' order by no' : '';    // dump order is not insertion order
    for (const r of db.prepare(`select * from ${t}${orderBy}`).all()) {
      const d = columns.map((s) => (typeof r[s] === 'number' ? r[s] : `'${r[s]}'`));
      lines.push(`insert into ${t}(${columns.join(',')}) values (${d.join(',')});`);
    }
  }
  return `${lines.join('\n')}\n`;
}
```

#### Three Backups, Five Checks

The checks are ordered by strength. **D1** only looks at whether the file exists. **D2** compares
the table names in the restored copy. **D3** compares each table's row count against the source.
**D4** tests the **write path**: is a duplicate loan record rejected. **D5** reads the first
record without an explicit order and compares it against the source.

```js
// verify.mjs — three backups are actually restored and run through five checks; then the
// verification set is grown to count false passes against false fails.
import { writeFileSync, statSync, rmSync } from 'node:fs';
import { DatabaseSync } from 'node:sqlite';
import { setup, takeBackup } from './library.mjs';

const source = setup(':memory:');
const tables = (db) => db.prepare(
  "select name from sqlite_master where type='table' order by name").all().map((r) => r.name);
const count = (db, t) => db.prepare(`select count(*) c from ${t}`).get().c;
const FIRST = 'select title from book limit 1';
const EXPECTED = { tables: tables(source), first: source.prepare(FIRST).get().title };
for (const t of EXPECTED.tables) EXPECTED[t] = count(source, t);

const CHECK = {
  D1: (_, file) => statSync(file).size > 0,
  D2: (db) => tables(db).join() === EXPECTED.tables.join(),
  D3: (db) => EXPECTED.tables.every((t) => tables(db).includes(t) && count(db, t) === EXPECTED[t]),
  D4: (db) => {                                   // write path: a duplicate loan must be rejected
    const r = db.prepare('select book, member from loan limit 1').get();
    try { db.prepare('insert into loan(book, member, day) values (?, ?, 1)').run(r.book, r.member); }
    catch { return true; }
    return false;
  },
  D5: (db) => db.prepare(FIRST).get().title === EXPECTED.first,
};
const TRUTH = { full: 'healthy', 'missing-table': 'defective', 'missing-index': 'defective' };
const NAMES = Object.keys(CHECK);

const result = {}, measure = {};
for (const format of Object.keys(TRUTH)) {
  const file = `backup-${format}.sql`;
  const dump = takeBackup(source, format);
  writeFileSync(file, dump);
  const db = new DatabaseSync(':memory:');
  db.exec(dump);
  result[format] = Object.fromEntries(NAMES.map((a) => {
    try { return [a, CHECK[a](db, file)]; } catch { return [a, false]; }
  }));
  measure[format] = { lines: dump.split('\n').length - 1, bytes: statSync(file).size };
  rmSync(file);
}

const p = (x, n) => String(x).padStart(n);
console.log(`source: ${EXPECTED.tables.length} tables, ` +
  `${EXPECTED.tables.map((t) => `${t}=${EXPECTED[t]}`).join(', ')} rows`);
console.log(`\n${'backup format'.padEnd(15)}${'actual'.padEnd(11)}` +
  `${NAMES.map((a) => a.padStart(8)).join('')}${'dump lines'.padStart(14)}${'bytes'.padStart(7)}`);
for (const b of Object.keys(TRUTH))
  console.log(`${b.padEnd(15)}${TRUTH[b].padEnd(11)}` +
    `${NAMES.map((a) => (result[b][a] ? 'passed' : 'failed').padStart(8)).join('')}` +
    `${p(measure[b].lines, 14)}${p(measure[b].bytes, 7)}`);
console.log(`D1 file exists  D2 table names  D3 row count  D4 write path  D5 first record (unordered)`);

console.log(`\n${'verification set'.padEnd(20)}${'passing backups'.padStart(34)}` +
  `${'false pass'.padStart(14)}${'false fail'.padStart(14)}`);
for (let n = 1; n <= NAMES.length; n += 1) {
  const set = NAMES.slice(0, n);
  const passing = Object.keys(TRUTH).filter((b) => set.every((a) => result[b][a]));
  const falsePass = passing.filter((b) => TRUTH[b] === 'defective').length;
  const falseFail = Object.keys(TRUTH).filter((b) => TRUTH[b] === 'healthy' && passing.includes(b) === false).length;
  console.log(`${set.join('+').padEnd(20)}${(passing.join(',') || '-').padStart(34)}` +
    `${p(falsePass, 14)}${p(falseFail, 14)}`);
}
```

```
source: 3 tables, book=6, fine=3, loan=10 rows

backup format  actual           D1      D2      D3      D4      D5    dump lines  bytes
full           healthy      passed  passed  passed  passed  failed            23   1518
missing-table  defective    passed  failed  failed  passed  failed            19   1282
missing-index  defective    passed  passed  passed  failed  failed            22   1463
D1 file exists  D2 table names  D3 row count  D4 write path  D5 first record (unordered)

verification set                       passing backups    false pass    false fail
D1                    full,missing-table,missing-index             2             0
D1+D2                               full,missing-index             1             0
D1+D2+D3                            full,missing-index             1             0
D1+D2+D3+D4                                       full             0             0
D1+D2+D3+D4+D5                                       -             0             1
```

What the first row says is the lesson's starting point: all three backups exist, and none of
them is empty. D1 does not distinguish any flaw; the backup job staying green says nothing more
than this check. **A backup's existence is not proof of its restorability.**

D2 and D3 catch the first flaw. The hand-written table list never picked up the `fine` table
added by the last migration; the restored copy has two tables and no fine records. The name for
this class of flaw is **backup scope drift**: the schema grows with migrations, the backup's
scope stays fixed in a hand-written list, and the gap between them is visible only by restoring.

The second flaw is far quieter. In the **missing-index** format, every table and every row is in
place: D2 passes, D3 passes. The only thing lost is the uniqueness index on the loan table. The
restored copy is **flawless when read**, and breaks only when written to — the same book can be
lent to the same member a second time. D4 is the only check that catches this. **A verification
that tests the read path cannot see a lost constraint.**

D5 shows the error in the opposite direction. The first record, read without an explicit order,
differs in the restored copy because the dump's rows were written in a different order — but the
restore is correct. **The flaw is in the check**: the result of an unordered query cannot be a
verification criterion. Once this check is added to the set, even the healthy backup fails, and
false fail rises to 1.

The chosen threshold is read from the table: the verification set is **D1+D2+D3+D4**, and the
passing criterion is that the whole set passes. The source of the threshold is not a requirement
but this **scan itself** — as the set grows to D4, false pass drops from 2 to 0 while false fail
stays at 0; adding one more check brings no gain, only loss. The decision belongs to the backup
job: when the set breaks, the backup is counted failed, and that day's backup is not accepted as
usable.

#### The Cost of the Verification Set

The recovery plan writes a single "restore" step and assigns it a fixed duration. The measurement
separates two quantities: **restore work and verification work do not grow the same way.**

```js
// cost.mjs — the cost of restore and of the verification set, at two data sizes. Query
// counts are measured with a counter; duration depends on the environment so it is not printed, work is.
import { DatabaseSync } from 'node:sqlite';
import { setup, takeBackup } from './library.mjs';

const counted = (db) => { const s = { n: 0 }; return { s, prepare: (q) => { s.n += 1; return db.prepare(q); } }; };
const tables = (db) => db.prepare(
  "select name from sqlite_master where type='table' order by name").all().map((r) => r.name);
const count = (db, t) => db.prepare(`select count(*) c from ${t}`).get().c;
const STEP = {
  'D2 schema': (d) => tables(d),
  'D3 count': (d) => tables(d).map((t) => count(d, t)),
  'D4 write': (d) => {
    const r = d.prepare('select book, member from loan limit 1').get();
    try { d.prepare('insert into loan(book, member, day) values (?, ?, 1)').run(r.book, r.member); }
    catch { /* constraint enforced: expected */ }
  },
};

const p = (x, n) => String(x).padStart(n);
console.log(`${'scale'.padEnd(8)}${'dump records'.padStart(13)}${'restored rows'.padStart(21)}` +
  `${'D2'.padStart(5)}${'D3'.padStart(5)}${'D4'.padStart(5)}${'set queries'.padStart(14)}`);
const measured = {};
for (const scale of [1, 10]) {
  const dump = takeBackup(setup(':memory:', scale), 'full');
  const restored = new DatabaseSync(':memory:');
  restored.exec(dump);
  const records = dump.split('\n').filter((x) => x.startsWith('insert')).length;
  const loaded = tables(restored).reduce((a, t) => a + count(restored, t), 0);
  const counts = [];
  for (const f of Object.values(STEP)) { const w = counted(restored); f(w); counts.push(w.s.n); }
  const total = counts.reduce((a, x) => a + x, 0);
  measured[scale] = { records, total };
  console.log(`${String(scale).padEnd(8)}${p(records, 13)}${p(loaded, 21)}` +
    `${counts.map((x) => p(x, 5)).join('')}${p(total, 14)}`);
}
console.log(`\nwhen data grows 10x, restore work grows ` +
  `${(measured[10].records / measured[1].records).toFixed(2)}x, the verification set grows ` +
  `${(measured[10].total / measured[1].total).toFixed(2)}x`);
```

```
scale    dump records        restored rows   D2   D3   D4   set queries
1                  19                   19    1    4    2             7
10                190                  190    1    4    2             7

when data grows 10x, restore work grows 10.00x, the verification set grows 1.00x
```

The numbers are independent of the run: work is measured instead of duration. Restore grew from
19 records to 190, a **10.00x** increase; the verification set stayed at 7 queries, **1.00x**.
**The cost of recovery grows with the data; the cost of verification does not.** The practical
consequence is this: skipping verification does not meaningfully shorten recovery time, but it
turns a recovery's success back into a claim.

The same table also names the plan's missing step. The plan's verification step was the row
count — D3. The drill showed that two more steps were needed: counting schema objects and
testing the write path. The plan wrote four steps; the measured count came to six, and the two
added steps together cost three queries. **The plan's most expensive gap is not a duration
estimate but a verification step that was never written.**

#### The Class Verification Cannot See

This set cannot see four things. The first is **content corruption**: values shifting while row
counts hold steady is invisible to D3. The second is **consistency at the moment the backup was
taken**: a dump taken mid-transaction can pass every check. The third is the **target
environment**; here the backup was restored within the same process, whereas in a real recovery
the file moves, and the move itself is a separate source of failure. The fourth is the **backup's
age**: every one of these checks also passes on yesterday's backup, and none of them counts the
data lost in between.

#### Summary

- A backup's existence is not proof of its restorability: all three backups passed D1, and two of
  them were defective.
- The hand-written table list did not pick up the latest migration's table; the name for this
  flaw class is backup scope drift, and it is visible only by restoring.
- The backup missing the index passed every read check (D2, D3) and failed only D4, which tests
  the write path; a lost constraint cannot be seen by reading.
- As the verification set grew to D4, false pass dropped from 2 to 0; once D5, which reads an
  unordered query, was added, even the healthy backup failed and false fail became 1.
- The chosen threshold is the whole D1+D2+D3+D4 set passing; its source is this scan, and when
  the set breaks, that day's backup is not counted usable.
- When data grew 10x, restore work grew 10.00x while the verification set stayed at 1.00x; the
  plan's four steps grew to six, and the two added steps cost three queries.

#### Next Step

The three experiments in this topic were all built the same way: a part of the system was chosen
and deliberately broken. The catalog slowed down, a batch was tagged, a table dropped out of the
backup. Every time, everything else kept working, and what was measured was whether the pattern
or check facing the broken part did its job. **None of these three experiments pushed the system
to its own limit.**

Where that limit sits was never asked. The library loan system gathers a batch in four requests,
cuts off a catalog call at 100 ms, and completes a recovery in 190 statements — but at what load
these numbers stop meaning anything is unknown. As load rises, the system reaches a point where
it stops slowing down and starts dropping instead; that point has to be found first, then written
down as a number. The next lesson raises load step by step to find that point, and asks by what
metric, with what threshold, the breaking point gets declared.

### Finding the Capacity Limit

Source: https://academia.sh/en/courses/non-functional-testing/finding-the-capacity-limit

Searching for the breaking point: raising load step by step and reading throughput, wait round, and successful request count separately at each step, the saturation step where throughput stops rising and the knee step where delay crosses the threshold not landing at the same place, the rule that declares the limit by throughput letting three steps pass falsely, and lengthening the queue growing throughput while leaving the successful request count unchanged.

The three experiments in this topic were all built the same way: a part of the system was chosen
and deliberately broken — the catalog slowed down, a batch was tagged, a table dropped out of the
backup. None of them pushed the system to its own limit.

A **breaking point test** raises load in stages and searches for the step where the system starts
to break down. The difficulty is not in measuring but in **recognizing**: the step where
throughput stops rising and the step where delay steepens may not be the same place. The
**saturation** and **utilization** terms from the Introduction to System Design course name
these two readings; the capacity planning from the Performance Anti-Patterns and Monitoring
course is this test's **input**. The question is: **which one gets declared the limit, and what
do the two error types of that decision come to.**

#### The Load Ladder and the Wait Round

The loan endpoint is built as a set of service slots with a fixed service duration and a bounded
queue. Instead of duration, **rounds** are counted: one round is one service duration, and a
request's **wait round** comes from its queue position at the moment it is accepted. The numbers
below do not change from run to run.

**NF22 — one service round is 40 ms, 4 service slots.** **NF23 — queue length 16, 5 waves per
step.** Both are assumptions; the requests in a wave arrive before the first service finishes,
which makes the queue position deterministic.

```js
// capacity.mjs — the loan endpoint runs with a fixed service duration, S service slots, and a
// queue of length Q. Every response carries the wait round computed from the queue position at
// the moment the request was accepted; if the queue is full the request is rejected (load shedding).
import { createServer } from 'node:http';

export const ROUND = 40;                       // NF22: one service round is 40 ms

export function setupEndpoint({ slots, queue }) {
  const measure = { accepted: 0, rejected: 0 };
  const waiting = [];
  let active = 0;
  const start = (finish) => {
    active += 1;
    setTimeout(() => {
      active -= 1;
      finish();
      if (waiting.length > 0) start(waiting.shift());
    }, ROUND);
  };
  const server = createServer((request, response) => {
    const occupied = active + waiting.length;
    if (occupied >= slots + queue) {
      measure.rejected += 1;
      response.writeHead(503, { 'content-type': 'application/json' });
      return response.end('{"status":"rejected"}');
    }
    measure.accepted += 1;
    const round = Math.floor(occupied / slots);      // which round it will enter service in
    const finish = () => {
      response.writeHead(200, { 'content-type': 'application/json' });
      response.end(`{"round":${round}}`);
    };
    if (active < slots) start(finish); else waiting.push(finish);
  });
  server.listen(0, '127.0.0.1');
  return { server, measure };
}
```

#### The Steps

The measure of acceptability comes from a number already measured in this topic: the timeout the
loan service places on its catalog call in the first lesson is 100 ms; with a 40 ms round, that
means a request can wait **at most one round**. One that waits longer is processed but never
reaches the caller. **NF24 — the acceptable wait is the number of rounds the timeout allows**; it
is not an assumption but a calculation derived from the first lesson's measurement.

```js
// ladder.mjs — load is raised step by step; at every step, throughput, wait round, and
// successful requests are counted. All numbers are integers and run-independent; duration is not printed.
import { writeFileSync } from 'node:fs';
import { setupEndpoint, ROUND } from './capacity.mjs';

const SLOTS = 4, QUEUE = 16, WAVE = 5;                 // NF22, NF23
const STEP = [1, 2, 4, 8, 12, 16, 20, 24, 32];
const TIMEOUT = 100;                                // lesson 01's NF16: 100 ms
const MAX_ROUND = Math.ceil(TIMEOUT / ROUND) - 2;    // NF24: wait round cap

async function runStep(base, c) {
  const rounds = [];
  let rejected = 0;
  for (let d = 0; d < WAVE; d += 1) {
    const s = await Promise.all(Array.from({ length: c }, async () => {
      const y = await fetch(base);
      const g = await y.json();
      return y.status === 503 ? null : g.round;
    }));
    for (const x of s) { if (x === null) rejected += 1; else rounds.push(x); }
  }
  const s = [...rounds].sort((a, b) => a - b);
  return { c, sent: c * WAVE, processed: rounds.length, rejected,
    p95: s.length > 0 ? s[Math.ceil(0.95 * s.length) - 1] : 0,
    successful: rounds.filter((x) => x <= MAX_ROUND).length };
}

async function ladder(queue, steps) {
  const { server } = setupEndpoint({ slots: SLOTS, queue });
  await new Promise((c) => server.once('listening', c));
  const base = `http://127.0.0.1:${server.address().port}/loan`;
  const r = [];
  for (const c of steps) r.push(await runStep(base, c));
  server.closeAllConnections();
  await new Promise((c) => server.close(c));
  return r;
}

const p = (x, n) => String(x).padStart(n);
const result = await ladder(QUEUE, STEP);
writeFileSync('ladder.json', `${JSON.stringify({ MAX_ROUND, result }, null, 0)}\n`);
console.log(`${SLOTS} service slots, ${QUEUE} queue, round ${ROUND} ms, ${WAVE} waves per step; ` +
  `timeout ${TIMEOUT} ms -> acceptable wait at most ${MAX_ROUND} rounds`);
console.log(`\n${'concurrent'.padStart(11)}${'sent'.padStart(10)}${'processed'.padStart(11)}` +
  `${'rejected'.padStart(10)}${'p95 wait rounds'.padStart(18)}${'successful'.padStart(12)}`);
for (const r of result)
  console.log(`${p(r.c, 11)}${p(r.sent, 10)}${p(r.processed, 11)}${p(r.rejected, 10)}` +
    `${p(r.p95, 18)}${p(r.successful, 12)}`);

const saturation = result.find((r, i) => i > 0 && r.processed === result[i - 1].processed);
const knee = result.find((r) => r.p95 > MAX_ROUND);
console.log(`\nsaturation step (where throughput stops rising): c=${saturation.c}, throughput ${saturation.processed} requests`);
console.log(`knee step (where p95 wait exceeds ${MAX_ROUND} rounds): c=${knee.c}`);

console.log(`\nsame step (c=32), two queue lengths:`);
for (const q of [QUEUE, 64]) {
  const [r] = await ladder(q, [32]);
  console.log(`queue ${p(q, 2)}: processed ${p(r.processed, 3)}, rejected ${p(r.rejected, 2)}, ` +
    `p95 ${r.p95} rounds, successful ${r.successful}`);
}
```

```
4 service slots, 16 queue, round 40 ms, 5 waves per step; timeout 100 ms -> acceptable wait at most 1 rounds

 concurrent      sent  processed  rejected   p95 wait rounds  successful
          1         5          5         0                 0           5
          2        10         10         0                 0          10
          4        20         20         0                 0          20
          8        40         40         0                 1          40
         12        60         60         0                 2          40
         16        80         80         0                 3          40
         20       100        100         0                 4          40
         24       120        100        20                 4          40
         32       160        100        60                 4          40

saturation step (where throughput stops rising): c=24, throughput 100 requests
knee step (where p95 wait exceeds 1 rounds): c=12

same step (c=32), two queue lengths:
queue 16: processed 100, rejected 60, p95 4 rounds, successful 40
queue 64: processed 160, rejected  0, p95 7 rounds, successful 40
```

#### Saturation or Knee

The three columns tell three different stories. **Processed requests** — throughput — rise up to
c = 24 and stop there: this is the **saturation step**. **p95 wait rounds** moves much earlier
and crosses the threshold at c = 12: this is the **knee step**. Three steps sit between them, and
across those three steps the system looks like it is "processing more requests" while most of its
responses are timing out.

The third column settles it. **Successful requests never rise after c = 8: fixed at 40.** Every
added request only joins the queue, lengthens the wait of the ones already there, and times out.
The real capacity is 40 successful requests, and it was found far earlier than where throughput
flattens out.

The last two lines say why this cannot be seen from throughput alone. The same step (c = 32) was
run at two queue lengths: at queue 16, throughput is 100 with 60 rejected; at queue 64, throughput
is 160 with none rejected — 160/100, that is, **1.60x**. Successful requests are 40 in both, that
is, **1.00x**. Lengthening the queue shifts the saturation step to the right and changes nothing
about what reaches the user; **a throughput criterion makes the breaking point something a queue
setting can hide.**

#### The Rule for Declaring the Limit

The limit is written down as a number, and the rule by which it was chosen **is** the threshold.
The scan compares two families of rules: one that looks at delay (p95 wait ≤ t rounds) and one
that looks at throughput (the last step before saturation).

**NF25 — a step is acceptable as long as the successful-request share does not drop below 99%.**
This is an assumption and is kept outside the rules; the rules decide without knowing this share.

```js
// limit.mjs — a scan of limit-declaration rules over ladder.json. The actual acceptance
// criterion is the successful-request share; the rules decide without knowing it.
import { readFileSync } from 'node:fs';

const { result } = JSON.parse(readFileSync('ladder.json', 'utf8'));
const SHARE = 0.99;                                        // NF25
const actual = (r) => r.successful / r.sent >= SHARE;
const saturation = result.findIndex((r, i) => i > 0 && r.processed === result[i - 1].processed);

const count = (accept) => {
  let falsePass = 0, falseFail = 0;
  result.forEach((r, i) => {
    if (accept(r, i) && actual(r) === false) falsePass += 1;
    if (accept(r, i) === false && actual(r)) falseFail += 1;
  });
  return { falsePass, falseFail, limit: result.filter(accept).at(-1)?.c };
};

const p = (x, n) => String(x).padStart(n);
console.log(`actual acceptance criterion: successful-request share >= ${100 * SHARE}%; ` +
  `the highest step meeting this criterion is c=${result.filter(actual).at(-1).c}`);
console.log(`\n${'rule'.padEnd(30)}${'declared limit'.padStart(18)}` +
  `${'false pass'.padStart(14)}${'false fail'.padStart(14)}`);
for (const t of [0, 1, 2, 3, 4]) {
  const r = count((x) => x.p95 <= t);
  console.log(`${`delay: p95 <= ${t} rounds`.padEnd(30)}${p(`c=${r.limit}`, 18)}${p(r.falsePass, 14)}${p(r.falseFail, 14)}`);
}
const v = count((_, i) => i < saturation);
console.log(`${'throughput: before saturation'.padEnd(30)}${p(`c=${v.limit}`, 18)}${p(v.falsePass, 14)}${p(v.falseFail, 14)}`);
```

```
actual acceptance criterion: successful-request share >= 99%; the highest step meeting this criterion is c=8

rule                              declared limit    false pass    false fail
delay: p95 <= 0 rounds                       c=4             0             1
delay: p95 <= 1 rounds                       c=8             0             0
delay: p95 <= 2 rounds                      c=12             1             0
delay: p95 <= 3 rounds                      c=16             2             0
delay: p95 <= 4 rounds                      c=32             5             0
throughput: before saturation               c=20             3             0
```

The last line is this lesson's outcome. **The throughput rule declares the limit at c = 20 and
lets three steps through falsely — 12, 16, and 20.** In all three, the system processes more
requests; in none of them do more users get an answer. When the delay rule's threshold is set to
one round, it declares the limit at c = 8 and zeroes out both error types.

The other end of the scan shows the cost of tightening the threshold: the p95 ≤ 0 rounds rule
accepts no wait at all and eliminates c = 8 too — a **false fail**; at that step the wait is below
the timeout, and half the capacity is written off for nothing. **The chosen threshold is p95 wait
≤ 1 round**, and its source is the timeout the first lesson measured. The decision belongs to the
capacity plan: this number does not stop a release, it feeds into a scaling decision.

#### Cost and the Limit Not Seen

The run-independent cost: nine steps, 595 requests sent, 515 accepted, and 80 rejected. That is
not the real cost — **the only way to find the limit is to go past it.** The last five steps
deliberately cause harm: of the 440 requests processed there, 240 timed out. In a production-like
environment, this cannot be done without bounding the blast radius, and it needs the previous
lesson's abort condition.

There are three classes the ladder cannot see. The first is a **limit tied to the workload mix**:
the ladder sent the same request to a single endpoint. The second is **cumulative degradation**:
a five-wave step is not the same window as load sustained for hours. The third is **collapse
behavior**: because this endpoint does load shedding, throughput stayed flat past saturation; in
a system without shedding, throughput would drop.

#### Summary

- A breaking point test raises load step by step; across nine steps, throughput, p95 wait round,
  and successful requests were read separately, and every number is an integer.
- The saturation step is c = 24, the knee step is c = 12; three steps sit between them.
- Successful requests stayed fixed at 40 after c = 8: every added request only lengthened the
  queue and timed out.
- When the queue was raised from 16 to 64, throughput grew 1.60x while successful requests stayed
  at 1.00x.
- The chosen threshold is p95 wait ≤ 1 round, its source is the first lesson's 100 ms timeout, and
  it gives 0 / 0; the throughput rule lets three steps through falsely.
- The cost is 595 requests sent, and it requires going past the limit: of the 400 requests
  processed in the last five steps, 240 timed out.

#### Course Wrap-Up

This course's fifteen lessons tested the same system with three questions — what happens under
load, under attack, and under failure — and every lesson tied its outcome to a threshold.

| Lesson | Threshold tested | Source of threshold | False pass / false fail | Cost |
|---|---|---|---|---|
| Performance Testing Types | p95 ≤ 20 ms, across four decision windows | NF1 assumption+calculation: the 60 ms budget split three ways | 1 / 0; at 3 ms 0 / 0, at 1 ms 0 / 3 | 9 processes, 55,000 requests, 1.00 billion row steps |
| Test Scenario Modeling | same 20 ms, as the scenario changes | inherited from NF1 | 1 / 0; at 5 ms 0 / 1; at 100 ms 3 / 0 | 3,000 requests per run; the scan step spreads 10.7x |
| Metric Selection | at least $1/(1-p)$ samples per percentile | NF5 calculation | at 20 samples 22/200 / 0; at 100 samples 0 / 0 | 8,496 and 8,494 latency values, 1,068,000 samples drawn |
| Bottleneck Analysis | layer share ≥ 0.50 | NF6 assumption | 1 misattribution / 0 unattributable; at 0.80 0 / 1 | 5 processes, 4 clock readings per request |
| Benchmarking Discipline | candidate/baseline ±5%, 8 rounds | NF9 assumption; lower bound is the measured spread | 0 / 0; at 2% and one round 155 / 179; at 20% 500 / 0 | 30 processes, 2.4 million rows of fill, 28,000 re-made decisions |
| Static Application Security Testing | finding score ≥ 40 | NF8 assumption: a missed defect = 5 false positives | 1 / 2; at 50 2 / 0 | 75 comparisons, 0 processes, 7 manual reviews |
| Dynamic Application Security Testing | finding score ≥ 50 | measurement: lowest cost at 50 | 1 / 0 | 1 process, 8 requests, a hand-built route table |
| Dependency and Component Scanning | severity ≥ 80 | measurement; weight from NF8 | 1 / 0 | 5 of 10 packages upgraded, each with a regression test |
| The Role of Penetration Testing | chain length ≤ 4 steps | budget (NF12), not a requirement | at k=3 2/2; at k=4 0/2; at k=5 0/8 | 40 steps at k=3, 84 at k=4, 169 at k=5 |
| Accessibility Testing | zero automated findings and 6 manual criteria | measurement: the automated check comes back green with three flaws | 3 / 1 | 1 tree traversal, 14 elements; 6 criteria per screen |
| Compatibility Testing | covered-user share ≥ 90% | fallacy: the share does not represent the defect distribution | 3 / 3; with the profile-and-layout criterion 0 / 5 | 8 environment runs, a hand-built support table, multiplicative growth |
| Fault Injection | drop rate ≤ 5% and one degraded response per injected request | measurement: 10/200; the second threshold is the harness's own counter | 0 / 0; with the drop rate alone 1 / 0 | 13 of 38 lines, 5 decision points, 1 opening path |
| Chaos Experiments | window full-response share ≥ baseline − 10% | measurement: the control run's average, 96.00% | 1 / 0; at 5% 0 / 4; at 20% detection shifts to window 8 | blast radius multiplier 4.00; 44 damaged requests at k = 3 |
| Recovery Verification | the whole of D1+D2+D3+D4 must pass | measurement: the set scan | 0 / 0; with D1 alone 2 / 0; with D5 0 / 1 | restore grows 10.00x, verification stays at 7 queries |
| Finding the Capacity Limit | p95 wait ≤ 1 round | measurement: the first lesson's 100 ms timeout | 0 / 0; with the throughput criterion 3 / 0 | 595 requests, 9 steps, 240 timeouts |

The third column is this course's rule: **a threshold without a source carries no decision.** The
fourth column is the other half of that rule. In functional testing, the outcome is a pass/fail,
because the expected value is known. In non-functional testing, the outcome is a **distribution**,
and calling it "successful" is choosing a threshold; that threshold can only be defended with two
error types — how many real defects it lets through, and how many healthy runs it turns red. None
of the rows in the table that zeroed out both did so by tightening the threshold; every one of
them changed **what it measured**.

Across four courses, testing itself was measured: which test catches which class of defect, which
it misses, and what it costs in return. But one question was never asked — **when will each test
run.** Some tests here are cheap enough to run on every change; some are expensive enough to run
once per release; one needs a production-like environment. How limited time gets divided among
these tests, and how a red result turns into a team decision, was never addressed in any lesson.
The Testing Process and Automation Infrastructure course opens with that question.