---
title: 'Integration, Contract and End-to-End Testing'
source: 'https://academia.sh/en/courses/integration-testing'
language: en
updated: '2026-08-23T14:25:13+00:00'
license: 'CC BY-SA 4.0'
lessons: 15
outcomes:
    - 'Sets up tests that work with external dependencies in isolation'
    - 'Secures agreement between services with contract tests'
    - 'Keeps end-to-end tests stable and maintainable'
---

# Integration, Contract and End-to-End Testing

Testing whether the world the unit test builds out of test doubles matches reality: deciding which boundary gets run for real, managing the test environment and its data, database and migration verification, imitating an external service with record and replay, securing agreement between services with consumer-driven contract testing, and keeping tests that run against the whole application stable and maintainable.

## Integration Tests

Choosing and paying for the boundary that runs for real: deciding which dependency runs with the real thing, managing the environment that brings dependent services up, preparing and masking production-like test data, putting migration and schema under test, and imitating an external service with record and replay approaches.

### The Scope of Integration Testing

Source: https://academia.sh/en/courses/integration-testing/scope-of-integration-testing

Treating which dependency runs for real as a decision: running the same test against a fake and a real catalog, catching a schema mismatch only at the real boundary, and measuring the boundary's cost in queries per run.

The Unit Testing and Test-Driven Development course closed by naming an assumption:
every test there ran in-process and without a real dependency, and it assumed that its
test doubles matched reality without ever checking it anywhere. This course tests that
assumption. The first question is where the testing should start.

The answer is not a virtue, it is a decision. The sentence "testing with the real thing
is safer" selects nothing; running the real thing for each dependency brings its own
cost and makes a separate defect class visible. This lesson makes that decision on one
example and measures three things together: the defect class the chosen boundary
**catches**, the defect class it **misses**, and its **cost**. That is the measurement
axis for the whole course.

#### The Boundary Decision

The loan service had four dependencies: the catalog holding records, the notification
channel that informs the member, the transaction log, and the clock that gives the day
number. Each one has two options — the fake dependency or the real one — and when the
four are chosen independently, sixteen configurations result. Not all of these are
meaningful. The clock's real version exposes no defect class in the test, it only
destroys determinism. The notification channel's real version requires bringing up a
process. The catalog sits between the two: its real version is a database that can run
in-process and carries its own constraints.

This lesson runs the catalog boundary for real and leaves the rest fake. This
configuration's name is the **integration test**: the tested code runs together with at
least one dependency's real implementation.

#### Two Configurations of the Same Test

The loan service stands exactly as it came from the previous course.

```js
// service.mjs — the loan service from M21/K02: with catalog, notification, and clock dependencies
export const RULES = {
  student: { loanDays: 28, maxLoans: 10 },
  member: { loanDays: 14, maxLoans: 5 },
};

export function createLoanService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      if (catalog.memberLoanCount(member.id) >= rule.maxLoans) {
        throw new Error(`loan limit exceeded: ${rule.maxLoans}`);
      }
      const today = clock();
      const record = { memberId: member.id, bookId, borrowedDay: today, dueDay: today + rule.loanDays };
      catalog.add(record);
      notification.send(member.id, `due date: ${record.dueDay}`);
      return record;
    },
    checkIn(bookId) {
      const record = catalog.find(bookId);
      if (record === undefined) throw new Error(`no loan record: ${bookId}`);
      catalog.remove(bookId);
      return record;
    },
  };
}
```

Two implementations of the catalog interface are written. The first is the fake object
from the previous course. The second meets the same interface on a real schema; the
schema is the version used in production and comes from the team that owns the catalog.

```js
// catalogs.mjs — two implementations of the same catalog interface: fake object and node:sqlite
import { DatabaseSync } from 'node:sqlite';

export const SCHEMA = `
CREATE TABLE loan (
  book_no    TEXT PRIMARY KEY,
  member_no  TEXT NOT NULL,
  branch     TEXT NOT NULL,
  borrow_day INTEGER NOT NULL,
  due_day    INTEGER NOT NULL,
  CHECK (due_day > borrow_day)
);`;

export function fakeCatalog() {
  const records = new Map();
  return {
    memberLoanCount: (memberId) => [...records.values()].filter((r) => r.memberId === memberId).length,
    add: (record) => records.set(record.bookId, record),
    find: (bookId) => records.get(bookId),
    remove: (bookId) => records.delete(bookId),
    queryCount: () => 0,
  };
}

export function sqliteCatalog(path = ':memory:') {
  const db = new DatabaseSync(path);
  db.exec(SCHEMA);
  let queries = 0;
  const run = (text, ...bound) => { queries += 1; return db.prepare(text).run(...bound); };
  const one = (text, ...bound) => { queries += 1; return db.prepare(text).get(...bound); };
  return {
    memberLoanCount: (memberId) => one('SELECT COUNT(*) AS n FROM loan WHERE member_no = ?', memberId).n,
    add: (record) => run(
      'INSERT INTO loan (book_no, member_no, branch, borrow_day, due_day) VALUES (?, ?, ?, ?, ?)',
      record.bookId, record.memberId, record.branch ?? null, record.borrowedDay, record.dueDay),
    find: (bookId) => {
      const s = one('SELECT book_no, member_no, borrow_day, due_day FROM loan WHERE book_no = ?', bookId);
      return s && { bookId: s.book_no, memberId: s.member_no, borrowedDay: s.borrow_day, dueDay: s.due_day };
    },
    remove: (bookId) => run('DELETE FROM loan WHERE book_no = ?', bookId),
    queryCount: () => queries,
  };
}
```

The fixture selects which implementation to build from a single place. The notification
is a spy object, the clock is a function; both keep their shape from the previous course.

```js
// fixture.mjs — the same test's two configurations: catalog fake or real
import { createLoanService } from './service.mjs';
import { fakeCatalog, sqliteCatalog } from './catalogs.mjs';

export function createFixture(configuration) {
  const catalog = configuration === 'real' ? sqliteCatalog() : fakeCatalog();
  const sent = [];
  const notification = { send: (memberId, text) => sent.push({ memberId, text }) };
  const service = createLoanService({ catalog, notification, clock: () => 1000 });
  return { service, catalog, sent };
}

export const MEMBER = { id: 'U-17', type: 'member', branch: 'central' };
```

The tests are a single file and read the configuration from an environment variable. The
third test exercises the boundary check directly, so it writes five records into the
catalog itself; the side writing those records knows the schema and supplies the
`branch` field.

```js
// loan.test.mjs — the same three tests; the CATALOG variable picks whether the boundary is real
import test from 'node:test';
import assert from 'node:assert/strict';
import { createFixture, MEMBER } from './fixture.mjs';

const configuration = process.env.CATALOG ?? 'fake';

test('a new loan appears in the catalog', () => {
  const { service, catalog } = createFixture(configuration);
  service.lendBook(MEMBER, 'K-903');
  assert.equal(catalog.find('K-903').dueDay, 1014);
});

test('a returned book drops from the catalog', () => {
  const { service, catalog } = createFixture(configuration);
  service.lendBook(MEMBER, 'K-903');
  service.checkIn('K-903');
  assert.equal(catalog.find('K-903'), undefined);
});

test('an error is raised when the loan limit is exceeded', () => {
  const { service, catalog } = createFixture(configuration);
  for (let i = 0; i < 5; i += 1) {
    catalog.add({ bookId: `K-90${i}`, memberId: MEMBER.id, branch: 'central', borrowedDay: 1000, dueDay: 1014 });
  }
  assert.throws(() => service.lendBook(MEMBER, 'K-999'), /loan limit exceeded: 5/);
});
```

#### Green at the Fake Boundary

This is the configuration the previous course left behind: the catalog is a fake object.

```sh
CATALOG=fake node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - a new loan appears in the catalog
ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 3
# fail 0
```

All three tests pass. This run does not show that the service is correct, it shows that
the service does whatever **the fake catalog accepts**. A fake catalog is a map, and a
map accepts a record of any shape.

#### Red at the Real Boundary

The one change is running the catalog boundary for real. The test file, the assertions,
and the service stay exactly as they are.

```sh
CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
not ok 1 - a new loan appears in the catalog
not ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 1
# fail 2
```

The first two tests fail. The reason is written in the run's detail.

```sh
CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep "error:" | sort -u
```

```
  error: 'NOT NULL constraint failed: loan.branch'
```

The name of the caught defect class is **schema mismatch**: the record the service
produces does not carry a field the real schema requires. This defect was in the code and
had been there through the whole previous course; the fake catalog could not see it,
because a fake object has no acceptance rule. The third test staying green also carries
information: the side that writes records while knowing the schema is defect-free, the
defect is only in the path the service writes.

The fix is in the service's record shape.

```js
// service.mjs — the record now carries the branch field too
export const RULES = {
  student: { loanDays: 28, maxLoans: 10 },
  member: { loanDays: 14, maxLoans: 5 },
};

export function createLoanService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      if (catalog.memberLoanCount(member.id) >= rule.maxLoans) {
        throw new Error(`loan limit exceeded: ${rule.maxLoans}`);
      }
      const today = clock();
      const record = {
        memberId: member.id, bookId, branch: member.branch,
        borrowedDay: today, dueDay: today + rule.loanDays,
      };
      catalog.add(record);
      notification.send(member.id, `due date: ${record.dueDay}`);
      return record;
    },
    checkIn(bookId) {
      const record = catalog.find(bookId);
      if (record === undefined) throw new Error(`no loan record: ${bookId}`);
      catalog.remove(bookId);
      return record;
    },
  };
}
```

```sh
CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - a new loan appears in the catalog
ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 3
# fail 0
```

#### What This Test Does Not See

The catalog boundary runs for real, the notification boundary is still a spy object. This
leaves one defect class untouched. Take a version that writes the member's type instead
of the member's id to the notification recipient.

```js
// buggy.mjs — a version that writes the member's type instead of the member's id to the notification recipient
import { RULES } from './service.mjs';

export function createBuggyService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      const today = clock();
      const record = { memberId: member.id, bookId, branch: member.branch, borrowedDay: today, dueDay: today + rule.loanDays };
      catalog.add(record);
      notification.send(member.type, `due date: ${record.dueDay}`);
      return record;
    },
  };
}
```

The test verifies the notification too: a notification must have gone out and its text
must carry the correct due date.

```js
// missed.test.mjs — catalog is real, notification is a spy; the recipient defect is not tested
import test from 'node:test';
import assert from 'node:assert/strict';
import { sqliteCatalog } from './catalogs.mjs';
import { createBuggyService } from './buggy.mjs';

test('a loan record is written and a notification is sent', () => {
  const catalog = sqliteCatalog();
  const sent = [];
  const notification = { send: (recipient, text) => sent.push({ recipient, text }) };
  const service = createBuggyService({ catalog, notification, clock: () => 1000 });

  service.lendBook({ id: 'U-17', type: 'member', branch: 'central' }, 'K-903');

  assert.equal(catalog.find('K-903').memberId, 'U-17');
  assert.equal(sent.length, 1);
  assert.match(sent[0].text, /due date: 1014/);
  console.log(`recipient sent to: ${sent[0].recipient}`);
});
```

```sh
node --test --test-reporter=tap missed.test.mjs | grep -E '^ *(ok|not ok|# (recipient|tests|pass|fail))'
```

```
# recipient sent to: member
ok 1 - a loan record is written and a notification is sent
# tests 1
# pass 1
# fail 0
```

The test is green, the recipient is wrong. The name of the missed defect class is
**recipient identity mismatch**: the real notification channel would have returned an
error for a recipient it did not recognize, while the spy object records whatever value
it is given. Catching this class also requires running the notification boundary for
real; making the catalog boundary real contributes nothing to it. The rule that follows
spreads across the whole course: the defect class a test sees is set by the boundary it
runs for real.

#### Cost

The real boundary's counterpart can be measured. Two quantities are run-independent — the
query count per run and the fixture's line count — a third, time, is machine-dependent
and is reported as a comparison rather than as a raw millisecond figure.

```js
// cost.mjs — the two configurations' query count, fixture line count, and relative time
import { createFixture, MEMBER } from './fixture.mjs';
import { fakeCatalog, sqliteCatalog } from './catalogs.mjs';

const ROUNDS = 500;
const lines = (fn) => fn.toString().split('\n').length - 2;

function measure(configuration) {
  const first = createFixture(configuration);
  first.service.lendBook(MEMBER, 'K-903');
  first.service.checkIn('K-903');
  const start = performance.now();
  for (let i = 0; i < ROUNDS; i += 1) {
    const { service } = createFixture(configuration);
    service.lendBook(MEMBER, 'K-903');
    service.checkIn('K-903');
  }
  return { time: performance.now() - start, queries: first.catalog.queryCount() };
}

const fake = measure('fake');
const real = measure('real');

console.log(`fake catalog: ${ROUNDS} rounds, ${fake.queries} queries per round, catalog ${lines(fakeCatalog)} lines`);
console.log(`real catalog: ${ROUNDS} rounds, ${real.queries} queries per round, catalog ${lines(sqliteCatalog)} lines`);
console.log(`real boundary took longer        : ${real.time > fake.time}`);
console.log(`time difference at least tenfold : ${real.time / fake.time >= 10}`);
```

```
fake catalog: 500 rounds, 0 queries per round, catalog 8 lines
real catalog: 500 rounds, 4 queries per round, catalog 17 lines
real boundary took longer        : true
time difference at least tenfold : true
```

One loan-and-return round runs four queries at the real boundary; the fake boundary runs
none. The catalog implementation grows from eight lines to seventeen, and those nine
extra lines are now fixture code that needs maintenance: if the schema changes, this
place changes too. The absolute value of the time difference depends on the machine and
the file system, not its direction.

These three numbers carry a decision. Four queries and a tenfold time cost were paid in
exchange for one defect class — schema mismatch. Paying that same cost for the clock
boundary buys no defect class; that is why the clock stays fake.

#### Summary

- Which boundary runs for real is a decision made dependency by dependency; four
  dependencies produce sixteen configurations and not all of them are meaningful.
- A fake catalog has no acceptance rule; a real schema does. The service that stayed
  green through the whole previous course failed two tests against the real schema, and
  the defect's name is schema mismatch.
- The defect class a test cannot see is set by the boundary it does not run for real: the
  notification stayed a spy object, so recipient identity mismatch stayed green.
- The boundary's cost is measured in run-independent quantities: four queries per round
  and nine lines of extra fixture code.
- Every lesson in this course writes the same triplet: the defect class caught, the
  defect class missed, and the cost.

#### Next Step

The catalog boundary ran for real in-process, because the catalog's real version could
live in the same process. The notification channel's real version cannot: it is a
separate process, it listens on a port, and it needs to be up before the test and torn
down after. If the catalog itself sits in a file rather than in memory, having the schema
already set up in that file is also a setup step. The next lesson takes up these steps as
test environment management: how a dependent service is brought up, how its readiness is
known, and how many processes and how many steps this fixture's cost per run comes to.

### Test Environment Management

Source: https://academia.sh/en/courses/integration-testing/test-environment-management

Bringing a dependent service up before the test: counting the setup and teardown steps, setting up a readiness probe as a step, catching the previous lesson's missed defect class at the real boundary, and showing why a process-lifetime-bound defect gets past it.

In the previous lesson the catalog boundary ran for real, because the catalog's real
version could live in the same process. The notification channel's real version cannot:
it is a separate process, it listens on a port, and it reads its own data. This requires
putting a **test environment** around the test — a fixture that is set up before the
test, torn down after it, and whose cost is paid again on every run.

This lesson builds that fixture step by step and counts the steps. The environment it
sets up catches the previous lesson's missed defect class — recipient identity mismatch.
The class it misses is just as visible, and is shown.

#### The Dependent Service

The notification channel is a small service that looks the recipient up in the member
table. It returns 404 for a recipient it does not recognize and queues one it does with
202. The health endpoint exists so the environment can be asked whether it is ready.

```js
// notification-service.mjs — the dependent service: reads the member table, returns 404 for an unknown recipient
import { createServer } from 'node:http';
import { DatabaseSync } from 'node:sqlite';

export function createNotificationService(port, catalogPath) {
  const db = new DatabaseSync(catalogPath);
  const queue = [];
  const server = createServer((request, response) => {
    if (request.url === '/health') {
      response.writeHead(200, { 'content-type': 'application/json' });
      return response.end('{"status":"ready"}');
    }
    if (request.method === 'GET' && request.url === '/notification') {
      response.writeHead(200, { 'content-type': 'application/json' });
      return response.end(JSON.stringify({ count: queue.length }));
    }
    let body = '';
    request.on('data', (p) => { body += p; });
    request.on('end', () => {
      const { recipient, text } = JSON.parse(body);
      const member = db.prepare('SELECT member_no FROM member WHERE member_no = ?').get(recipient);
      if (member === undefined) {
        response.writeHead(404, { 'content-type': 'application/json' });
        return response.end(`{"error":"unknown recipient: ${recipient}"}`);
      }
      queue.push({ recipient, text });
      response.writeHead(202, { 'content-type': 'application/json' });
      response.end('{"status":"queued"}');
    });
  });
  server.listen(port, '127.0.0.1');
  return server;
}

if (process.argv[2] !== undefined) createNotificationService(Number(process.argv[2]), process.argv[3]);
```

#### The Environment's Steps

The environment consists of four steps and their order is binding: a temporary directory
is opened, the schema and seed data are written, the process is started, and the
readiness probe runs. Teardown is two steps.

```js
// environment.mjs — test environment: temporary directory, schema, dependent process, and readiness probe
import { spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { DatabaseSync } from 'node:sqlite';

export const STEPS = ['temporary directory', 'schema and seed data', 'process startup', 'readiness probe'];
export const SCHEMA = `
CREATE TABLE member (member_no TEXT PRIMARY KEY, branch TEXT NOT NULL);
CREATE TABLE loan (
  book_no TEXT PRIMARY KEY, member_no TEXT NOT NULL, branch TEXT NOT NULL,
  borrow_day INTEGER NOT NULL, due_day INTEGER NOT NULL);
INSERT INTO member VALUES ('U-17', 'central');`;

async function waitUntilReady(base, maxAttempts = 100) {
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    try {
      const r = await fetch(`${base}/health`);
      if (r.ok) { await r.text(); return attempt; }
    } catch { /* process is not listening yet */ }
    await new Promise((c) => setTimeout(c, 20));
  }
  throw new Error('dependent service did not become ready');
}

export async function createEnvironment(port) {
  const root = mkdtempSync(join(tmpdir(), 'env-'));
  const catalogPath = join(root, 'catalog.db');
  const db = new DatabaseSync(catalogPath);
  db.exec(SCHEMA);
  db.close();
  const base = `http://127.0.0.1:${port}`;
  const start = () => spawn(process.execPath, ['notification-service.mjs', String(port), catalogPath],
    { stdio: 'ignore' });
  let child = start();
  await waitUntilReady(base);
  return {
    base, catalogPath, processCount: 1,
    restart: async () => {
      child.kill('SIGTERM');
      await new Promise((c) => setTimeout(c, 200));
      child = start();
      await waitUntilReady(base);
    },
    close: () => { child.kill('SIGTERM'); rmSync(root, { recursive: true, force: true }); },
  };
}
```

The fourth step can look unnecessary: if the process has started, it is ready. It is
not. Starting a process does not mean the process has started listening on its port;
between the two lies opening the interpreter, loading modules, and opening the database.
What happens when the probe is skipped can be measured.

```js
// not-ready.mjs — when the readiness probe is skipped, process startup races with the first request
import { spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { DatabaseSync } from 'node:sqlite';
import { SCHEMA } from './environment.mjs';

const root = mkdtempSync(join(tmpdir(), 'not-ready-'));
const path = join(root, 'catalog.db');
const db = new DatabaseSync(path);
db.exec(SCHEMA);
db.close();

const child = spawn(process.execPath, ['notification-service.mjs', '8934', path], { stdio: 'ignore' });
try {
  await fetch('http://127.0.0.1:8934/health');
  console.log('no-probe first request: response received');
} catch (error) {
  console.log(`no-probe first request: ${error.cause?.code ?? error.code}`);
}
child.kill('SIGTERM');
rmSync(root, { recursive: true, force: true });
```

```
no-probe first request: ECONNREFUSED
```

This is the environment management's own defect class and its name is **readiness
race**. Putting a fixed wait in place of a probe does not solve the problem, it only
hides it: when the wait is too short the test fails, when it is kept long every run pays
that time. A probe has the environment itself answer for the duration.

#### Catching the Missed Class

The notification is no longer a spy object, it is a real client.

```js
// client.mjs — the real implementation of the notification boundary: goes to the configured address
export function createNotificationClient(base) {
  return {
    send: async (recipient, text) => {
      const response = await fetch(`${base}/notification`, {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({ recipient, text }),
      });
      if (response.status !== 202) {
        const body = await response.json();
        throw new Error(`notification rejected: ${response.status} ${body.error}`);
      }
    },
  };
}
```

Two versions of the loan service are tested: the previous lesson's buggy version, which
writes the member's type as the recipient, and the fixed version, which writes the
member's id. Because the notification call now goes outside the process, `lendBook` is
asynchronous now; this is a consequence of the boundary itself.

```js
// services.mjs — two versions of the loan service: writes the member id or the member type as the recipient
import { DatabaseSync } from 'node:sqlite';

export const RULES = { student: { loanDays: 28 }, member: { loanDays: 14 } };

export function fileCatalog(path) {
  const db = new DatabaseSync(path);
  return {
    add: (r) => db.prepare('INSERT INTO loan VALUES (?, ?, ?, ?, ?)')
      .run(r.bookId, r.memberId, r.branch, r.borrowedDay, r.dueDay),
    find: (bookId) => db.prepare('SELECT member_no FROM loan WHERE book_no = ?').get(bookId),
  };
}

function createService({ catalog, notification, clock }, selectRecipient) {
  return {
    async lendBook(member, bookId) {
      const today = clock();
      const record = {
        memberId: member.id, bookId, branch: member.branch,
        borrowedDay: today, dueDay: today + RULES[member.type].loanDays,
      };
      catalog.add(record);
      await notification.send(selectRecipient(member), `due date: ${record.dueDay}`);
      return record;
    },
  };
}

export const buggyService = (deps) => createService(deps, (member) => member.type);
export const fixedService = (deps) => createService(deps, (member) => member.id);
```

The test hooks the environment to setup and teardown callbacks. The environment is set
up once, a single test uses it, and teardown runs in every case.

```js
// environment.test.mjs — the notification boundary runs for real; the SERVICE variable picks which version is tested
import test, { before, after } from 'node:test';
import assert from 'node:assert/strict';
import { createEnvironment } from './environment.mjs';
import { createNotificationClient } from './client.mjs';
import { fileCatalog, buggyService, fixedService } from './services.mjs';

const MEMBER = { id: 'U-17', type: 'member', branch: 'central' };
let environment;

before(async () => { environment = await createEnvironment(8931); });
after(() => environment.close());

test('a notification is accepted when a loan is made', async () => {
  const build = process.env.SERVICE === 'fixed' ? fixedService : buggyService;
  const service = build({
    catalog: fileCatalog(environment.catalogPath),
    notification: createNotificationClient(environment.base),
    clock: () => 1000,
  });
  await service.lendBook(MEMBER, 'K-903');
  const queue = await (await fetch(`${environment.base}/notification`)).json();
  assert.equal(queue.count, 1);
});
```

```sh
SERVICE=buggy node --test --test-reporter=tap environment.test.mjs | grep -E '^ *(ok|not ok|error:|# (tests|pass|fail))'
```

```
not ok 1 - a notification is accepted when a loan is made
  error: 'notification rejected: 404 unknown recipient: member'
# tests 1
# pass 0
# fail 1
```

In the previous lesson this defect stayed green: the spy object accepted whatever
recipient it was given. The real service does not accept it, and it writes the reason
for the rejection. The fixed version passes in the same environment.

```sh
SERVICE=fixed node --test --test-reporter=tap environment.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - a notification is accepted when a loan is made
# tests 1
# pass 1
# fail 0
```

#### What This Environment Does Not See

The environment brings the process up once and takes it down after the test. Every run
in between passes within a single process lifetime. This leaves one defect class
untouched: accepted notifications sit in memory only.

```js
// persistence.mjs — the state while the test is green versus the state after the process restarts
import { createEnvironment } from './environment.mjs';

const environment = await createEnvironment(8932);
const send = (recipient) => fetch(`${environment.base}/notification`, {
  method: 'POST', headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ recipient, text: 'due date: 1014' }),
});
const queueCount = async () => (await (await fetch(`${environment.base}/notification`)).json()).count;

await send('U-17');
console.log(`queue while the test is green    : ${await queueCount()}`);
await environment.restart();
console.log(`queue after the process restarts : ${await queueCount()}`);
console.log(`catalog file survived            : ${environment.catalogPath.endsWith('catalog.db')}`);
environment.close();
```

```
queue while the test is green    : 1
queue after the process restarts : 0
catalog file survived            : true
```

The test is green, the queue is volatile. The name of the missed defect class is
**process-lifetime state**: the dependency's data does not survive a restart, and a
single-lifetime environment never asks this in any run. Catching this class requires
adding a fifth step to the environment — restarting the dependency mid-run — and that
step's cost is paid on every run too.

#### Cost

The environment's cost is written as three run-independent quantities and one time comparison; the
absolute value of the multiplier depends on the machine and the file system, the direction of the
difference does not.

```js
// cost.mjs — the real boundary's cost: process, setup step, and time relative to a spy object
import { createEnvironment, STEPS } from './environment.mjs';
import { createNotificationClient } from './client.mjs';

const N = 100;
const environment = await createEnvironment(8933);
const real = createNotificationClient(environment.base);
const spy = { send: async () => {} };

async function measure(notification) {
  const start = performance.now();
  for (let i = 0; i < N; i += 1) await notification.send('U-17', 'due date: 1014');
  return performance.now() - start;
}

await measure(real);
const spyTime = await measure(spy);
const realTime = await measure(real);
environment.close();

console.log(`process brought up : ${environment.processCount}`);
console.log(`setup steps        : ${STEPS.length} (${STEPS.join(', ')})`);
console.log(`teardown steps     : 2 (process termination, temporary directory removal)`);
console.log(`file per run       : 1 (catalog file)`);
console.log(`${N} notifications, real boundary took longer  : ${realTime > spyTime}`);
console.log(`time difference at least twentyfold           : ${realTime / spyTime >= 20}`);
```

```
process brought up : 1
setup steps        : 4 (temporary directory, schema and seed data, process startup, readiness probe)
teardown steps     : 2 (process termination, temporary directory removal)
file per run       : 1 (catalog file)
100 notifications, real boundary took longer  : true
time difference at least twentyfold           : true
```

One process, four setup steps, two teardown steps, and one file per run. Each of these
six steps needs maintenance: the second step is rewritten when the schema changes, the
third when the service's startup interface changes, the fourth when the health endpoint
changes.

A tool class exists that runs the same steps inside an isolated **container**. A
container does not reduce the number of steps — a temporary container replaces the
temporary directory, container startup replaces process startup — but it changes two
things: the dependency's version and configuration stay the same from run to run, and
restarting the dependency becomes cheap. The missed defect class above becomes exactly
the kind that can be caught there. In exchange, every run pays the cost of pulling an
image or reading from a cache.

#### Summary

- A test environment is a fixture set up and torn down around the test; here it consists
  of four setup steps and two teardown steps.
- The readiness probe is a step: skipping it makes the first request come back with a
  connection refusal, and a fixed wait put in its place produces either flakiness or a
  fixed delay.
- The real notification boundary caught the previous lesson's missed defect class: an
  unknown recipient was rejected with 404 and the test turned red.
- A single-lifetime environment cannot see process-lifetime state; accepted
  notifications vanished after a restart, and the test was still green.
- The cost is one process, six steps, and one file per run; a container-based tool class
  does not remove these steps, it trades them for version stability and cheap restarts.

#### Next Step

The seed data written together with this environment's schema was a single row: one
member, one branch. The tests passed because the questions asked could be answered with
a single row. The catalog in production is not like this — the number of loans per
member shows a wide distribution, records are linked to one another, and constraints are
violated only in particular combinations. The next lesson takes up the test data itself:
which property production-like data has to preserve, which property breaks when
personal fields are masked, and what defect class a single-row seed cannot see.

### Test Data Management

Source: https://academia.sh/en/courses/integration-testing/test-data-management

Measuring which property production-like test data has to preserve: a report defect caught by the distribution's tail rather than volume, the constraint, distribution, and relationship properties masking breaks, and the defect class the masked value destroys.

The previous lesson's environment was set up with a single-row seed: one member, one
branch. The tests passed because the questions asked could be answered with a single
row. The catalog in production is not like this, and the difference is not volume
alone.

This lesson's question is: **which property** does test data have to preserve? The
answer is not "resemble production," because that cannot be measured. There are three
properties that can be measured — distribution, constraint, and relationship — and each
makes a different defect class visible. Whichever of these three properties the masking
of personal fields breaks, it takes along with it the defect class bound to that
property.

#### The Data Generator

The generator derives catalog data from a seed with the linear congruential generator
carried over from the previous course; as long as the seed is visible, the same data
can be reproduced. The `mode` argument changes only one thing: whether a member can
have more than one overdue loan.

```js
// generator.mjs — catalog data derived from a seed; mode decides the distribution's tail
export const SCHEMA = `
CREATE TABLE member (
  member_no TEXT PRIMARY KEY, name TEXT NOT NULL,
  email TEXT NOT NULL UNIQUE, branch TEXT NOT NULL);
CREATE TABLE loan (
  book_no TEXT PRIMARY KEY, member_no TEXT NOT NULL REFERENCES member(member_no),
  borrowed_day INTEGER NOT NULL, due_day INTEGER NOT NULL, returned_day INTEGER);`;

export function createGenerator(seed) {
  let state = seed;
  return () => { state = (state * 1103515245 + 12345) % 2147483648; return state / 2147483648; };
}

const NAMES = ['Alice', 'Marcus', 'Elena', 'Kevin', 'Sylvia', 'Grace', 'Owen', 'Nadia'];
const SURNAMES = ['Kane, Marsh', 'Reyes', 'Ashford', 'Renner', 'Doyle'];
const BRANCHES = ['central', 'campus', 'arts'];

export function generateData(seed, memberCount, mode = 'tailed') {
  const random = createGenerator(seed);
  const members = [];
  const loans = [];
  let bookNo = 0;
  for (let i = 0; i < memberCount; i += 1) {
    const id = `U-${String(i).padStart(4, '0')}`;
    members.push({
      id,
      name: `${NAMES[Math.floor(random() * NAMES.length)]} ${SURNAMES[Math.floor(random() * SURNAMES.length)]}`,
      email: `member${i}@example.invalid`,
      branch: BRANCHES[Math.floor(random() * BRANCHES.length)],
    });
    const tailed = random() > 0.85;
    const count = tailed ? 6 + Math.floor(random() * 5) : 1 + Math.floor(random() * 2);
    let overdueCount = 0;
    for (let k = 0; k < count; k += 1) {
      const borrowed = 900 + Math.floor(random() * 80);
      // in flat mode each member has at most one overdue loan
      const late = random() > 0.55 && (mode === 'tailed' || overdueCount === 0);
      if (late) overdueCount += 1;
      loans.push({
        bookId: `K-${String(bookNo += 1).padStart(5, '0')}`,
        memberId: id,
        borrowedDay: borrowed,
        dueDay: borrowed + 14,
        returnedDay: late ? null : borrowed + 5,
      });
    }
  }
  return { members, loans };
}
```

The work under test is the report that counts members with an overdue loan. The buggy
version counts loan rows, not members. The expected result is computed from the
generated data, independent of the query.

```js
// load.mjs — loads the generated data into node:sqlite and queries the overdue-member report in two versions
import { DatabaseSync } from 'node:sqlite';
import { SCHEMA, generateData } from './generator.mjs';

export const TODAY = 1014;

export function createDatabase({ members, loans }, path = ':memory:') {
  const db = new DatabaseSync(path);
  db.exec('PRAGMA foreign_keys = ON;');
  db.exec(SCHEMA);
  const insertMember = db.prepare('INSERT INTO member VALUES (?, ?, ?, ?)');
  const insertLoan = db.prepare('INSERT INTO loan VALUES (?, ?, ?, ?, ?)');
  let queries = 0;
  for (const m of members) { insertMember.run(m.id, m.name, m.email, m.branch); queries += 1; }
  for (const l of loans) {
    insertLoan.run(l.bookId, l.memberId, l.borrowedDay, l.dueDay, l.returnedDay);
    queries += 1;
  }
  return { db, queries };
}

const BUGGY = `SELECT COUNT(*) AS member_count FROM member m
  JOIN loan l ON l.member_no = m.member_no
  WHERE l.returned_day IS NULL AND l.due_day < ?`;
const FIXED = BUGGY.replace('COUNT(*)', 'COUNT(DISTINCT m.member_no)');

export function overdueMemberCount(db, today, version) {
  return db.prepare(version === 'fixed' ? FIXED : BUGGY).get(today).member_count;
}

// test oracle: the count is computed from the generated data, independent of SQL
export function expectedOverdueMembers({ loans }, today) {
  const set = new Set();
  for (const l of loans) if (l.returnedDay === null && l.dueDay < today) set.add(l.memberId);
  return set.size;
}

export const DATA_SETS = {
  minimal: () => ({
    members: [{ id: 'U-0000', name: 'Alice Kane', email: 'member0@example.invalid', branch: 'central' }],
    loans: [{ bookId: 'K-00001', memberId: 'U-0000', borrowedDay: 900, dueDay: 914, returnedDay: null }],
  }),
  flat: () => generateData(20260731, 200, 'flat'),
  tailed: () => generateData(20260731, 200, 'tailed'),
};
```

#### Not Volume, the Tail

The two generated sets are exactly equal in volume. The only place they diverge is the
distribution of overdue loans per member.

```js
// distribution.mjs — two data sets of the same volume: one tailed, one flat
import { DATA_SETS, TODAY } from './load.mjs';

for (const label of ['flat', 'tailed']) {
  const data = DATA_SETS[label]();
  const counter = new Map();
  for (const l of data.loans) {
    if (l.returnedDay === null && l.dueDay < TODAY) counter.set(l.memberId, (counter.get(l.memberId) ?? 0) + 1);
  }
  const overdueLoans = [...counter.values()].reduce((a, b) => a + b, 0);
  const max = Math.max(...counter.values());
  console.log(`${label.padEnd(8)}: ${data.members.length} members, ${data.loans.length} loans, ` +
    `${counter.size} overdue members, ${overdueLoans} overdue loans, at most ${max} per member`);
}
```

```
flat    : 200 members, 572 loans, 143 overdue members, 143 overdue loans, at most 1 per member
tailed  : 200 members, 572 loans, 143 overdue members, 297 overdue loans, at most 8 per member
```

The same test is run on three sets.

```js
// report.test.mjs — the same report test on three data sets; picked by the DATA and REPORT variables
import test from 'node:test';
import assert from 'node:assert/strict';
import { createDatabase, overdueMemberCount, expectedOverdueMembers, DATA_SETS, TODAY } from './load.mjs';

const dataName = process.env.DATA ?? 'minimal';
const version = process.env.REPORT ?? 'buggy';

test(`overdue member count is correct (${dataName})`, () => {
  const data = DATA_SETS[dataName]();
  const { db } = createDatabase(data);
  assert.equal(overdueMemberCount(db, TODAY, version), expectedOverdueMembers(data, TODAY));
});
```

```sh
for v in minimal flat tailed; do
  DATA=$v REPORT=buggy node --test --test-reporter=tap report.test.mjs | grep -E '^ *(ok|not ok)'
done
DATA=tailed REPORT=fixed node --test --test-reporter=tap report.test.mjs | grep -E '^ *(ok|not ok)'
```

```
ok 1 - overdue member count is correct (minimal)
ok 1 - overdue member count is correct (flat)
not ok 1 - overdue member count is correct (tailed)
ok 1 - overdue member count is correct (tailed)
```

The second line is this lesson's central measurement. The flat set, though two hundred
times the size of the minimal set, could not see the defect; what caught it was not
volume, but a member having more than one overdue loan. The name of the caught defect
class is **distribution-dependent defect**, and the property that has to be preserved
is the distribution's tail. The fix is writing `COUNT(DISTINCT m.member_no)` in place of
`COUNT(*)`; the fourth line shows the fixed version passing on the same set.

#### The Property Masking Breaks

Personal fields are masked when production data is carried into the test. The masking
form determines which property survives. Four forms are tried on the same source set.

```js
// masking.mjs — four masking forms and the property each one breaks
import { createDatabase, DATA_SETS } from './load.mjs';

const source = DATA_SETS.tailed();

const FORMS = {
  'fixed email  ': (v) => ({ ...v, members: v.members.map((m) => ({ ...m, email: 'hidden@example.invalid' })) }),
  'fixed branch ': (v) => ({ ...v, members: v.members.map((m) => ({ ...m, branch: 'hidden' })) }),
  'new member id': (v) => ({ ...v, members: v.members.map((m) => ({ ...m, id: `M-${m.id.slice(2)}` })) }),
  'derived      ': (v) => ({
    ...v,
    members: v.members.map((m) => ({
      ...m, name: `Member-${m.id.slice(2)}`, email: `u${m.id.slice(2)}@example.invalid`,
    })),
  }),
};

const branchCount = (db) => db.prepare('SELECT COUNT(DISTINCT branch) AS n FROM member').get().n;

console.log(`source        : loaded, distinct-branch count ${branchCount(createDatabase(source).db)}`);
for (const [label, mask] of Object.entries(FORMS)) {
  try {
    const { db } = createDatabase(mask(source));
    console.log(`${label} : loaded, distinct-branch count ${branchCount(db)}`);
  } catch (error) {
    console.log(`${label} : failed to load, ${error.message}`);
  }
}
```

```
source        : loaded, distinct-branch count 3
fixed email   : failed to load, UNIQUE constraint failed: member.email
fixed branch  : loaded, distinct-branch count 1
new member id : failed to load, FOREIGN KEY constraint failed
derived       : loaded, distinct-branch count 3
```

Three breakages correspond to three separate properties. The fixed-email mask breaks
the **constraint**: the uniqueness constraint even blocks loading into the database.
The fixed-branch mask breaks the **distribution**: the data loads, but every
branch-broken-down report now sees a single branch, and no branch-dependent defect
class can be tested. Changing the member id in only one table breaks the
**relationship**: the foreign key constraint stops the load; had the constraint been
off, the join would have silently returned empty. The derived mask preserves all three,
because the masked value is derived from the source value's uniqueness and is applied
in the same form across every table.

#### What This Data Does Not See

The derived mask preserved all three properties, but it did not preserve the masked
field's **content**. When the report is exported, fields are joined with a comma and
not quoted.

```js
// export.mjs — turns the overdue-member report into comma-separated rows; fields are not quoted
import { createDatabase, DATA_SETS, TODAY } from './load.mjs';

const REPORT = `SELECT m.member_no, m.name, m.branch, COUNT(*) AS overdue FROM member m
  JOIN loan l ON l.member_no = m.member_no
  WHERE l.returned_day IS NULL AND l.due_day < ?
  GROUP BY m.member_no, m.name, m.branch ORDER BY m.member_no`;

export function exportRows(rows) {
  return rows.map((r) => [r.member_no, r.name, r.branch, r.overdue].join(','));
}

export function reportRows(format) {
  const source = DATA_SETS.tailed();
  const data = format === 'masked'
    ? { ...source, members: source.members.map((m) => ({ ...m, name: `Member-${m.id.slice(2)}` })) }
    : source;
  const { db } = createDatabase(data);
  return db.prepare(REPORT).all(TODAY);
}
```

```js
// export.test.mjs — each exported row must carry four fields; the data format is picked with a variable
import test from 'node:test';
import assert from 'node:assert/strict';
import { exportRows, reportRows } from './export.mjs';

const format = process.env.DATA_FORMAT ?? 'masked';

test(`exported rows carry four fields (${format})`, () => {
  const rows = exportRows(reportRows(format));
  const broken = rows.filter((r) => r.split(',').length !== 4);
  assert.deepEqual(broken.slice(0, 1), []);
});
```

```sh
DATA_FORMAT=masked node --test --test-reporter=tap export.test.mjs | grep -E '^ *(ok|not ok)'
DATA_FORMAT=source node --test --test-reporter=tap export.test.mjs | grep -E '^ *(ok|not ok)'
node -e "const m = await import('./export.mjs'); console.log(m.exportRows(m.reportRows('source')).find((s) => s.split(',').length !== 4));" --input-type=module
```

```
ok 1 - exported rows carry four fields (masked)
not ok 1 - exported rows carry four fields (source)
U-0004,Sylvia Kane, Marsh,arts,1
```

The test is green with masked data; the same test is red with source data. The name of
the missed defect class is **content-dependent defect**: what triggers the defect is
the comma inside the name field, and the derived mask destroyed that comma. This is
masking's unavoidable cost — a personal field's content cannot be kept in the test. The
counterpart is testing the content shape with a separate rule list: unmasked, generated
names that carry a separator are added to the export test by hand.

#### Cost

```js
// cost.mjs — the cost of production-like data: rows, queries, file size, and relative time
import { mkdtempSync, rmSync, statSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createHash } from 'node:crypto';
import { createDatabase, DATA_SETS } from './load.mjs';

const root = mkdtempSync(join(tmpdir(), 'data-'));
const digest = (data) => createHash('sha256').update(JSON.stringify(data)).digest('hex').slice(0, 12);

function measure(label) {
  const data = DATA_SETS[label]();
  const path = join(root, `${label}.db`);
  const start = performance.now();
  const { queries } = createDatabase(data, path);
  const time = performance.now() - start;
  return { rows: data.members.length + data.loans.length, queries, time, size: statSync(path).size, digest: digest(data) };
}

const minimal = measure('minimal');
const tailed = measure('tailed');
rmSync(root, { recursive: true, force: true });

console.log(`minimal : ${String(minimal.rows).padStart(3)} rows, ${String(minimal.queries).padStart(3)} load queries, ${minimal.size} bytes`);
console.log(`tailed  : ${tailed.rows} rows, ${tailed.queries} load queries, ${tailed.size} bytes`);
console.log(`seed 20260731 gives the same digest  : ${digest(DATA_SETS.tailed()) === tailed.digest}`);
console.log(`tailed set took longer to load       : ${tailed.time > minimal.time}`);
```

```
minimal :   2 rows,   2 load queries, 24576 bytes
tailed  : 772 rows, 772 load queries, 77824 bytes
seed 20260731 gives the same digest  : true
tailed set took longer to load       : true
```

Seven hundred seventy-two load queries and a seventy-eight-kilobyte file are paid again on every
run. This cost's counterpart is a single defect class: distribution-dependent defect. Paying the
same cost with the flat set did not buy that class; volume alone is not a measure. The seed being
visible fixes the cost — instead of a data set stored in a file, an eight-digit number is carried,
and the same data is reproduced.

#### Summary

- Test data has three measurable properties: distribution, constraint, and
  relationship; each makes a separate defect class visible.
- Of two equal-volume sets, only the tailed one caught the report defect; the flat set,
  two hundred times the size, could not see it.
- A fixed-value mask breaks the uniqueness constraint, a fixed-field mask breaks the
  distribution, renumbering in a single table breaks the relationship; the derived mask
  preserves all three.
- The derived mask cannot preserve a field's content: the separator-dependent export
  defect inside the name stayed green with masked data and red with source data.
- The cost is seven hundred seventy-two load queries and seventy-eight kilobytes; as
  long as the seed is visible, the data set is not carried, it is reproduced.

#### Next Step

Every measurement here assumed a single schema: tables ready, columns in place,
constraints defined. The schema is itself a product piece and changes over time — a
column is added, a column is split, a constraint is tightened. These changes are
applied through ordered steps called a **migration**, and each step has an up
direction and a down direction. The next lesson takes the migration itself under test:
what happens when the step order breaks, whether the down migration can preserve data,
and how the schema is verified to be in the expected shape.

### Database Testing

Source: https://academia.sh/en/courses/integration-testing/database-testing

Putting migration steps and the schema under test: verifying the shape the up migration produces, migration-number uniqueness, catching data loss in the down migration, a migration failing on existing data with a constraint violation, and the backfill value a schema test cannot see.

In the previous lesson the schema was a fixed assumption: tables ready, columns in
place, constraints defined. The schema is itself a product piece and it changes through
ordered steps. Each of these steps has an up direction and a down direction, and both
can break just as code can.

This lesson does not teach the database; it **puts the migration and the schema under
test**. The questions are: does the up migration produce the expected shape, is the
step numbering sound, does the down migration preserve data, and what does the
migration do to existing data. Every measurement runs on a real `node:sqlite` database.

#### The Migration Set

There are four steps. The third step's down direction is given in two versions; that is
what gets tested.

```js
// migrations.mjs — ordered migration steps; the third step's down direction is given in two versions
export const DOWN_VERSION = process.env.DOWN_VERSION ?? 'buggy';

export const MIGRATIONS = [
  {
    no: 1,
    name: 'initial schema',
    up: [
      `CREATE TABLE member (member_no TEXT PRIMARY KEY, name TEXT NOT NULL, branch TEXT NOT NULL)`,
      `CREATE TABLE loan (book_no TEXT PRIMARY KEY, member_no TEXT NOT NULL,
         borrowed_day INTEGER NOT NULL, due_day INTEGER NOT NULL)`,
    ],
    down: [`DROP TABLE loan`, `DROP TABLE member`],
  },
  {
    no: 2,
    name: 'branch column on loan',
    up: [
      `ALTER TABLE loan ADD COLUMN branch TEXT`,
      `UPDATE loan SET branch = 'central'`,
    ],
    down: [`ALTER TABLE loan DROP COLUMN branch`],
  },
  {
    no: 3,
    name: 'splitting the member name',
    up: [
      `ALTER TABLE member ADD COLUMN surname TEXT`,
      `UPDATE member SET surname = substr(name, instr(name, ' ') + 1),
         name = substr(name, 1, instr(name, ' ') - 1) WHERE instr(name, ' ') > 0`,
    ],
    down: DOWN_VERSION === 'fixed'
      ? [`UPDATE member SET name = name || ' ' || surname WHERE surname IS NOT NULL AND surname <> ''`,
         `ALTER TABLE member DROP COLUMN surname`]
      : [`ALTER TABLE member DROP COLUMN surname`],
  },
  {
    no: 4,
    name: 'name and surname uniqueness constraint',
    up: [`CREATE UNIQUE INDEX member_name_surname ON member(name, surname)`],
    down: [`DROP INDEX member_name_surname`],
  },
];
```

The runner does three jobs: it checks the number sequence, keeps the applied steps in a
record table, and counts the queries it runs. The counter is the cost's measure.

```js
// runner.mjs — migration runner: checks the number sequence, records what was applied, counts queries
import { DatabaseSync } from 'node:sqlite';
import { MIGRATIONS } from './migrations.mjs';

export function checkOrder(migrations) {
  const numbers = migrations.map((m) => m.no);
  const unique = new Set(numbers);
  if (unique.size !== numbers.length) {
    const repeated = numbers.find((n, i) => numbers.indexOf(n) !== i);
    throw new Error(`migration number repeated: ${repeated}`);
  }
  const ordered = [...unique].sort((a, b) => a - b);
  ordered.forEach((n, i) => {
    if (n !== i + 1) throw new Error(`gap in migration numbering: ${i + 1} missing`);
  });
  return ordered.length;
}

export function openDatabase(path = ':memory:') {
  const db = new DatabaseSync(path);
  db.exec('CREATE TABLE IF NOT EXISTS migration_record (no INTEGER PRIMARY KEY)');
  db.queries = 0;
  return db;
}

export const applied = (db) =>
  db.prepare('SELECT no FROM migration_record ORDER BY no').all().map((s) => s.no);

export function up(db, target) {
  checkOrder(MIGRATIONS);
  for (const migration of MIGRATIONS.filter((m) => m.no <= target)) {
    const done = applied(db);
    if (done.includes(migration.no)) continue;
    if (migration.no > 1 && !done.includes(migration.no - 1)) {
      throw new Error(`migration order broken: ${migration.no - 1} not applied before ${migration.no}`);
    }
    for (const command of migration.up) { db.exec(command); db.queries += 1; }
    db.prepare('INSERT INTO migration_record VALUES (?)').run(migration.no);
    db.queries += 1;
  }
  return db;
}

export function down(db, target) {
  for (const migration of [...MIGRATIONS].reverse().filter((m) => m.no > target)) {
    if (!applied(db).includes(migration.no)) continue;
    for (const command of migration.down) { db.exec(command); db.queries += 1; }
    db.prepare('DELETE FROM migration_record WHERE no = ?').run(migration.no);
    db.queries += 1;
  }
  return db;
}

export const columns = (db, table) =>
  db.prepare(`PRAGMA table_info(${table})`).all().map((s) => `${s.name}:${s.notnull}`);
```

#### Four Tests

The first test verifies the schema's **shape**: which columns, in which order, and
which cannot be left blank. The expected list is written by hand; this list diverges
when a migration adds or forgets a column. The second test asks whether migration
numbers are unique and gap-free; the class it catches is two migrations opened with the
same number on two separate branches. The third test does a round trip. The fourth test
runs the migration against existing data.

```js
// migration.test.mjs — up migration, migration numbering, round trip, and constraint violation tests
import test from 'node:test';
import assert from 'node:assert/strict';
import { MIGRATIONS } from './migrations.mjs';
import { openDatabase, up, down, applied, columns, checkOrder } from './runner.mjs';

const insertMember = (db, no, name, branch) =>
  db.prepare('INSERT INTO member (member_no, name, branch) VALUES (?, ?, ?)').run(no, name, branch);

test('the up migration brings the schema to the expected shape', () => {
  const db = up(openDatabase(), 4);
  assert.deepEqual(applied(db), [1, 2, 3, 4]);
  assert.deepEqual(columns(db, 'member'), ['member_no:0', 'name:1', 'branch:1', 'surname:0']);
  assert.deepEqual(columns(db, 'loan'),
    ['book_no:0', 'member_no:1', 'borrowed_day:1', 'due_day:1', 'branch:0']);
});

test('migration numbers must be unique and gap-free', () => {
  assert.equal(checkOrder(MIGRATIONS), 4);
  const conflicting = [...MIGRATIONS, { no: 4, name: 'migration opened on another branch', up: [], down: [] }];
  assert.throws(() => checkOrder(conflicting), /migration number repeated: 4/);
});

test('the round trip preserves the member name', () => {
  const db = up(openDatabase(), 2);
  insertMember(db, 'U-0001', 'Alice Kane', 'central');
  up(db, 3);
  down(db, 2);
  assert.equal(db.prepare('SELECT name FROM member WHERE member_no = ?').get('U-0001').name, 'Alice Kane');
});

test('the fourth migration fails with a constraint violation on a duplicate name', () => {
  const db = up(openDatabase(), 2);
  insertMember(db, 'U-0001', 'Alice Kane', 'central');
  insertMember(db, 'U-0002', 'Alice Kane', 'campus');
  up(db, 3);
  assert.throws(() => up(db, 4), /UNIQUE constraint failed/);
});
```

```sh
DOWN_VERSION=buggy node --test --test-reporter=tap migration.test.mjs | grep -E '^ *(ok|not ok|(expected|actual):|# (tests|pass|fail))'
```

```
ok 1 - the up migration brings the schema to the expected shape
ok 2 - migration numbers must be unique and gap-free
not ok 3 - the round trip preserves the member name
  expected: 'Alice Kane'
  actual: 'Alice'
ok 4 - the fourth migration fails with a constraint violation on a duplicate name
# tests 4
# pass 3
# fail 1
```

The name of the caught defect class is **data loss in the down migration**. The third
migration's up direction splits the name in two, and its down direction drops the
column carrying the second part directly; since that column is the only place holding
the surname, the down direction leaves the name half-finished. The first test staying
green also says why this class gets past it: the column list looks correct after the
down migration, the value it carries is wrong.

The fix is merging the column's content back before dropping it. This version is
already written in the migration set.

```sh
DOWN_VERSION=fixed node --test --test-reporter=tap migration.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the up migration brings the schema to the expected shape
ok 2 - migration numbers must be unique and gap-free
ok 3 - the round trip preserves the member name
ok 4 - the fourth migration fails with a constraint violation on a duplicate name
# tests 4
# pass 4
# fail 0
```

The fourth test passing in both runs is separate information. The migration adding the
uniqueness constraint fails when it finds a duplicate pair in existing data; the test
fixes this not as a defect but as **documented behavior**. From this it follows that
applying such a migration in production requires a cleanup step ahead of it.

#### What the Schema Test Does Not See

The first test verifies the schema's shape: the `branch` column was added to the `loan`
table, its name and position are correct. **What** the second migration writes into
that column is not asked in any test.

```js
// backfill.mjs — while the schema test is green, what the second migration actually writes into that column
import { openDatabase, up, columns } from './runner.mjs';

const db = up(openDatabase(), 1);
const insertMember = db.prepare('INSERT INTO member (member_no, name, branch) VALUES (?, ?, ?)');
const insertLoan = db.prepare('INSERT INTO loan (book_no, member_no, borrowed_day, due_day) VALUES (?, ?, 900, 914)');
for (const [no, name, branch] of [['U-1', 'Alice Kane', 'central'], ['U-2', 'Marcus Reyes', 'campus'], ['U-3', 'Elena Ashford', 'arts']]) {
  insertMember.run(no, name, branch);
  insertLoan.run(`K-${no}`, no);
}
up(db, 2);

const distribution = (table) => db.prepare(`SELECT branch, COUNT(*) AS n FROM ${table} GROUP BY branch ORDER BY branch`)
  .all().map((s) => `${s.branch}=${s.n}`).join(' ');

console.log(`what the schema test sees      : loan columns ${columns(db, 'loan').join(' ')}`);
console.log(`branch in the member table     : ${distribution('member')}`);
console.log(`loan branch after migration    : ${distribution('loan')}`);
```

```
what the schema test sees      : loan columns book_no:0 member_no:1 borrowed_day:1 due_day:1 branch:0
branch in the member table     : arts=1 campus=1 central=1
loan branch after migration    : central=3
```

Members are in three separate branches, every loan is in the central branch. The second
migration filled the column with a fixed value; it did not read from the member's
branch. The name of the missed defect class is **backfill value mismatch**, and the
schema test cannot see it, because the schema test's oracle is the column list, not row
contents. Catching this class requires building invariants over the data before and
after the migration — something like "every loan's branch equals its member's branch" —
and that claim is tested with the previous lesson's data sets.

#### Cost

```js
// cost.mjs — the migration test's cost: steps, queries, file, and relative time
import { mkdtempSync, rmSync, statSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { MIGRATIONS } from './migrations.mjs';
import { openDatabase, up, down } from './runner.mjs';

const root = mkdtempSync(join(tmpdir(), 'migration-'));
const steps = MIGRATIONS.reduce((t, m) => t + m.up.length + m.down.length, 0);

const fromScratch = performance.now();
const db = up(openDatabase(join(root, 'catalog.db')), 4);
const fromScratchTime = performance.now() - fromScratch;
const upQueries = db.queries;

const roundTrip = performance.now();
down(db, 2);
up(db, 4);
const roundTripTime = performance.now() - roundTrip;

const size = statSync(join(root, 'catalog.db')).size;
rmSync(root, { recursive: true, force: true });

console.log(`migration steps (up + down commands)                        : ${steps}`);
console.log(`queries from scratch to four                                : ${upQueries}`);
console.log(`total after round trip                                      : ${db.queries}`);
console.log(`file created                                                : 1, ${size} bytes`);
console.log(`round trip did not take longer than migrating from scratch  : ${roundTripTime < fromScratchTime * 3}`);
```

```sh
DOWN_VERSION=fixed node cost.mjs
```

```
migration steps (up + down commands)                        : 13
queries from scratch to four                                : 11
total after round trip                                      : 21
file created                                                : 1, 28672 bytes
round trip did not take longer than migrating from scratch  : true
```

Thirteen migration commands, eleven queries to go from scratch to four, twenty-one
queries including the round trip, and one file. These numbers grow linearly as the
migration set grows: every new step brings both an up and a down command, and one
line's change in the first test's expected column list. The maintenance cost is here —
adding one column requires touching three files: the migration set, the expected schema
list, and the queries that read that column.

#### Summary

- The schema test's oracle is a hand-written column list; this list diverges when a
  migration forgets to add or drop a column.
- Migration-number uniqueness is a separate test and it catches migrations opened with
  the same number on two branches.
- The round-trip test caught data loss in the down migration: a down migration that did
  not merge the column's content before dropping it left the name half-finished.
- The migration adding the uniqueness constraint fails when it finds a duplicate pair in
  existing data; the test fixes this as documented behavior.
- The schema test cannot see the backfill value: the column was added, a fixed value was
  written to every row, and the test stayed green.

#### Next Step

In this lesson the dependency that ran for real sat on the same machine: its schema was
on hand, its data could be generated, its migration could be repeated. Not every
dependency is like this. The library's loan flow depends on an external service that
authenticates and returns a book's bibliographic record: that service belongs to
another team, it does not come up with the test's command, it can give a different
answer on every run, and on some runs it does not answer at all. The next lesson takes
up this dependency with record and replay: real responses are recorded once, later runs
are fed from the recording, and which defect class this catches and which it misses is
measured.

### Mocking External Services

Source: https://academia.sh/en/courses/integration-testing/mocking-external-services

Imitating another team's service with record and replay: a run fed from the recording working without a process or a network call, request drift being caught when it finds no match in the cassette, and the response the recording freezes staying green when the contract changes.

In the previous lesson the dependency that ran for real sat on the same machine: its
schema was on hand, its migration could be repeated. The library's loan flow also
depends on a bibliography service, and that service belongs to another team. It does
not come up with the test's command, it can give a different answer on every run, and
on some runs it does not answer at all.

The approach used for this dependency is called **record and replay**: real responses
are recorded once, and later runs are fed from the recording. The recording file is
called a **cassette**. This lesson measures the defect classes the cassette catches and
the ones it misses.

#### The External Service and the Transport

The bibliography service is represented here by a local process; in reality it sits on
the other end of the network. It has two versions, and the difference between them is a
single field name — that difference is measured at the end of the lesson.

```js
// bibliography-service.mjs — the local process standing in for another team's bibliography service
import { createServer } from 'node:http';

const BOOKS = {
  'K-903': { author: 'Sabahattin Ali', year: 1943 },
  'K-904': { author: 'Halide Edip', year: 1922 },
};

export function createBibliographyService(port, version) {
  const server = createServer((request, response) => {
    const url = new URL(request.url, 'http://local');
    const no = url.searchParams.get('no');
    const book = BOOKS[no];
    if (url.pathname !== '/bibliography' || book === undefined) {
      response.writeHead(404, { 'content-type': 'application/json' });
      return response.end(`{"error":"not found: ${no}"}`);
    }
    // v2 renamed the title field to name
    const body = version === 'v2'
      ? { no, name: `book ${no}`, author: book.author, year: book.year }
      : { no, title: `book ${no}`, author: book.author, year: book.year };
    response.writeHead(200, { 'content-type': 'application/json' });
    response.end(JSON.stringify(body));
  });
  server.listen(port, '127.0.0.1');
  return server;
}

if (process.argv[2] !== undefined) createBibliographyService(Number(process.argv[2]), process.argv[3] ?? 'v1');
```

The transport runs in two modes. In record mode it goes to the real service and appends
every interaction to the cassette; in replay mode it never goes out to the network, it
looks the request up in the cassette. The key is built from the method and the full
path — including the query fields.

```js
// cassette.mjs — record and replay: record mode goes to the real service, replay mode reads from the file
import { existsSync, readFileSync, writeFileSync } from 'node:fs';

export function createCassetteClient({ mode, cassettePath, base }) {
  const cassette = existsSync(cassettePath) ? JSON.parse(readFileSync(cassettePath, 'utf8')) : [];
  let networkCalls = 0;
  return {
    async request(path) {
      const key = `GET ${path}`;
      if (mode === 'replay') {
        const entry = cassette.find((c) => c.key === key);
        if (entry === undefined) throw new Error(`no match in cassette: ${key}`);
        return { status: entry.status, body: entry.body };
      }
      const response = await fetch(`${base}${path}`);
      networkCalls += 1;
      const body = await response.json();
      cassette.push({ key, status: response.status, body });
      writeFileSync(cassettePath, `${JSON.stringify(cassette, null, 1)}\n`);
      return { status: response.status, body };
    },
    metrics: () => ({ networkCalls, interactions: cassette.length }),
  };
}
```

```js
// ready.mjs — waits for the service to start listening
export async function waitUntilReady(base, maxAttempts = 100) {
  for (let i = 0; i < maxAttempts; i += 1) {
    try { await (await fetch(`${base}/bibliography?no=K-903`)).text(); return; }
    catch { await new Promise((c) => setTimeout(c, 20)); }
  }
  throw new Error('service did not become ready');
}
```

The bibliography client takes the query field's name from a variable; the buggy version
will change that name.

```js
// client.mjs — the bibliography client; the FIELD_NAME variable picks the query field's name
export const QUERY_FIELD = process.env.FIELD_NAME ?? 'no';

export function createBibliographyClient(transport) {
  return {
    async getBibliography(bookId) {
      const { status, body } = await transport.request(`/bibliography?${QUERY_FIELD}=${bookId}`);
      if (status !== 200) throw new Error(`could not get bibliography: ${status}`);
      return body;
    },
  };
}
```

```js
// bibliography.test.mjs — the same test in three modes: record, replay, and the live service
import test, { before } from 'node:test';
import assert from 'node:assert/strict';
import { createCassetteClient } from './cassette.mjs';
import { createBibliographyClient } from './client.mjs';
import { waitUntilReady } from './ready.mjs';

const mode = process.env.MODE ?? 'replay';
const cassettePath = process.env.CASSETTE ?? 'cassette.json';
const base = `http://127.0.0.1:${process.env.PORT ?? 8941}`;
const createClient = () => createBibliographyClient(createCassetteClient({ mode, cassettePath, base }));

before(async () => { if (mode === 'record') await waitUntilReady(base); });

test('the K-903 bibliography carries a title and an author', async () => {
  const bibliography = await createClient().getBibliography('K-903');
  assert.equal(bibliography.title, 'book K-903');
  assert.equal(bibliography.author, 'Sabahattin Ali');
});

test('the K-904 bibliography carries a year', async () => {
  const bibliography = await createClient().getBibliography('K-904');
  assert.equal(bibliography.year, 1922);
});
```

#### Record and Replay

Recording happens once and requires the real service.

```sh
node bibliography-service.mjs 8941 v1 & SERVICE=$!
MODE=record node --test --test-reporter=tap bibliography.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
kill $SERVICE
echo "cassette interactions: $(node -e "console.log(JSON.parse(require('fs').readFileSync('cassette.json')).length)")"
```

```
ok 1 - the K-903 bibliography carries a title and an author
ok 2 - the K-904 bibliography carries a year
# tests 2
# pass 2
# fail 0
cassette interactions: 2
```

Later runs never call the service at all.

```sh
node --test --test-reporter=tap bibliography.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
echo "bibliography service alive during the run: $(pgrep -f 'bibliography-service.mjs' | wc -l | tr -d ' ')"
```

```
ok 1 - the K-903 bibliography carries a title and an author
ok 2 - the K-904 bibliography carries a year
# tests 2
# pass 2
# fail 0
bibliography service alive during the run: 0
```

The cost paid in the second and third lessons is not paid here: no process comes up,
there is no readiness probe, there is no teardown step. In exchange, a dependency on the
cassette was born, and the cassette's validity is an assumption.

#### The Caught Class: Request Drift

Let the bibliography client's query field be changed from `no` to `book`. The real
service would return 404 for this; the cassette finds no match.

```sh
FIELD_NAME=book node --test --test-reporter=tap bibliography.test.mjs \
  | grep -E "^ *(not ok|error:|# fail)" | sed 's/^ *//;s/^/buggy    /'
FIELD_NAME=no node --test --test-reporter=tap bibliography.test.mjs \
  | grep -E '^ *(ok|# (pass|fail))' | sed 's/^ *//;s/^/fixed    /'
```

```
buggy    not ok 1 - the K-903 bibliography carries a title and an author
buggy    error: 'no match in cassette: GET /bibliography?book=K-903'
buggy    not ok 2 - the K-904 bibliography carries a year
buggy    error: 'no match in cassette: GET /bibliography?book=K-904'
buggy    # fail 2
fixed    ok 1 - the K-903 bibliography carries a title and an author
fixed    ok 2 - the K-904 bibliography carries a year
fixed    # pass 2
fixed    # fail 0
```

The name of the caught defect class is **request drift**. The cassette fixes not only
the responses but also the **requests** that fetch them; because the path and the query
fields sit inside the key, every change on the request side goes unmatched. This is the
cassette's strongest side, and at the same time its noisiest: when the code's call to
the external service genuinely changes too, the same error comes out and the cassette
has to be re-recorded — and re-recording requires the real service again.

#### The Missed Class: Frozen Response

The cassette freezes the service's behavior **at the moment of recording**. When the
team that owns the service changes the response body, the cassette knows nothing about
it.

```sh
node --test --test-reporter=tap bibliography.test.mjs | grep -E '^ *(ok|not ok|# (pass|fail))' | sed 's/^/replay   /'
node bibliography-service.mjs 8942 v2 & SERVICE=$!
MODE=record CASSETTE=cassette-v2.json PORT=8942 node --test --test-reporter=tap bibliography.test.mjs \
  | grep -E '^ *(ok|not ok|# (pass|fail))' | sed 's/^/live v2  /'
kill $SERVICE
rm -f cassette-v2.json
```

```
replay   ok 1 - the K-903 bibliography carries a title and an author
replay   ok 2 - the K-904 bibliography carries a year
replay   # pass 2
replay   # fail 0
live v2  not ok 1 - the K-903 bibliography carries a title and an author
live v2  ok 2 - the K-904 bibliography carries a year
live v2  # pass 1
live v2  # fail 1
```

Same test, same code, two different responses. The service's second version renamed the
`title` field to `name`; the live run sees this, the run fed from the cassette does not.
The name of the missed defect class is **frozen response**: the cassette carries
yesterday's behavior, and that behavior is no longer correct.

This class's weight comes from being silent. Schema mismatch announced itself with a
constraint violation, the migration defect with a failed assertion, request drift with a
missing match. In frozen response, nothing fails — the suite is green, and because it is
green, no one looks.

#### Cost

```js
// cost.mjs — the cost of record and replay modes: process, network calls, file, and relative time
import { spawn } from 'node:child_process';
import { statSync, rmSync } from 'node:fs';
import { createCassetteClient } from './cassette.mjs';
import { createBibliographyClient } from './client.mjs';
import { waitUntilReady } from './ready.mjs';

const N = 100;
const base = 'http://127.0.0.1:8943';
const child = spawn(process.execPath, ['bibliography-service.mjs', '8943', 'v1'], { stdio: 'ignore' });
await waitUntilReady(base);

async function measure(mode, cassettePath) {
  const transport = createCassetteClient({ mode, cassettePath, base });
  const client = createBibliographyClient(transport);
  await client.getBibliography('K-903');
  const start = performance.now();
  for (let i = 0; i < N; i += 1) await client.getBibliography('K-903');
  return { time: performance.now() - start, ...transport.metrics() };
}

const live = await measure('record', 'cassette-cost.json');
const replay = await measure('replay', 'cassette.json');
child.kill('SIGTERM');
const size = statSync('cassette.json').size;
rmSync('cassette-cost.json', { force: true });

console.log(`record mode : 1 process, ${live.networkCalls} network calls (${N} requests + 1 warm-up)`);
console.log(`replay mode : 0 processes, ${replay.networkCalls} network calls, 1 file read`);
console.log(`cassette    : ${replay.interactions} interactions, ${size} bytes`);
console.log(`replay took less time               : ${replay.time < live.time}`);
console.log(`time difference at least twentyfold : ${live.time / replay.time >= 20}`);
```

```
record mode : 1 process, 101 network calls (100 requests + 1 warm-up)
replay mode : 0 processes, 0 network calls, 1 file read
cassette    : 2 interactions, 340 bytes
replay took less time               : true
time difference at least twentyfold : true
```

Replay mode's run cost is the lowest of the three lessons: zero processes, zero network
calls, a single file read. The cost has left the run and moved somewhere else — to the
cassette's **freshness**, at three hundred forty bytes. This cost is not paid on every
run, it is paid when the service changes, and it stays invisible when it is not paid. It
also grows as the cassette grows: two interactions can be checked by eye, two hundred
interactions cannot.

#### Summary

- Record and replay records real responses once and feeds later runs from the cassette;
  in replay mode, processes and network calls are zero.
- The cassette fixes not only responses but requests too; when the query field's name
  changed, both tests found no match in the cassette.
- The response the recording freezes can go stale: when the service's second version
  renamed a field, the live run turned red and the run fed from the cassette stayed
  green.
- Frozen response is a silent defect class; because no test fails, no one looks.
- The cost moves from the run to the cassette's freshness, and it becomes impossible to
  check by hand as the cassette grows.

#### Next Step

The five lessons in this topic asked the same question at different boundaries: which
dependency runs for real, and what is the cost. The cassette reached the edge of the
question. The recording freezes the external service's **yesterday's** behavior; when
the team that owns the service changes its contract, the recording still stays green and
the tests start lying. Refreshing the cassette is not a solution, because there is no
signal that says when refreshing is needed.

What is missing is testing the **contract itself** that the two sides agreed on: do the
fields the consumer expects match the fields the provider gives, and who notices when a
field name changes, and when. The next topic opens with this question and takes up
verifying the request and the response, carrying the expectation over to the provider,
and checking for a breaking change.

## Contract and API Tests

Checking one by one that both sides honor the same contract: verifying the request and response with a schema check, carrying the consumer's expectation to the provider, catching a breaking change before publication, and the deviation risk a mock server carries while it makes parallel development possible.

### API Testing

Source: https://academia.sh/en/courses/integration-testing/api-testing

Checking a live service's response against a schema: catching with a real request the field change a frozen recording cannot see, separating structural mismatch from semantic mismatch, and measuring the check's cost by the fields checked, the requests sent, and the share of time.

The previous lesson set up recording and replaying an external service's response, and it
closed with a warning: a recording freezes the service's behavior from yesterday. When the
catalog service renames a field today, the test that runs against the recording still stays
green, because what it compares against is not the service itself but yesterday's copy.

This lesson builds the check that closes that gap. Its question is not API design; request
methods, status codes, and versioning were established in the Web API Design course and are
not repeated here. The question here is: does a live service's response arrive in the shape
the consumer expects, which check tells you that, which defect class it catches, and which
one it misses.

#### The Silence of a Frozen Record

The library loan system has two services: the **catalog service**, which holds book records,
and the **loan service**, which lends books to members. The loan service reads a book record
from the catalog and looks at four of its fields: status, shelf count, loan duration, and ID.
The script below feeds yesterday's recorded body and today's body into the same decision
function.

```js
// frozen.mjs — yesterday's recorded catalog response vs. today's, fed into the same consumer function
const RECORD = {
  isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
  publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2,
  status: "shelved", loanDays: 14,
};

// The catalog service renamed a field today: loanDays -> loanDurationDays.
const LIVE = { ...RECORD, loanDurationDays: RECORD.loanDays };
delete LIVE.loanDays;

// The loan service's decision function: it reads whatever is in the record.
const loanDecision = (book, today) =>
  book.status !== "shelved" || book.shelfCount < 1
    ? { granted: false, dueDay: null }
    : { granted: true, dueDay: today + book.loanDays };

const TODAY = 20260, common = Object.keys(RECORD).filter((a) => a in LIVE);
for (const [name, body] of [["record", RECORD], ["live", LIVE]]) {
  const decision = loanDecision(body, TODAY);
  console.log(`${name.padEnd(6)} fields ${String(Object.keys(body).length).padStart(2)}` +
    `  granted=${decision.granted}  dueDay=${decision.dueDay}`);
}
const missing = Object.keys(RECORD).filter((a) => !(a in LIVE));
console.log(`overlapping fields ${common.length}/${Object.keys(RECORD).length}; missing from live response: ${missing.join(", ")}`);
console.log(`replay test runs against the record -> dueDay 20274 expected, green`);
```

```
record fields  9  granted=true  dueDay=20274
live   fields  9  granted=true  dueDay=NaN
overlapping fields 8/9; missing from live response: loanDays
replay test runs against the record -> dueDay 20274 expected, green
```

Both bodies carry nine fields and eight of them overlap. But when the loan service works
with the live response, it cannot compute the due day. The record-and-replay setup cannot
see this, because **no request ever reaches the live service**; the caught defect class is
zero. The missing check is one that takes the response from where it actually comes from and
tests it against a written expectation.

#### The Response Schema

A **schema** puts in writing which fields the response carries and in what type. The
definition below has nine fields and sixteen rules; the rules amount to type, pattern, value
set, and lower bound. The checker itself is a few lines and returns a list of mismatches.

```js
// schema.mjs — the catalog response schema and a field-by-field checker
export const BOOK_SCHEMA = {
  isbn: { type: "string", pattern: /^97[89]-\d{10}$/ },
  title: { type: "string" },
  author: { type: "string" },
  publicationYear: { type: "integer", min: 1400 },
  branch: { type: "string", set: ["central", "coast", "hill"] },
  totalCount: { type: "integer", min: 0 },
  shelfCount: { type: "integer", min: 0 },
  status: { type: "string", set: ["shelved", "loaned", "lost"] },
  loanDays: { type: "integer", min: 1 },
};

const typeHolds = (rule, d) => rule.type === "integer"
  ? Number.isInteger(d)
  : typeof d === "string";

// The returned array is empty, or each line is one mismatch.
export function checkAgainstSchema(schema, body) {
  const mismatch = [];
  for (const [name, rule] of Object.entries(schema)) {
    const d = body[name];
    if (d === undefined) { mismatch.push(`${name}: field missing`); continue; }
    if (!typeHolds(rule, d)) { mismatch.push(`${name}: not type ${rule.type} (${typeof d})`); continue; }
    if (rule.pattern && !rule.pattern.test(d)) mismatch.push(`${name}: does not match pattern`);
    if (rule.set && !rule.set.includes(d)) mismatch.push(`${name}: value outside set ${d}`);
    if (rule.min !== undefined && d < rule.min) mismatch.push(`${name}: below ${rule.min}`);
  }
  const extra = Object.keys(body).filter((a) => !(a in schema));
  if (extra.length) mismatch.push(`field not in schema: ${extra.join(", ")}`);
  return mismatch;
}
```

The boundary between what the schema says and what it does not say is this lesson's axis. The
schema says the `shelfCount` field cannot be less than zero; it does not say that number is
the **correct** number.

#### Checking Against the Live Service

The catalog service is a local process and can be deliberately broken with an environment
variable. Three defects are defined: `type` returns the shelf count as a string, `meaning`
writes the total count in place of the shelf count, and `value` bumps the reference book's
seven-day loan duration up to fourteen.

```js
// catalog.mjs — catalog service; deliberately broken by the CATALOG_DEFECT variable
// Usage: node catalog.mjs <port|0>   port 0 picks a free port.
import { createServer } from "node:http";

if (process.argv[2] === undefined) {
  console.log("usage: node catalog.mjs <port|0>");
  process.exit(0);
}

const DEFECT = process.env.CATALOG_DEFECT ?? "none";
const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
    publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2, status: "shelved", loanDays: 14 },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", author: "Knuth",
    publicationYear: 1997, branch: "hill", totalCount: 3, shelfCount: 0, status: "loaned", loanDays: 7 },
};

const output = (k) => {
  if (DEFECT === "type") return { ...k, shelfCount: String(k.shelfCount) };
  if (DEFECT === "meaning") return { ...k, shelfCount: k.totalCount };
  if (DEFECT === "value") return { ...k, loanDays: 14 };
  return { ...k };
};

const server = createServer((request, response) => {
  response.sendDate = false;
  const book = BOOKS[request.url.split("?")[0].replace("/book/", "")];
  response.writeHead(book ? 200 : 404, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(book ? output(book) : { error: "book not found" }));
});

server.listen(Number(process.argv[2]), "127.0.0.1", () =>
  console.log(`ready ${server.address().port} defect=${DEFECT}`));
```

The test file starts the service as a separate process, reads the chosen port from the
process's first line, and checks three things: schema conformance, an **invariant**, and the
consumer's decision.

```js
// api.test.mjs — request/response check against the live catalog service
// Usage: CATALOG_DEFECT=none|type|meaning|value node --test api.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
  proc.stdout.once("data", (v) => {
    base = `http://127.0.0.1:${String(v).split(" ")[1]}`;
    resolve();
  });
}));
after(() => proc.kill());

const book = async (isbn) => (await fetch(`${base}/book/${isbn}`)).json();

test("schema: both book responses match the schema", async () => {
  for (const isbn of ["978-0262033848", "978-0201896831"]) {
    const mismatch = checkAgainstSchema(BOOK_SCHEMA, await book(isbn));
    assert.equal(mismatch.length, 0, `${isbn} -> ${mismatch.join("; ")}`);
  }
});

test("invariant: a loaned book has a shelf count of zero", async () => {
  const k = await book("978-0201896831");
  assert.equal(k.status === "loaned" && k.shelfCount > 0, false,
    `${k.isbn} -> status ${k.status} but shelfCount ${k.shelfCount}`);
});

test("consumer: the due day is computed for a shelved book", async () => {
  const k = await book("978-0262033848");
  assert.equal(k.status, "shelved");
  assert.equal(Number.isInteger(20260 + k.loanDays), true);
});
```

The same set of tests runs against four states of the service: the sound version and three
defects.

```bash
for k in none type meaning value; do
  echo "== defect=$k =="
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs | grep -E '^(ok|not ok|# (pass|fail))'
done
```

```
== defect=none ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0
== defect=type ==
not ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=meaning ==
ok 1 - schema: both book responses match the schema
not ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=value ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0
```

The red-to-green turn is inside this table. When the type defect is introduced, the first
test fails; when the catalog service returns the shelf count as an integer again — that is,
in the `defect=none` row — the same test turns green. This cycle is proof that the check
works.

#### Caught and Missed Classes

The reason a test fails can be read in a single line.

```bash
for k in type meaning value; do
  printf 'defect=%-7s ' "$k"
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs |
    grep -A1 'error: |-' | tail -1 | sed 's/^ *//' | grep . || echo "no failing test"
done
```

```
defect=type    978-0262033848 -> shelfCount: not type integer (string)
defect=meaning 978-0201896831 -> status loaned but shelfCount 3
defect=value   no failing test
```

**The class the schema check catches is structural mismatch:** a field's absence, a change in
its type, or falling outside the pattern or value set. The type defect falls into this class,
and the mismatch is reported by field name. The field rename from the lesson's opening is of
the same class; a schema check that runs against the live service sees it on the very first
request.

**The class the schema check misses is semantic mismatch.** With the meaning defect, the
shelf count is still an integer greater than zero; every rule of the schema is satisfied and
the first test stays green. What catches this defect is not the schema but a hand-written
invariant: a loaned book's shelf count must be zero. The invariant states the relationship
**between** two fields; the schema, on the other hand, treats each field on its own. This is
the cost of closing the missed class — every relationship has to be written by hand, one at a
time.

The third defect falls outside both. The reference book's loan duration should be seven days
but comes back as fourteen; fourteen is an integer the schema accepts and violates no
invariant. All three tests stay green. What's missing here is not a check but **the source of
the expected result**: the only place that knows this book's duration is seven is the
contract itself. The test oracle problem from the Quality and Testing Fundamentals course
resurfaces here, and the next two lessons bring in that source from two different places.

#### The Cost of the Check

Cost is measured two ways: numbers independent of the run, and where the run's time goes.

```js
// cost.mjs — the cost of the API test: how many fields checked, how many requests, where the time goes
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

const ISBN = ["978-0262033848", "978-0201896831"];
const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
const base = `http://127.0.0.1:${port}`;
const REPEAT = 50, bodies = [];

const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) {
  for (const isbn of ISBN) bodies.push(await (await fetch(`${base}/book/${isbn}`)).json());
}
const t1 = performance.now();
for (const g of bodies) checkAgainstSchema(BOOK_SCHEMA, g);
const t2 = performance.now();
proc.kill();

const fields = Object.keys(BOOK_SCHEMA).length;
const rules = Object.values(BOOK_SCHEMA).reduce((n, k) => n + Object.keys(k).length, 0);
console.log(`schema: ${fields} fields, ${rules} rules`);
console.log(`run: ${bodies.length} requests, ${bodies.length * fields} field checks, 2 processes`);
console.log(`request time greater than check time : ${t1 - t0 > t2 - t1}`);
console.log(`ratio at least fifty times           : ${(t1 - t0) / (t2 - t1) >= 50}`);
```

```
schema: 9 fields, 16 rules
run: 100 requests, 900 field checks, 2 processes
request time greater than check time : true
ratio at least fifty times           : true
```

Absolute times are machine-dependent, so the output reports two comparisons instead of raw
milliseconds. What does not change is the direction of the ratio: nearly all of the time goes
into the request round trip, and the schema check itself is negligible. The cost of an API
test is not the complexity of the check but the decision to **step outside the process** —
this suite brings up two processes and sends a request over the network for every test.

The numbers independent of the run are the measure of upkeep. Nine fields and sixteen rules
are hand-written, and when the catalog service adds a field, the schema has to be updated
too; if it is not, the "field not in schema" line fails the test. This is a strict choice, and
a deliberate one: a check that stays silent is indistinguishable from a check that does not
exist.

#### Summary

- Record and replay never sends a request to the live service; a field rename is invisible to
  a test running against the recording, and visible on the very first call to a check that
  runs a real request.
- A schema puts the response's fields in writing with type, pattern, value set, and bound;
  sixteen rules were enough for a nine-field record.
- The class the schema check catches is structural mismatch: a missing field, a type change,
  a value outside the set. The type defect turned the test red, and it went back to green
  once the defect was reverted.
- The class the schema check misses is semantic mismatch: a value that follows the rules but
  is wrong. Only a hand-written invariant can see the relationship between two fields, and
  neither one knows what the correct value actually is.
- Cost: two processes, at least one request per test, sixteen hand-maintained rules; nearly
  all of the run's time goes into the request round trip, and the check itself is negligible.

#### Next Step

This lesson's schema describes the shape the catalog service **produces**. How many of these
fields the loan service actually reads is written nowhere: nine fields were checked, yet the
decision function looked at four. The gap is not pointless — when the provider wants to
remove a field, the schema has no answer to "who reads this." The next lesson gathers the
expectation from the consumer's side instead: the fields the consumer's running code reads
are recorded, a contract is generated from that recording, and the contract runs in the
provider's team. The measure is the gap between the number of fields the consumer actually
uses and the number of fields the contract covers.

### Consumer-Driven Contract Testing

Source: https://academia.sh/en/courses/integration-testing/consumer-driven-contract-testing

Gathering the expectation from the consumer's running code and carrying it to the provider's team: recording the fields read during the run, the generated contract catching a breaking change on the provider's side, and measuring how a branch that never ran stays outside the contract's coverage.

The previous lesson's schema described the shape the catalog service produces: nine fields,
sixteen rules. How many of these fields the loan service reads was written nowhere. The gap
is not pointless; when the provider wants to remove a field, the schema has no answer to
"who reads this."

This lesson gathers the expectation from the other direction. In **consumer-driven contract
testing**, the expectation is written not by the provider but by the consumer, and the
provider runs it in its own team. The Application Layer and Service Interaction course
measured this decision's cost — the number of consumers broken, the number of units
published together, the expand–contract sequence. Here that same decision is turned into a
test, and two numbers are asked: how many fields does the consumer actually read, and how
many of them does the contract cover.

#### Gathering the Expectation From the Consumer's Code

Writing the expectation file by hand has a flaw: the person writing it has to remember what
the code reads. The loan service's decision function splits into three branches, and each
branch reads different fields.

```js
// loan.mjs — the loan service's decision function; three branches split off the catalog record
export function loanDecision(book, today) {
  if (book.status === "lost")
    return { granted: false, message: `lost record: ${book.title} / ${book.author}` };
  if (book.status !== "shelved" || book.shelfCount < 1)
    return { granted: false, message: `not on shelf: ${book.isbn}` };
  return { granted: true, dueDay: today + book.loanDays, message: `${book.isbn} loaned` };
}
```

The catalog service is the previous lesson's process; in this lesson it carries three books
and has two defects. The `rename` defect renames the loan duration field, and the `missing`
defect removes the title field.

```js
// catalog.mjs — catalog service; deliberately broken by the CATALOG_DEFECT variable
// Usage: node catalog.mjs <port|0>   defect: rename (loanDays is renamed)
//                                    defect: missing (title field is removed)
import { createServer } from "node:http";

if (process.argv[2] === undefined) {
  console.log("usage: node catalog.mjs <port|0>");
  process.exit(0);
}

const DEFECT = process.env.CATALOG_DEFECT ?? "none";
const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
    publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2, status: "shelved", loanDays: 14 },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", author: "Knuth",
    publicationYear: 1997, branch: "hill", totalCount: 3, shelfCount: 0, status: "loaned", loanDays: 7 },
  "978-0134685991": { isbn: "978-0134685991", title: "Lost Volume", author: "Bloch",
    publicationYear: 2018, branch: "coast", totalCount: 1, shelfCount: 0, status: "lost", loanDays: 14 },
};

const output = (k) => {
  if (DEFECT === "rename") { const { loanDays, ...rest } = k; return { ...rest, loanDurationDays: loanDays }; }
  if (DEFECT === "missing") { const { title, ...rest } = k; return rest; }
  return { ...k };
};

const server = createServer((request, response) => {
  response.sendDate = false;
  const book = BOOKS[request.url.split("?")[0].replace("/book/", "")];
  response.writeHead(book ? 200 : 404, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(book ? output(book) : { error: "book not found" }));
});

server.listen(Number(process.argv[2]), "127.0.0.1", () =>
  console.log(`ready ${server.address().port} defect=${DEFECT}`));
```

Instead of remembering the expectation, it is possible to **record** it. The body the
consumer reads is placed behind a watcher; the watcher records every field read, by name and
type. The script below runs the two scenarios from the loan service's own test suite against
the real catalog service and generates a contract file from the run.

```js
// collect.mjs — generates a contract from the consumer's run and measures its coverage
import { spawn } from "node:child_process";
import { writeFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
const base = `http://127.0.0.1:${port}`;
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");

// The consumer reads the real response through a watcher; the watcher records the field name it read.
async function run(isbn) {
  const body = await (await fetch(`${base}/book/${isbn}`)).json();
  const read = new Map();
  const watched = new Proxy(body, {
    get: (t, field) => { if (typeof field === "string") read.set(field, typeName(t[field])); return t[field]; },
  });
  return { read, decision: loanDecision(watched, 20260) };
}

const ROUTINE = ["978-0262033848", "978-0201896831"];         // the loan service's own test scenarios
const LOST = "978-0134685991";                                 // lost-book branch: not in the suite
const contract = { consumer: "loan-service", interaction: [] };

for (const isbn of ROUTINE) {
  const { read, decision } = await run(isbn);
  contract.interaction.push({ path: `/book/${isbn}`, fields: Object.fromEntries(read) });
  console.log(`${isbn} -> ${decision.message}\n  fields read ${read.size}: ${[...read.keys()].join(", ")}`);
}
writeFileSync("contract.json", JSON.stringify(contract, null, 1));

const all = new Set(contract.interaction.flatMap((e) => Object.keys(e.fields)));
const covered = new Set(all);
for (const a of (await run(LOST)).read.keys()) all.add(a);
const produced = Object.keys(await (await fetch(`${base}/book/${LOST}`)).json());
proc.kill();

console.log(`fields produced by the provider : ${produced.length}`);
console.log(`fields covered by the contract  : ${covered.size} (${[...covered].join(", ")})`);
console.log(`fields the consumer can read    : ${all.size} (${[...all].join(", ")})`);
console.log(`left outside coverage           : ${[...all].filter((a) => !covered.has(a)).join(", ")}`);
console.log(`contract file ${JSON.stringify(contract).length} bytes, ${contract.interaction.length} interactions`);
```

```
978-0262033848 -> 978-0262033848 loaned
  fields read 4: status, shelfCount, loanDays, isbn
978-0201896831 -> not on shelf: 978-0201896831
  fields read 2: status, isbn
fields produced by the provider : 9
fields covered by the contract  : 4 (status, shelfCount, loanDays, isbn)
fields the consumer can read    : 6 (status, shelfCount, loanDays, isbn, title, author)
left outside coverage           : title, author
contract file 241 bytes, 2 interactions
```

The two scenarios read two different sets of fields. Only two fields being read in the
second scenario is not an accident: once the condition `status !== "shelved"` is true, the
second comparison never runs, and the shelf count is never read either. The contract writes
down not what the code **could** read, but what it **did** read in that run.

The first three numbers are this lesson's measure. The provider produces nine fields, the
contract covers four of them, and the consumer's branches together read six. The gap between
nine and four is the provider's room to maneuver: five fields that nobody reads. The gap
between six and four, on the other hand, is the contract's **blind spot**.

#### The Contract Running on the Provider's Side

The contract file leaves the consumer's repository and enters the provider's team. The
provider brings up its own version and actually sends every interaction in the file.

```js
// provider.test.mjs — the consumer contract runs in the provider's team
// Usage: CATALOG_DEFECT=none|rename|missing node --test provider.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const CONTRACT = JSON.parse(readFileSync("contract.json", "utf8"));
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");
let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
  proc.stdout.once("data", (v) => { base = `http://127.0.0.1:${String(v).split(" ")[1]}`; resolve(); });
}));
after(() => proc.kill());

for (const e of CONTRACT.interaction) {
  test(`${CONTRACT.consumer} ${e.path}`, async () => {
    const body = await (await fetch(base + e.path)).json();
    const missing = Object.entries(e.fields)
      .filter(([name, type]) => body[name] === undefined || typeName(body[name]) !== type)
      .map(([name, type]) => `${name} (expected ${type})`);
    assert.equal(missing.length, 0, `unmet field: ${missing.join(", ")}`);
  });
}
```

```bash
for k in none rename missing; do
  echo "== defect=$k =="
  CATALOG_DEFECT=$k node --test --test-reporter=tap provider.test.mjs |
    grep -E '^(ok|not ok|# (pass|fail))|unmet' | sed 's/^ *//'
done
```

```
== defect=none ==
ok 1 - loan-service /book/978-0262033848
ok 2 - loan-service /book/978-0201896831
# pass 2
# fail 0
== defect=rename ==
not ok 1 - loan-service /book/978-0262033848
unmet field: loanDays (expected integer)
ok 2 - loan-service /book/978-0201896831
# pass 1
# fail 1
== defect=missing ==
ok 1 - loan-service /book/978-0262033848
ok 2 - loan-service /book/978-0201896831
# pass 2
# fail 0
```

**The defect class caught is the loss of a field written into the contract.** When the loan
duration field was renamed, the test turned red and reported by name which field went unmet
on which consumer's path. When the field was put back under its old name — the
`defect=none` row — the test turned green. The value of this signal is that the provider sees
it **before publishing, in its own repository**; the side that would break is another team's
service.

The previous lesson's schema check could not give this information. A schema check says "the
response deviated from the definition"; a contract test says "the loan service's book-reading
path broke."

#### The Branch Left Outside Coverage

The third column stayed silent. When the title field was removed, both tests still passed,
because the title is not in the contract. Yet one of the consumer's branches reads it.

```js
// deviation.mjs — the consumer's lost branch when the contract stays green despite the defect
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

async function run(defect, isbn) {
  const proc = spawn("node", ["catalog.mjs", "0"], { env: { ...process.env, CATALOG_DEFECT: defect } });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const body = await (await fetch(`http://127.0.0.1:${port}/book/${isbn}`)).json();
  proc.kill();
  return loanDecision(body, 20260);
}

for (const defect of ["none", "missing"]) {
  console.log(`defect=${defect.padEnd(7)} lost branch -> ${(await run(defect, "978-0134685991")).message}`);
}
const s = JSON.parse(readFileSync("contract.json", "utf8"));
const covered = new Set(s.interaction.flatMap((e) => Object.keys(e.fields)));
console.log(`contract ${s.interaction.length} interactions, ${covered.size} fields; title covered: ${covered.has("title")}`);
```

```
defect=none    lost branch -> lost record: Lost Volume / Bloch
defect=missing lost branch -> lost record: undefined / Bloch
contract 2 interactions, 4 fields; title covered: false
```

**This is the missed defect class: a field read by a branch that never ran.** The provider's
team is green, and the consumer is broken. The generated contract is a snapshot of the
consumer's test suite; whatever branches the suite runs, the contract covers. M21/K02's
coverage measurement takes on a second meaning here — a gap in branch coverage opens up a gap
not only in its own test, but also in **the provider's decision**.

A missing expectation looks like a dependency that does not exist. The provider believes five
fields are read by no one; the actual number of unread fields is three, because the branch
that reads the title and author fields never entered the contract at all. The contract's
accuracy is exactly as good as the coverage of the run that produced it.

#### The Cost of the Contract

Running the same decision function in two separate fixtures shows where the cost comes from.

```js
// cost.mjs — the cost of contract verification relative to the unit test
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

const CONTRACT = JSON.parse(readFileSync("contract.json", "utf8"));
const FAKE = [                                       // fake catalog: the contract's in-memory counterpart
  { isbn: "978-0262033848", status: "shelved", shelfCount: 2, loanDays: 14 },
  { isbn: "978-0201896831", status: "loaned", shelfCount: 0, loanDays: 7 },
];
const REPEAT = 50;

const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) for (const k of FAKE) loanDecision(k, 20260);
const t1 = performance.now();

const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
for (let i = 0; i < REPEAT; i += 1) {
  for (const e of CONTRACT.interaction) await (await fetch(`http://127.0.0.1:${port}${e.path}`)).json();
}
const t2 = performance.now();
proc.kill();

const fields = CONTRACT.interaction.reduce((n, e) => n + Object.keys(e.fields).length, 0);
console.log(`unit run     : 1 component, 0 requests, ${REPEAT * FAKE.length} decisions`);
console.log(`contract run : 2 components, ${REPEAT * CONTRACT.interaction.length} requests, ${REPEAT * fields} field checks`);
console.log(`contract run longer than unit run : ${t2 - t1 > t1 - t0}`);
console.log(`ratio at least a hundred times     : ${(t2 - t1) / (t1 - t0) >= 100}`);
```

```
unit run     : 1 component, 0 requests, 100 decisions
contract run : 2 components, 100 requests, 300 field checks
contract run longer than unit run : true
ratio at least a hundred times     : true
```

Absolute times are machine-dependent; the direction of the ratio does not change. The point
of the comparison is this: the unit test running against the fake catalog and the contract
run look at the **same four fields**, but one does it in-process, and the other with two
components and a hundred requests. What the cost buys is a single thing: the assumption that
the fake catalog's four fields match the real service's is no longer assumed — it is tested.

The numbers independent of the run give the maintenance side. The contract file is 241 bytes
and carries two interactions; because it is not hand-written, it gets regenerated as the
consumer's tests change. In return, the provider knows that four fields are read **based on a
written record**; for the remaining five fields, the only thing it knows is that they appear
in no contract.

#### Summary

- In a consumer-driven contract, the consumer writes the expectation and the provider runs it
  in its own team; the schema says what is produced, and the contract says what is read.
- The expectation does not have to be written by hand: when the consumer's run is read
  through a watcher, the fields read are recorded by name and type.
- The measure is three numbers: the provider produces nine fields, the contract covers four
  of them, and the consumer's branches together read six.
- The caught class is the loss of a field written into the contract; when the loan duration
  field was renamed, the test turned red, and it turned green again once the name was
  restored.
- The missed class is a field read by a branch that never ran; when the title field was
  removed, the provider stayed green while the consumer's lost-book branch broke.
- Cost: two components, a hundred requests per run, a generated file of 241 bytes; in return,
  the assumption that the fake catalog matches reality gets tested.

#### Next Step

So far, both checks have worked by **running**: a process comes up, a request goes out, the
response gets examined. The provider's question, though, is asked before the change is
published, and it has a form that requires no run. Adding a field, removing a field,
narrowing a type, and making an optional field required — which of these four change types
breaks the old consumer, and which does not? The next lesson turns this question into a check
that compares two schema versions, sorts the change types into compatibility classes, and
shows, in that same run, a change the check counts as compatible even though it breaks the
consumer.

### Schema Compatibility Checking

Source: https://academia.sh/en/courses/integration-testing/schema-compatibility-checking

A check that compares two schema versions and stops a breaking change before publication: change types falling into separate compatibility classes for the request and response directions, the expand–contract sequence turning the gate green, and a value-set widening the check counts as compatible breaking the consumer.

The previous two checks both worked by running: a process comes up, a request goes out, the
response gets examined. The provider's real question, though, is asked before the change is
published, and it requires no run. Adding a field, removing a field, narrowing a type, making
an optional field required: which of these four change types breaks the old consumer?

The **definition** of a breaking change was established in the Web API Design course, and the
**measurement of its cost** in the Application Layer and Service Interaction course; neither
is repeated here. This lesson's subject is turning that knowledge into an automatic check:
two schema versions are compared, every change is placed into a type and a compatibility
class, and if there is a change in the breaking class, the suite turns red.

#### Schema Versions and Proposals

The published schema describes the book response and the checkout request together. There
are three proposals: `A` carries all four change types at once, `B` meets the same need with
the first step of the expand–contract sequence, and `C` only widens a value set.

```js
// schema.mjs — the published schema and three proposals; each field carries a type, requiredness, and value set
const A = (type, required, set) => (set ? { type, required, set } : { type, required });

export const PUBLISHED = {
  response: {
    isbn: A("string", true), title: A("string", true), shelfCount: A("number", true),
    status: A("string", true, ["shelved", "loaned", "lost"]),
    loanDays: A("number", true), tags: A("array", false),
  },
  request: { memberNo: A("string", true), branch: A("string", false), count: A("number", false) },
};

// A: four change types at once. B: the expand step. C: only the set widens.
export const PROPOSAL = {
  A: {
    response: { ...PUBLISHED.response, dueDay: A("string", false) },
    request: { ...PUBLISHED.request, branch: A("string", true), count: A("integer", false) },
  },
  B: { response: { ...PUBLISHED.response, dueDay: A("string", false) }, request: { ...PUBLISHED.request } },
  C: {
    response: { ...PUBLISHED.response, status: A("string", true, ["shelved", "loaned", "lost", "newly-shelved"]) },
    request: { ...PUBLISHED.request },
  },
};
delete PROPOSAL.A.response.tags;                      // field removal
```

The check itself has two parts: a comparison that sorts the diff into types, and a table that
binds types to classes. The table having two columns is this lesson's central distinction —
the same change type falls into a different class in the request direction than in the
response direction.

```js
// compatibility.mjs — sorts the changes between two schema versions into types and classes
const NARROW = { number: "integer" };

export const CLASS = {
  "optional field addition": { response: "compatible", request: "compatible" },
  "required field addition": { response: "compatible", request: "breaking" },
  "field removal": { response: "breaking", request: "compatible" },
  "type narrowing": { response: "compatible", request: "breaking" },
  "type widening": { response: "breaking", request: "compatible" },
  "making required": { response: "compatible", request: "breaking" },
  "making optional": { response: "breaking", request: "compatible" },
  "adding a value to the set": { response: "compatible", request: "breaking" },
  "removing a value from the set": { response: "breaking", request: "compatible" },
};

export function diff(old, next) {
  const d = [];
  for (const [name, rule] of Object.entries(old)) {
    const n = next[name];
    if (n === undefined) { d.push({ name, type: "field removal" }); continue; }
    if (n.type !== rule.type) d.push({ name, type: NARROW[rule.type] === n.type ? "type narrowing" : "type widening" });
    if (n.required && !rule.required) d.push({ name, type: "making required" });
    if (rule.required && !n.required) d.push({ name, type: "making optional" });
    if ((n.set ?? []).some((v) => !(rule.set ?? []).includes(v))) d.push({ name, type: "adding a value to the set" });
    if ((rule.set ?? []).some((v) => !(n.set ?? []).includes(v))) d.push({ name, type: "removing a value from the set" });
  }
  for (const [name, n] of Object.entries(next)) {
    if (name in old) continue;
    d.push({ name, type: n.required ? "required field addition" : "optional field addition" });
  }
  return d;
}

// Evaluates a proposal in both directions; each line of the returned list is one classified change.
export const evaluate = (published, proposal) => ["response", "request"].flatMap((direction) =>
  diff(published[direction], proposal[direction]).map((x) => ({ direction, ...x, class: CLASS[x.type][direction] })));
```

#### The Distribution of Change Types

```js
// distribution.mjs — how change types distribute over compatibility classes, and a count of the three proposals
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { CLASS, evaluate } from "./compatibility.mjs";

console.log(`${"change type".padEnd(32)}${"response".padStart(12)}${"request".padStart(12)}`);
for (const [type, s] of Object.entries(CLASS)) {
  console.log(`${type.padEnd(32)}${s.response.padStart(12)}${s.request.padStart(12)}`);
}

for (const [name, o] of Object.entries(PROPOSAL)) {
  const d = evaluate(PUBLISHED, o);
  console.log(`\nproposal ${name}: ${d.length} changes, ${d.filter((x) => x.class === "breaking").length} breaking`);
  for (const x of d) console.log(`  ${`${x.direction}.${x.name}`.padEnd(20)}${x.type.padEnd(30)}-> ${x.class}`);
}
```

```
change type                         response     request
optional field addition           compatible  compatible
required field addition           compatible    breaking
field removal                       breaking  compatible
type narrowing                    compatible    breaking
type widening                       breaking  compatible
making required                   compatible    breaking
making optional                     breaking  compatible
adding a value to the set         compatible    breaking
removing a value from the set       breaking  compatible

proposal A: 4 changes, 3 breaking
  response.tags       field removal                 -> breaking
  response.dueDay     optional field addition       -> compatible
  request.branch      making required               -> breaking
  request.count       type narrowing                -> breaking

proposal B: 1 changes, 0 breaking
  response.dueDay     optional field addition       -> compatible

proposal C: 1 changes, 0 breaking
  response.status     adding a value to the set     -> compatible
```

The table's two columns mirror each other. Removing a field in the response is breaking,
because the consumer is reading it; removing a field in the request is not breaking, because
the old client keeps sending it and the field is ignored. In the same way, making a
field required does no harm in the response — the field was always going to arrive anyway —
but in the request it causes every request the old client sends to be rejected. Narrowing a
type is compatible in the response, because an integer is still a number; it is breaking in
the request, because the old client might send a decimal.

This symmetry shows that the eight-row table is not a list to be memorized. There is a single
question, and direction decides it: **is the old code on the other side still valid after the
change?** In the response, the other side reads; in the request, it writes.

#### The Acceptance Gate

The check works as the **acceptance gate** defined in the Quality and Testing Fundamentals
course: if there is a change in the breaking class, the release stops.

```js
// compatibility.test.mjs — pre-publish gate: if there is a breaking change, the suite turns red
// Usage: PROPOSAL=A|B|C node --test compatibility.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";
import { PUBLISHED, PROPOSAL as PROPOSALS } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

const NAME = process.env.PROPOSAL ?? "B";

test(`proposal ${NAME}: no breaking change against the published schema`, () => {
  const breaking = evaluate(PUBLISHED, PROPOSALS[NAME]).filter((x) => x.class === "breaking");
  assert.equal(breaking.length, 0,
    `breaking change: ${breaking.map((x) => `${x.direction}.${x.name} ${x.type}`).join(", ")}`);
});
```

```bash
for o in A B C; do
  PROPOSAL=$o node --test --test-reporter=tap compatibility.test.mjs |
    grep -E '^(ok|not ok)|breaking change:' | sed 's/^ *//'
done
```

```
not ok 1 - proposal A: no breaking change against the published schema
breaking change: response.tags field removal, request.branch making required, request.count type narrowing
ok 1 - proposal B: no breaking change against the published schema
ok 1 - proposal C: no breaking change against the published schema
```

**The defect class caught is the contract breaking before publication.** Proposal A turned
red and named the three breaking changes along with their direction. The fix is to turn the
same need into the first step of the expand–contract sequence: the new field is added
alongside the old ones, and removal and making-required are left to the third step. Proposal
B is exactly this, and the gate turns green.

What sets this check apart from the previous two lessons is that it sees the break **without
running a single consumer**. The contract test had to run the consumer's expectation and the
provider's version together to show a break; the check here compares two texts. In exchange,
what it knows is less: it cannot say which consumer breaks, only that one might.

#### The Break the Check Counts as Compatible

Proposal C passed through the gate. A new value was added to the defined value set for book
status: books that have newly entered the collection and are on the shelf are now reported as
`newly-shelved`. All of the old responses are still valid, no field was lost, and no type
changed.

```js
// missed.mjs — a change the check counts as compatible breaks the consumer
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

// The loan service's decision branch: an unknown status value counts as "not on shelf".
const loanDecision = (book) => (book.status === "shelved" && book.shelfCount > 0
  ? { granted: true, message: `${book.isbn} loaned` }
  : { granted: false, message: `not on shelf (status ${book.status}, shelfCount ${book.shelfCount})` });

const d = evaluate(PUBLISHED, PROPOSAL.C);
console.log(`check: proposal C -> ${d.length} changes, ${d.filter((x) => x.class === "breaking").length} breaking, gate green`);

const ON_SHELF = { isbn: "978-0262033848", status: "shelved", shelfCount: 2 };
const NEWLY_SHELVED = { ...ON_SHELF, status: "newly-shelved" };  // freshly arrived shelf: book is shelved and loanable
for (const [label, book] of [["old value", ON_SHELF], ["new value", NEWLY_SHELVED]]) {
  const decision = loanDecision(book);
  console.log(`${label.padEnd(11)} -> granted=${String(decision.granted).padEnd(5)} ${decision.message}`);
}
console.log(`schema status values ${PROPOSAL.C.response.status.set.length}, values the consumer treats as loanable 1`);
console.log(`a unit change does not change the schema -> ${evaluate(PUBLISHED, PUBLISHED).length} changes`);
```

```
check: proposal C -> 1 changes, 0 breaking, gate green
old value   -> granted=true  978-0262033848 loaned
new value   -> granted=false not on shelf (status newly-shelved, shelfCount 2)
schema status values 4, values the consumer treats as loanable 1
a unit change does not change the schema -> 0 changes
```

**The missed defect class is a change that alters meaning without breaking shape.** A book
sitting on the shelf became impossible to loan; from the schema's point of view, no rule was
violated. Adding a value to the set counts as compatible in the response direction, because
it is, as far as schema validation goes — the old values keep their validity. What breaks is
not the schema but the consumer's code, which **assumes it has exhausted** that set. This
assumption is written in no schema.

The last line shows the boundary even more plainly. The loan duration starting to be counted
in weeks instead of days changes neither the field name nor its type; the two schemas stay
byte-for-byte identical and the check reports zero changes. No check that only looks at the
schema can see a change in unit, scale, or meaning.

#### The Cost of the Check

```js
// cost.mjs — the cost of a check that only looks at the schema: zero processes, zero requests
import { createServer } from "node:http";
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

const REPEAT = 200, proposalCount = Object.keys(PROPOSAL).length;
const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) for (const o of Object.values(PROPOSAL)) evaluate(PUBLISHED, o);
const t1 = performance.now();

// Comparison point: a single local request's round trip on the same machine.
const server = createServer((_, res) => res.end("{}"));
await new Promise((c) => server.listen(0, "127.0.0.1", c));
const t2 = performance.now();
await (await fetch(`http://127.0.0.1:${server.address().port}/`)).text();
const t3 = performance.now();
server.close();

const fields = Object.keys(PUBLISHED.response).length + Object.keys(PUBLISHED.request).length;
console.log(`schema ${fields} fields (response ${Object.keys(PUBLISHED.response).length}, request ${Object.keys(PUBLISHED.request).length}), ${proposalCount} proposals`);
console.log(`check ${REPEAT * proposalCount} evaluations, 0 processes, 0 requests, 0 test data`);
console.log(`one evaluation shorter than a single local request : ${(t1 - t0) / (REPEAT * proposalCount) < t3 - t2}`);
console.log(`ratio at least fifty times                         : ${(t3 - t2) / ((t1 - t0) / (REPEAT * proposalCount)) >= 50}`);
```

```
schema 9 fields (response 6, request 3), 3 proposals
check 600 evaluations, 0 processes, 0 requests, 0 test data
one evaluation shorter than a single local request : true
ratio at least fifty times                         : true
```

Absolute times are machine-dependent; the direction of the ratio does not change. This check
brings up no process, sends no request, and prepares no test data: its input is two texts,
and its output is a classified list. Hundreds of evaluations fit inside the time of a single
local request; that is why the check can run on every change proposal.

The cost independent of the run sits in two places. First, the **published** version of the
schema has to be stored somewhere: the comparison needs two texts, and the compatibility
question cannot be asked with only one. Second is the upkeep of the class table — nine rows
are hand-written, and when a new constraint is added to the schema language, a row has to be
added to the table too. A missing row stays silent, and a check that stays silent is
indistinguishable from a check that does not exist.

#### Summary

- A schema compatibility check compares two schema versions, sorts the diff into change
  types, and places each type into a compatibility class.
- The same type falls into a different class in the request direction than in the response
  direction; the single question is whether the old code on the other side is still valid
  after the change.
- The proposal carrying four change types at once turned red with three breaking changes;
  once the same need was turned into the first step of the expand–contract sequence, the gate
  turned green.
- The missed class is a change that alters meaning without breaking shape: a value added to a
  value set was counted as compatible, yet a book sitting on the shelf became impossible to
  loan. With a unit change, the two schemas stay identical and the check reports zero changes.
- Cost: zero processes, zero requests, zero test data; in return, the published schema has to
  be stored and the nine-row class table maintained by hand.
- The check cannot say which consumer breaks; it can only say that one might.

#### Next Step

All three checks assumed the provider was ready: the schema check sent a request to the live
service, the contract test brought up the provider's version, and the compatibility check
read the published schema. Yet the moment a contract is most useful is the moment the
provider is **not yet written**. The consumer team does not have to wait, because the
contract is already settled. The next lesson builds a mock server that generates responses
from the contract, develops the loan service against it, and measures a single question: at
how many points does the mock server deviate from the real thing, and how many of those
deviations can the checks built so far close.

### Developing With Mock Servers

Source: https://academia.sh/en/courses/integration-testing/developing-with-mock-servers

Counting the points where a mock server that lets development continue without a ready provider deviates from reality: sending the same set of requests to two servers, measuring the deviation the contract run closes, and showing that a deviation tied to state across requests cannot be closed by any contract.

All three previous checks assumed the provider was ready. Yet the moment a contract is most
useful is the moment the provider is not yet written: the team developing the loan service
does not have to wait for the catalog service to be finished, because the contract is already
settled.

A **mock server** turns the contract into something runnable and removes the waiting time. In
exchange, it creates a risk: what stands in for reality is not reality itself. This lesson
turns that risk into a number — at **how many points** does the mock server deviate from
reality, and how many of those deviations can the contract run built so far close.

#### Two Servers, One Contract

The contract is the knowledge produced by the previous lessons: the five fields the loan
service reads, along with their types. The sample values the mock server will generate also
live in the same file.

```js
// contract.mjs — the book response contract: fields read, their types, and sample values
export const FIELD = { isbn: "string", title: "string", shelfCount: "integer", status: "string", loanDays: "integer" };
export const SAMPLE = { isbn: "978-0262033848", title: "Sample Book", shelfCount: 2, status: "shelved", loanDays: 14 };
export const SAMPLE_V1 = { ...SAMPLE, shelfCount: "2" };   // the mock server's first version: sample value is a string
export const INTERACTION = [{ path: "/book/978-0262033848", fields: FIELD }];
```

The real catalog service carries two books, decrements the shelf count on a checkout request,
and rejects the request if there is no copy on the shelf.

```js
// real.mjs — real catalog service: holds state, applies the business rule
// Usage: node real.mjs <port|0>
import { createServer } from "node:http";
if (process.argv[2] === undefined) { console.log("usage: node real.mjs <port|0>"); process.exit(0); }

const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", shelfCount: 2,
    status: "shelved", loanDays: 14, tags: ["algorithms"] },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", shelfCount: 0,
    status: "loaned", loanDays: 7, tags: [] },
};
const readBody = (i) => new Promise((c) => { let v = ""; i.on("data", (p) => (v += p)); i.on("end", () => c(v)); });

const server = createServer(async (request, response) => {
  response.sendDate = false;
  const path = request.url.split("?")[0];
  const json = (code, body) => {
    response.writeHead(code, { "content-type": "application/json; charset=utf-8" });
    response.end(JSON.stringify(body));
  };
  if (request.method === "POST" && path === "/checkout") {
    const k = BOOKS[JSON.parse((await readBody(request)) || "{}").isbn];
    if (k === undefined) return json(404, { error: "book not found" });
    if (k.shelfCount < 1) return json(409, { error: "shelf empty" });
    k.shelfCount -= 1;
    return json(201, { checkoutNo: "A-1", isbn: k.isbn });
  }
  const k = BOOKS[path.replace("/book/", "")];
  return k ? json(200, k) : json(404, { error: "book not found" });
});
server.listen(Number(process.argv[2]), "127.0.0.1", () => console.log(`real ${server.address().port}`));
```

The mock server is generated from the contract and does everything the contract has written
down: it returns the sample body. It does nothing the contract does not write down — it holds
no state and applies no rule.

```js
// mock.mjs — mock server generated from the contract: holds no state, applies no business rule
// Usage: MOCK_VERSION=1|2 node mock.mjs <port|0>
import { createServer } from "node:http";
import { SAMPLE, SAMPLE_V1 } from "./contract.mjs";
if (process.argv[2] === undefined) { console.log("usage: node mock.mjs <port|0>"); process.exit(0); }

const BODY = process.env.MOCK_VERSION === "2" ? SAMPLE : SAMPLE_V1;
const server = createServer((request, response) => {
  request.resume();
  response.sendDate = false;
  const checkout = request.url.split("?")[0] === "/checkout";
  response.writeHead(checkout ? 201 : 200, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(checkout ? { checkoutNo: "A-1", isbn: BODY.isbn } : BODY));
});
server.listen(Number(process.argv[2]), "127.0.0.1", () => console.log(`mock ${server.address().port}`));
```

#### Counting the Deviation Points

Deviation is not an opinion but a number: the same set of requests is sent to both servers,
and five metrics are compared.

```js
// deviation.mjs — the same set of requests is sent to both servers, and deviation points are counted
import { spawn } from "node:child_process";
import { FIELD } from "./contract.mjs";

const ISBN = "978-0262033848";
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");

async function drive(file) {
  const proc = spawn("node", [file, "0"], { env: process.env });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const base = `http://127.0.0.1:${port}`;
  const call = async (path, options) => {
    const res = await fetch(base + path, options);
    return { status: res.status, body: await res.json() };
  };
  const book = await call(`/book/${ISBN}`);
  const missing = await call("/book/978-0000000000");
  const empty = await call("/checkout", { method: "POST", body: JSON.stringify({ isbn: "978-0201896831" }) });
  await call("/checkout", { method: "POST", body: JSON.stringify({ isbn: ISBN }) });
  const after = await call(`/book/${ISBN}`);
  proc.kill();
  return {
    "success field count": String(Object.keys(book.body).length),
    "shelfCount type": typeName(book.body.shelfCount),
    "unknown book code": String(missing.status),
    "checkout on empty shelf": String(empty.status),
    "shelfCount after checkout": String(after.body.shelfCount),
  };
}

const mock = await drive("mock.mjs");
const real = await drive("real.mjs");
console.log(`${"metric".padEnd(27)}${"mock".padStart(11)}${"real".padStart(11)}${"deviation".padStart(11)}`);
let deviation = 0;
for (const metric of Object.keys(mock)) {
  const different = mock[metric] !== real[metric];
  deviation += different ? 1 : 0;
  console.log(`${metric.padEnd(27)}${mock[metric].padStart(11)}${real[metric].padStart(11)}` +
    `${(different ? "yes" : "no").padStart(11)}`);
}
console.log(`mock version ${process.env.MOCK_VERSION ?? "1"}: deviation ${deviation}/${Object.keys(mock).length}, ` +
  `fields checked by the contract ${Object.keys(FIELD).length}`);
```

```
metric                            mock       real  deviation
success field count                  5          6        yes
shelfCount type                 string    integer        yes
unknown book code                  200        404        yes
checkout on empty shelf            201        409        yes
shelfCount after checkout            2          1        yes
mock version 1: deviation 5/5, fields checked by the contract 5
```

All five of the five metrics deviate. The deviations are not of the same kind, and that
distinction is this lesson's axis. The first two are **shape** deviations: field count and
type. The third and fourth are **decision** deviations: the mock server answers every request
positively, because it does not know what to return under which condition. The fifth is a
**state** deviation: the real service decrements the shelf count after a checkout, while the
mock server sees the two requests as independent of each other.

#### The Deviation the Contract Run Closes

The mock server has to pass the contract run too. The same test file runs against both the
mock and the real server; the server name is supplied from outside.

```js
// contract.test.mjs — the same contract run against both the mock and the real server
// Usage: SERVER=mock.mjs|real.mjs MOCK_VERSION=1|2 node --test contract.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { INTERACTION } from "./contract.mjs";

const FILE = process.env.SERVER ?? "real.mjs";
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");
let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", [FILE, "0"], { env: process.env });
  proc.stdout.once("data", (v) => { base = `http://127.0.0.1:${String(v).split(" ")[1]}`; resolve(); });
}));
after(() => proc.kill());

for (const e of INTERACTION) {
  test(`${FILE} version ${process.env.MOCK_VERSION ?? "-"} ${e.path}`, async () => {
    const body = await (await fetch(base + e.path)).json();
    const missing = Object.entries(e.fields)
      .filter(([name, type]) => body[name] === undefined || typeName(body[name]) !== type)
      .map(([name, type]) => `${name} (expected ${type})`);
    assert.equal(missing.length, 0, `unmet field: ${missing.join(", ")}`);
  });
}
```

```bash
for s in "mock.mjs 1" "mock.mjs 2" "real.mjs -"; do
  set -- $s
  SERVER=$1 MOCK_VERSION=$2 node --test --test-reporter=tap contract.test.mjs |
    grep -E '^(ok|not ok)|unmet' | sed 's/^ *//'
done
MOCK_VERSION=2 node deviation.mjs | tail -1
```

```
not ok 1 - mock.mjs version 1 /book/978-0262033848
unmet field: shelfCount (expected integer)
ok 1 - mock.mjs version 2 /book/978-0262033848
ok 1 - real.mjs version - /book/978-0262033848
mock version 2: deviation 4/5, fields checked by the contract 5
```

**The defect class caught is the mock server deviating from the contract.** In the mock
server's first version, the sample value was written as a string; the contract run turned red
and reported the field by name. After the sample was fixed, the same run turned green, and
the deviation count dropped from five to four. The rule that follows is a single sentence:
**the mock server must be checked with the exact same contract the consumer is checked
against**; otherwise the consumer ends up developed against the mock server itself, not
against the contract.

**The missed defect class is the deviations outside the contract's field of view, and it
splits into three.** The first is field count: the real service also returns a tags field,
and the mock server does not. The contract does not cover it, because the consumer does not
read it; if it ever starts reading it, the recording setup from the previous lesson adds it
to the contract, and the deviation closes on its own.

The second is decision deviation: the status code for an unknown book, and a checkout request
when there is no copy on the shelf. These can be closed by adding **new interactions** to the
contract; the cost is writing one scenario and one rule in the mock server for each. With
every deviation it closes, the mock server grows a little more like the real service; at the
limit, the mock server turns into a second implementation of the real service, and at that
point it loses its reason to exist.

The third cannot be closed. The shelf count decrementing after a checkout is not a property
of a single request; it is **the relationship between two requests**. The unit a contract
test looks at is one request and its response; state across requests falls outside this unit.
Making the mock server hold state does not close this deviation, it only moves it — now the
mock server's state rule can deviate from reality, and there is no contract to check it
against.

#### The Cost of the Mock Server

```js
// cost.mjs — the cost of a mock server: how many lines of fixture, how many requests, what timing
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const lineCount = (d) => readFileSync(d, "utf8").trimEnd().split("\n").length;

async function measure(file) {
  const proc = spawn("node", [file, "0"], { env: { ...process.env, MOCK_VERSION: "2" } });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const url = `http://127.0.0.1:${port}/book/978-0262033848`;
  for (let i = 0; i < 20; i += 1) await (await fetch(url)).json();      // warm-up
  const t = performance.now();
  for (let i = 0; i < 100; i += 1) await (await fetch(url)).json();
  const duration = performance.now() - t;
  proc.kill();
  return duration;
}

const mock = await measure("mock.mjs");
const real = await measure("real.mjs");
console.log(`fixture: mock.mjs ${lineCount("mock.mjs")} lines, real.mjs ${lineCount("real.mjs")} lines`);
console.log(`deviation run 2 processes and 10 requests, contract run 1 process and 1 interaction`);
console.log(`both servers' durations same order of magnitude : ${mock / real > 0.33 && mock / real < 3}`);
console.log(`mock server run shorter than half the real one  : ${mock < real / 2}`);
```

```
fixture: mock.mjs 15 lines, real.mjs 31 lines
deviation run 2 processes and 10 requests, contract run 1 process and 1 interaction
both servers' durations same order of magnitude : true
mock server run shorter than half the real one  : false
```

The last line corrects a misunderstanding. A mock server is not used for **speed**: both
servers are local processes, and the time for a hundred requests is the same order of
magnitude. What a mock server provides is not duration but **timing** — the ability to
develop the consumer before the provider is ready. This same distinction also says that using
a mock server in place of the real service on an ongoing basis has no justification.

The cost independent of the run sits in two numbers. As a fixture, the mock server is fifteen
lines and the real service is thirty-one; the sixteen lines in between are exactly the work
the mock server **does not do**, and they are the counterpart of the five rows in the
deviation table. The second number is the deviation table itself: five metrics, two processes
per run, and ten requests. This table is written once and run every time the contract
changes; there is no other way to estimate how much the mock server actually deviates.

#### Summary

- A mock server turns the contract into something runnable and lets the consumer be developed
  without waiting for the provider; in exchange, it creates the risk of deviating from
  reality.
- Deviation is a number: when the same set of requests was sent to both servers, all five of
  the five metrics deviated.
- Deviations are of three kinds — shape (field count, type), decision (status code, rule),
  and state (effects across requests).
- When the mock server was checked against the same contract as the consumer, a shape
  deviation was caught; once the sample value was fixed, the run turned green and the
  deviation dropped from five to four.
- Decision deviations can be closed by adding interactions to the contract; every closure
  brings the mock server closer to reality, and at the limit turns it into a second
  implementation.
- State deviation cannot be closed: a contract test's unit is a single request and response,
  and the relationship across requests falls outside this unit. Cost: 2 processes, 10
  requests, a fifteen-line fixture; the gain is not duration but timing.

#### Next Step

This topic built four checks, and all four had the same shape: looking at **a single
boundary**. The schema check examined one response, the contract test counted fields between
one consumer and one provider, the compatibility check compared two schema texts, and the
deviation table measured one mock server against one real service. Each is strong at its own
boundary, and even when all of them run together, one question stays unanswered.

The path where a member looks up a book on the shelf, checks it out, and borrows it is not
within the scope of any of these checks. That path uses multiple requests, multiple services,
and state carried across requests, all together; the fifth metric turning out unclosable was
the first sign of this. The next topic takes up tests that run this path as a whole, and its
first question is a cost question: what does catching the same bug at three separate levels
cost, and how many tests should be kept at which level.

## End-to-End and Interface Tests

Balancing the cost and value of tests that run over the whole application: establishing the cross-level distribution with the pyramid, the effect of selector strategy and wait management on flakiness, device and emulator decisions, the benefit and noise of screen comparison, managing flakiness by cause with quarantine and repair discipline, and the trade-off parallelizing the run sets between duration and isolation.

### The Test Pyramid

Source: https://academia.sh/en/courses/integration-testing/test-pyramid

Running the same borrowing flow at the unit, integration, and end-to-end levels; showing the defect class each level catches and misses; and measuring the cost per level with query, request, and setup counts.

The previous topic built four checks: whether the request and response matched the
schema, whether the consumer's expectation ran against the provider, whether a schema
change was breaking, and whether the mock server stayed faithful to the contract. What
they share is this: each looks at **a single boundary**. A member borrowing a book fits
none of those boundaries — the rule lives in one place, the record in another, and the
response the interface sees in a third.

Classifying the levels was done in the Quality and Testing Fundamentals course; what is
asked here is not the classification but the **distribution**. The answer to how many
tests belong at which level is not a preference: the cost of running each level, the
defect class it catches, and the cost of catching the same defect at more than one level
are all measured, and the distribution follows from these three numbers.

#### The Same Flow, Three Levels

The borrowing flow is split into three parts: the eligibility rule is a pure function,
the record is a real database, and the outer surface is an HTTP interface. The rule
places a limit on the number of open loans.

```js
// rule.mjs — version 1: loan eligibility rule
export const LIMIT = 5;

export function isEligible({ openLoans, daysOverdue, membership, bookStatus }) {
  if (membership !== 'active') return { eligible: false, reason: 'membership-not-active' };
  if (bookStatus !== 'shelved') return { eligible: false, reason: 'book-not-shelved' };
  if (daysOverdue > 0) return { eligible: false, reason: 'has-overdue-loan' };
  if (openLoans > LIMIT) return { eligible: false, reason: 'loan-limit-exceeded' };
  return { eligible: true, reason: 'eligible' };
}
```

The repository layer turns the decision into real rows. Every query is counted; this
counter is the lesson's measure of cost.

```js
// repository.mjs — catalog and loan records over node:sqlite
import { DatabaseSync } from 'node:sqlite';
import { isEligible } from './rule.mjs';

export const counter = { query: 0 };

export function setupRepository() {
  const db = new DatabaseSync(':memory:');
  db.exec(`CREATE TABLE member (member_no TEXT PRIMARY KEY, membership TEXT, days_overdue INTEGER);
           CREATE TABLE book (book_no TEXT PRIMARY KEY, status TEXT);
           CREATE TABLE loan (member_no TEXT, book_no TEXT UNIQUE)`);
  return db;
}

export function borrowBook(db, memberNo, bookNo) {
  const read = (s, ...p) => { counter.query += 1; return db.prepare(s).all(...p); };
  const write = (s, ...p) => { counter.query += 1; db.prepare(s).run(...p); };
  const [member] = read('SELECT membership, days_overdue FROM member WHERE member_no = ?', memberNo);
  const [book] = read('SELECT status FROM book WHERE book_no = ?', bookNo);
  const [{ n }] = read('SELECT COUNT(*) AS n FROM loan WHERE member_no = ?', memberNo);
  const decision = isEligible({
    openLoans: n, daysOverdue: member.days_overdue, membership: member.membership, bookStatus: book.status,
  });
  if (decision.eligible === false) return decision;
  write('INSERT INTO loan (member_no, book_no) VALUES (?, ?)', memberNo, bookNo);
  write("UPDATE book SET status = 'loaned' WHERE book_no = ?", bookNo);
  return decision;
}
```

The outer surface has two parts: a small function that maps the decision to a status
code, and the server that answers the request.

```js
// statuscode.mjs — version 1: decision mapped to an HTTP status code
export const statusCode = (decision) => (decision.eligible ? 201 : 409);
```

```js
// service.mjs — HTTP interface for the loan flow
import { createServer } from 'node:http';
import { borrowBook } from './repository.mjs';
import { statusCode } from './statuscode.mjs';

export function setupServer(db) {
  return createServer((request, response) => {
    let body = '';
    request.on('data', (chunk) => { body += chunk; });
    request.on('end', () => {
      const { memberNo, bookNo } = JSON.parse(body);
      const decision = borrowBook(db, memberNo, bookNo);
      response.writeHead(statusCode(decision), { 'content-type': 'application/json' });
      response.end(JSON.stringify(decision));
    });
  });
}
```

Four tests exercise the same flow at three levels: one tests only the rule, one tests
the rule together with the database, and two test the path the user sees.

```js
// levels.test.mjs — the same loan flow tested at three levels
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { isEligible } from './rule.mjs';
import { setupRepository, borrowBook } from './repository.mjs';
import { setupServer } from './service.mjs';

const seed = () => {
  const db = setupRepository();
  db.prepare("INSERT INTO member VALUES ('U-17', 'active', 0)").run();
  db.prepare("INSERT INTO member VALUES ('U-31', 'suspended', 0)").run();
  db.prepare("INSERT INTO book VALUES ('K-90', 'shelved')").run();
  return db;
};

const send = async (body) => {
  const server = setupServer(seed());
  await new Promise((done) => server.listen(0, done));
  const response = await fetch(`http://127.0.0.1:${server.address().port}/loan`,
    { method: 'POST', body: JSON.stringify(body) });
  const payload = await response.json();
  server.close();
  return { status: response.status, reason: payload.reason };
};

test('unit: the fifth open loan reaches the limit', () => {
  const base = { daysOverdue: 0, membership: 'active', bookStatus: 'shelved' };

  assert.equal(isEligible({ ...base, openLoans: 4 }).eligible, true);
  assert.equal(isEligible({ ...base, openLoans: 5 }).reason, 'loan-limit-exceeded');
});

test('integration: a loan row is written and the book status returns', () => {
  const db = seed();

  assert.equal(borrowBook(db, 'U-17', 'K-90').eligible, true);
  const row = db.prepare('SELECT member_no, book_no FROM loan').get();
  assert.equal(`${row.member_no} ${row.book_no}`, 'U-17 K-90');
  assert.equal(db.prepare("SELECT status FROM book WHERE book_no = 'K-90'").get().status, 'loaned');
});

test('end-to-end: an eligible request returns approval', async () => {
  assert.deepEqual(await send({ memberNo: 'U-17', bookNo: 'K-90' }), { status: 201, reason: 'eligible' });
});

test('end-to-end: a suspended membership is rejected', async () => {
  assert.deepEqual(await send({ memberNo: 'U-31', bookNo: 'K-90' }),
    { status: 409, reason: 'membership-not-active' });
});
```

#### The Boundary Comparison Defect

The first version of the rule has the boundary comparison wrong: if the limit is five
open loans, a member with five should not be able to take a sixth. Writing
`openLoans > LIMIT` grants the sixth and stops only at the seventh.

```bash
node --test --test-reporter=tap levels.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
not ok 1 - unit: the fifth open loan reaches the limit
ok 2 - integration: a loan row is written and the book status returns
ok 3 - end-to-end: an eligible request returns approval
ok 4 - end-to-end: a suspended membership is rejected
# tests 4
# pass 3
# fail 1
```

The unit level caught the defect. What stands out in the same run is the other three
lines: with the defect still in the code, the integration and end-to-end tests **stayed
green**. The reason is a simple choice — those two tests run the happy path, a member
with zero open loans. Catching the boundary defect at an upper level would have required
preparing a member with five open loans there too; the cost of that is the subject of
the next section.

The fix is in a single comparison.

```js
// rule.mjs — version 2: boundary comparison fixed
export const LIMIT = 5;

export function isEligible({ openLoans, daysOverdue, membership, bookStatus }) {
  if (membership !== 'active') return { eligible: false, reason: 'membership-not-active' };
  if (bookStatus !== 'shelved') return { eligible: false, reason: 'book-not-shelved' };
  if (daysOverdue > 0) return { eligible: false, reason: 'has-overdue-loan' };
  if (openLoans >= LIMIT) return { eligible: false, reason: 'loan-limit-exceeded' };
  return { eligible: true, reason: 'eligible' };
}
```

```bash
node --test --test-reporter=tap levels.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - unit: the fifth open loan reaches the limit
ok 2 - integration: a loan row is written and the book status returns
ok 3 - end-to-end: an eligible request returns approval
ok 4 - end-to-end: a suspended membership is rejected
# tests 4
# pass 4
# fail 0
```

#### The Defect Only the Upper Level Sees

There is also a defect class running the other way: one with no counterpart at a lower
level. The status code of a rejected loan request is one of these — for the rule it is a
single value, "not eligible"; for the repository it is not writing a record; but
for the client calling the interface, that value is a status code.

```js
// statuscode.mjs — version 2: rejection now returns a success code
export const statusCode = (decision) => (decision.eligible ? 201 : 200);
```

```bash
node --test --test-reporter=tap levels.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - unit: the fifth open loan reaches the limit
ok 2 - integration: a loan row is written and the book status returns
ok 3 - end-to-end: an eligible request returns approval
not ok 4 - end-to-end: a suspended membership is rejected
# tests 4
# pass 3
# fail 1
```

The rule and repository tests stay green; the defect is in no value they see. The fix is
to reverse the mapping.

```js
// statuscode.mjs — version 3: rejection returns a conflict code again
export const statusCode = (decision) => (decision.eligible ? 201 : 409);
```

```bash
node --test --test-reporter=tap levels.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - unit: the fifth open loan reaches the limit
ok 2 - integration: a loan row is written and the book status returns
ok 3 - end-to-end: an eligible request returns approval
ok 4 - end-to-end: a suspended membership is rejected
# tests 4
# pass 4
# fail 0
```

That the mapping was pulled out into its own function is no accident here: because
`statusCode` is a separate function, a unit test can be written for it, and that defect
is never again left to the end-to-end level. The pyramid is a design pressure before it
is a counting rule — a check that can be pushed down is pushed down.

#### Cost

The rule checks four conditions: membership, book status, overdue days, and the
open-loan count. Seeing the boundary behavior needs three values of open loans (four,
five, six); the other three conditions have two values each. Twenty-four cases in total.
The measurement below runs the same twenty-four cases at three levels and counts the
cost per level.

```js
// cost.mjs — the cost of running the same case set at three levels
import { isEligible } from './rule.mjs';
import { setupRepository, borrowBook, counter } from './repository.mjs';
import { setupServer } from './service.mjs';

const cases = [];
for (const membership of ['active', 'suspended']) {
  for (const bookStatus of ['shelved', 'loaned']) {
    for (const daysOverdue of [0, 3]) {
      for (const openLoans of [4, 5, 6]) cases.push({ membership, bookStatus, daysOverdue, openLoans });
    }
  }
}

const measurement = { setup: 0, request: 0 };
const prepare = (c) => {
  const db = setupRepository();
  db.prepare('INSERT INTO member VALUES (?, ?, ?)').run('U-17', c.membership, c.daysOverdue);
  db.prepare('INSERT INTO book VALUES (?, ?)').run('K-90', c.bookStatus);
  measurement.setup += 2;
  for (let i = 0; i < c.openLoans; i += 1) {
    db.prepare('INSERT INTO loan VALUES (?, ?)').run('U-17', `E-${i}`);
    measurement.setup += 1;
  }
  return db;
};

const s = (n, g) => String(n).padStart(g);
const measure = async (name, component, run) => {
  measurement.setup = 0;
  measurement.request = 0;
  counter.query = 0;
  const start = performance.now();
  await run();
  const duration = performance.now() - start;
  console.log(`${name.padEnd(13)}${s(component, 11)}${s(cases.length, 6)}${s(measurement.setup, 9)}`
    + `${s(counter.query, 7)}${s(measurement.request, 9)}`);
  return duration;
};

console.log(`${'level'.padEnd(13)}${s('component', 11)}${s('case', 6)}${s('setup', 9)}${s('query', 7)}${s('request', 9)}`);

const unit = await measure('unit', 1, () => {
  for (const c of cases) isEligible(c);
});

const integration = await measure('integration', 2, () => {
  for (const c of cases) borrowBook(prepare(c), 'U-17', 'K-90');
});

const endToEnd = await measure('end-to-end', 3, async () => {
  for (const c of cases) {
    const server = setupServer(prepare(c));
    await new Promise((done) => server.listen(0, done));
    await fetch(`http://127.0.0.1:${server.address().port}/loan`,
      { method: 'POST', body: JSON.stringify({ memberNo: 'U-17', bookNo: 'K-90' }) });
    measurement.request += 1;
    server.close();
  }
});

const ratio = (a, b, threshold) => (a / b > threshold ? 'yes' : 'no');
console.log(`is integration duration more than 10x the unit duration: ${ratio(integration, unit, 10)}`);
console.log(`is end-to-end duration more than 100x the unit duration: ${ratio(endToEnd, unit, 100)}`);
```

```
level          component  case    setup  query  request
unit                   1    24        0      0        0
integration            2    24      168     74        0
end-to-end             3    24      168     74       24
is integration duration more than 10x the unit duration: yes
is end-to-end duration more than 100x the unit duration: yes
```

Duration depends on the machine and the load, so the output prints a ratio threshold
rather than raw milliseconds. In this run, the twenty-four cases took 0.02 ms at the
unit level, 1.85 ms at the integration level, and 31.79 ms at the end-to-end level; on
another machine all three numbers change, their ordering does not. The run-independent
quantities are in the table: the same twenty-four cases produce no query and no request
at the unit level, while the upper two levels produce 168 setup rows and 74 flow
queries, and the topmost level adds 24 network requests on top of that. The component
column is also a cost: however many parts a level requires to be up at the same time,
that many sources of failure it carries.

Repeating the same claim at three levels adds up these costs. If the twenty-four cases
were repeated at every level, that would be seventy-two tests; when the rule changes one
day, three separate files get updated and three runs get brought back to green.
Repetition catches no new defect class in return: the upper levels read the same
decision, with a transport layer stacked on top.

The shape of the pyramid is the sum of these two observations. At the lower level a case
counts as free, so the case set is consumed there. At an upper level, every case buys
setup, queries, and requests; that is why only defect classes **with no counterpart at a
lower level** are tested there — the wiring between parts, the mapping in the transport
layer, the integrity of the path the user sees. The reverse shape, consuming the case
set at the top, catches the same defects far more expensively and far more slowly.

#### Summary

- The borrowing flow ran at three levels: the rule as a pure function, the record with a
  real database, the outer surface with an HTTP interface.
- Only the unit level caught the boundary comparison defect; the upper two levels stayed
  green because they ran the happy path.
- Only the end-to-end level caught the status-code mapping defect; the rule and
  repository tests never see that value.
- The cost of the twenty-four cases rose from 0 queries and 0 requests to 242 queries and
  24 requests, depending on the level; the duration ratio in this run exceeded a hundred
  times the unit level.
- The pyramid is not a counting rule but the result of two measures: the case set is
  consumed where its cost is low, and only the defect classes with no lower-level
  counterpart are left for the upper level.

#### Next Step

The end-to-end tests here drove the HTTP interface: a request was sent, a status code
was read. The surface the member actually sees, though, is a page — with a search box, a
book card, and a button on it. A test that drives that surface has to say which element
to touch with a **selector** and wait for the element to be ready. The next lesson
measures these two decisions: how many structural changes a test survives when the
selector type changes, and how the wait policy changes the drop rate.

### Browser Automation

Source: https://academia.sh/en/courses/integration-testing/browser-automation

Measuring the two decisions a page-driving test makes — selector choice and wait policy; the resilience of four selector types against structural change, and the drop rate and wait cost of sleep, implicit wait, and explicit wait.

The previous lesson's end-to-end tests drove the HTTP interface: a request was sent, a
status code was read. The surface the member actually sees, though, is a page. A test
that drives a page has to make two decisions, and neither is visible while writing the
code — both show up only at run time: how to say which element to touch, and how to
wait for that element to be ready.

This lesson does not drive a real browser. What it drives is a small tree model of the
page; both decisions are actually computed over that model. What is measured is not a
tool's behavior but the two decisions themselves — which selector type breaks under
which change, and how much each wait policy shifts the drop rate.

**Selector** here is the same word as the selector in the Visual Presentation with CSS
course, but the job is different: there it says which elements a rule applies to; here
it says which element a test drives.

#### The Page Model

The loan page carries a card: a title, the book's status, and a borrow button. The
button has three separate handles — its position, its class name, and its test id.

```js
// page.mjs — small tree model of the loan page
export const element = (tag, attributes = {}, children = [], text = '') =>
  ({ tag, attributes, children, text });

export function buildPage(state) {
  const button = element('button', { class: 'primary', id: 'borrow-book' }, [],
    state.buttonText ?? 'Borrow');
  const card = element('div', { class: 'card', id: 'book-card' }, [
    element('h2', { class: 'title' }, [], 'Lost Books'),
    element('span', { class: 'status', id: 'book-status' }, [], state.book),
    button,
  ]);
  return { root: element('main', {}, [element('nav', {}, [], 'Catalog'), card]), button, card };
}
```

Four selector types search for the same button four separate ways: by position in the
tree, by class name, by visible text, and by test id. The number of nodes visited is
counted; this is the selector's running cost.

```js
// selector.mjs — four selector strategies and a visited-node counter
export const counter = { node: 0 };

function* walk(node) {
  counter.node += 1;
  yield node;
  for (const child of node.children) yield* walk(child);
}

const first = (root, predicate) => {
  for (const node of walk(root)) if (predicate(node)) return node;
  return null;
};

const path = (root, indices) => {
  let node = root;
  for (const i of indices) {
    counter.node += 1;
    node = node.children[i];
    if (node === undefined) return null;
  }
  return node;
};

export const selectors = {
  position: (root) => path(root, [1, 2]),
  class: (root) => first(root, (d) => d.attributes.class === 'primary'),
  text: (root) => first(root, (d) => d.text === 'Borrow'),
  id: (root) => first(root, (d) => d.attributes.id === 'borrow-book'),
};
```

#### Resilience to Structural Change

An interface changes as it lives, and most of those changes do not change behavior: the
card gets wrapped in a layout container, the class name gets renewed, the button's label
gets corrected, the order inside the card changes, an attribute gets added for
measurement. Five changes are applied one at a time, each to a page built fresh from
scratch, and the element each selector finds is compared against the actual button.

```js
// resilience.mjs — selector types against five structural changes
import { element, buildPage } from './page.mjs';
import { selectors, counter } from './selector.mjs';

const changes = {
  wrapping: (p) => { p.root.children[1] = element('div', { class: 'layout' }, [p.card]); },
  'class name': (p) => { p.button.attributes.class = 'action-primary'; },
  'label text': (p) => { p.button.text = 'Borrow Now'; },
  'card order': (p) => { p.card.children = [p.card.children[0], p.button, p.card.children[1]]; },
  'tracking attribute': (p) => { p.button.attributes.tracking = 'card-primary'; },
};

const s = (n, g) => String(n).padStart(g);
console.log(`${'selector'.padEnd(8)}${s('ok', 7)}${s('not found', 12)}${s('wrong element', 15)}${s('node', 8)}`);

for (const [name, selector] of Object.entries(selectors)) {
  const result = { ok: 0, notFound: 0, wrong: 0 };
  counter.node = 0;
  for (const apply of Object.values(changes)) {
    const page = buildPage({ book: 'shelved' });
    apply(page);
    const found = selector(page.root);
    if (found === page.button) result.ok += 1;
    else if (found === null) result.notFound += 1;
    else result.wrong += 1;
  }
  console.log(`${name.padEnd(8)}${s(result.ok, 7)}${s(result.notFound, 12)}${s(result.wrong, 15)}${s(counter.node, 8)}`);
}
```

```
selector     ok   not found  wrong element    node
position      3           1              1      10
class         4           1              0      30
text          4           1              0      30
id            5           0              0      30
```

Three results can be read off. The first is the ranking: the position-based selector
broke on two of the five changes, the class and text selectors on one each, the test id
on none. The id selector's resilience is not a superiority but a contract — that
attribute exists only for the test, and design changes never touch it.

The second is the two separate kinds of breakage. `not found` turns the test red and the
reason is read immediately. `wrong element` is silent: when the order inside the card
changes, the position-based selector finds the status element instead of the button, the
test touches it, and nothing happens. This is the most expensive form a flaky test can
take; it is as likely to stay green as it is to turn red and point at the wrong place.

The third is cost. The position-based selector touched only ten nodes across five
searches, the other three touched thirty. That gap widens as the tree grows — but a
thirty-node traversal is not comparable to the cost of a test that silently touches the
wrong element.

#### Wait Policy

A page cannot be driven before it is ready, but "ready" is not a single moment: an
element first enters the tree, then becomes clickable. Three policies build three
separate relationships with these two moments. **Sleep** waits a fixed duration and
tries once. **Implicit wait** polls for the element's presence and returns the moment it
appears. **Explicit wait** polls the expected condition itself — here, clickability.

The run below produces two thousand page loads from its own seeded generator; two
percent of the pages load heavily, and forty percent have a gap between the element
appearing and becoming clickable.

```js
// wait.mjs — drop rate of three wait policies, seed 20
function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const POLL_INTERVAL = 25;
const TIMEOUT = 1000;
const FIXED_SLEEP = 250;

const generateEvent = (random) => {
  const visibleAt = random() < 0.02 ? 900 + random() * 600 : 20 + random() * 180;
  const gap = random() < 0.6 ? 0 : 10 + random() * 200;
  return { visibleAt, clickableAt: visibleAt + gap };
};

const poll = (predicate) => {
  let t = 0;
  let polls = 0;
  while (t <= TIMEOUT) {
    polls += 1;
    if (predicate(t)) return { t, polls, reached: true };
    t += POLL_INTERVAL;
  }
  return { t, polls, reached: false };
};

const policies = {
  sleep: (event) => ({ success: event.clickableAt <= FIXED_SLEEP, wait: FIXED_SLEEP, polls: 1 }),
  'implicit wait': (event) => {
    const r = poll((t) => event.visibleAt <= t);
    return { success: r.reached && event.clickableAt <= r.t, wait: r.t, polls: r.polls };
  },
  'explicit wait': (event) => {
    const r = poll((t) => event.clickableAt <= t);
    return { success: r.reached, wait: r.t, polls: r.polls };
  },
};

const RUNS = 2000;
const s = (n, g) => String(n).padStart(g);
console.log(`${'policy'.padEnd(15)}${s('drop rate', 12)}${s('avg. wait', 14)}${s('polls', 9)}`);

for (const [name, policy] of Object.entries(policies)) {
  const random = createRng(20);
  let dropped = 0;
  let wait = 0;
  let polls = 0;
  for (let i = 0; i < RUNS; i += 1) {
    const result = policy(generateEvent(random));
    if (result.success === false) dropped += 1;
    wait += result.wait;
    polls += result.polls;
  }
  console.log(`${name.padEnd(15)}${s(`%${((dropped / RUNS) * 100).toFixed(1)}`, 12)}`
    + `${s(`${(wait / RUNS).toFixed(0)} ms`, 14)}${s(polls, 9)}`);
}
```

```
policy            drop rate     avg. wait    polls
sleep                 %16.0        250 ms     2000
implicit wait         %40.9        136 ms    12878
explicit wait          %1.4        181 ms    16477
```

It is no accident that implicit wait gives the lowest average wait while also producing
the highest drop rate: it polls the wrong condition. It returns the moment the element
enters the tree, while the test wanted the element to be clickable; every time that gap
opens, it produces a drop. Sleep sits at the opposite end — it pays two hundred fifty
milliseconds on every run and still prevents only a sixth of the drops; wasted on a
fast-loading page, insufficient on a heavily loaded one. Explicit wait returns early on a
fast page, so on average it comes in cheaper than sleep, and it brings the drop rate down
to a twentieth.

Explicit wait's rate is not zero — it is 1.4 percent. The remaining drops are heavy
loads that exceed the timeout. A wait policy manages flakiness, it does not eliminate
it; the rising poll count is the cost of that.

#### The Defect Caught and the Defect Missed

What defect class does a page-driving test catch? The card's status element not
updating even though the loan request succeeded is one of them — the server returns the
right response, the previous lesson's end-to-end test stays green, and the member still
sees "shelved" on screen.

```js
// update.mjs — version 1: page state after the click
export const afterClick = (state) => ({ ...state, loaned: true });
```

```js
// flow.test.mjs — loan flow driven through the page model
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { buildPage } from './page.mjs';
import { selectors } from './selector.mjs';
import { afterClick } from './update.mjs';

const textById = (root, id) => {
  if (root.attributes.id === id) return root.text;
  for (const child of root.children) {
    const found = textById(child, id);
    if (found !== null) return found;
  }
  return null;
};

test('the id selector finds the button even if the class name changes', () => {
  const page = buildPage({ book: 'shelved' });
  page.button.attributes.class = 'action-primary';

  assert.equal(selectors.id(page.root), page.button);
});

test('the card status updates after the loan is taken', () => {
  const state = { book: 'shelved' };
  const button = selectors.id(buildPage(state).root);
  assert.equal(button.tag, 'button');

  const next = buildPage(afterClick(state));

  assert.equal(textById(next.root, 'book-status'), 'loaned');
});

test('a button with the wrong label is still found and clicked', () => {
  const page = buildPage({ book: 'shelved', buttonText: 'Return' });

  const button = selectors.id(page.root);

  assert.equal(button.tag, 'button');
  assert.equal(button.attributes.id, 'borrow-book');
});
```

```bash
node --test --test-reporter=tap flow.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the id selector finds the button even if the class name changes
not ok 2 - the card status updates after the loan is taken
ok 3 - a button with the wrong label is still found and clicked
# tests 3
# pass 2
# fail 1
```

The fix is for the post-click state to also carry the book's status.

```js
// update.mjs — version 2: book status is also updated
export const afterClick = (state) => ({ ...state, loaned: true, book: 'loaned' });
```

```bash
node --test --test-reporter=tap flow.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the id selector finds the button even if the class name changes
ok 2 - the card status updates after the loan is taken
ok 3 - a button with the wrong label is still found and clicked
# tests 3
# pass 3
# fail 0
```

The third test shows the class that is missed, and it is green in both runs: even if
the button's label is "Return", the test finds it, touches it, and the flow completes.
A selector bound to the test id never looks at the label at all. So the cost of the
selector's resilience is that a defect in the label text falls outside this test's
scope; unless that check is written separately, a mislabeled button ships with a green
run.

On the cost side, these three tests took about 33 ms in this run; duration depends on
the machine and grows by an order of magnitude once a real browser is driven. The
run-independent numbers stand in the measurements themselves: the tree driven has six
nodes, the id selector visited thirty nodes across five searches, and the
two-thousand-load run with explicit wait makes 16,477 polls.

#### Summary

- A test that drives a page makes two decisions: which handle to find the element by,
  and how to wait for it to be ready.
- Across five structural changes, the position-based selector broke twice, the class and
  text selectors once each; the test id never broke.
- The two kinds of breakage carry separate costs: not-found turns the test red, a
  wrong-element match silently touches the wrong place.
- Across two thousand loads, sleep gave a 16.0 percent drop rate, implicit wait 40.9
  percent, explicit wait 1.4 percent; implicit wait's cheapness comes from polling the
  wrong condition.
- The page test caught the card-not-updating defect and missed the mislabeled button; a
  selector bound to the id never looks at the label.

#### Next Step

All the measurements here were over a single page model: one tree, one timeline. The
question changes once the same interface runs on a handful of devices, several
operating system versions, and different screen widths. The question is no longer which
selector is resilient but **which devices** the test runs on, because the product of
device and version grows the number of runs far faster than the number of pages. The
next lesson measures how fast that product grows, what share of users the selected
subset covers, and the risk of a defect falling outside that coverage.

### Mobile Automation

Source: https://academia.sh/en/courses/integration-testing/mobile-automation

Measuring the run matrix that grows with the product of the device and version axes; comparing the share covered and the defect classes caught by subsets selected by user share and by coverage; and the limit of the dimension an emulator carries.

All the previous lesson's measurements were over a single page model: one tree, one
timeline. The borrowing flow's mobile interface, though, does not run in one place. The
same screen opens on a handful of device classes and several operating system versions,
and the test's question is no longer "which selector is resilient" but "which devices
do I run on."

This lesson does not set up a real device or emulator. What is measured is not the
device itself but **the selection decision**: how fast the matrix grows, how many users
the selected subset represents, and what the defect class left out of coverage is. All
three are countable.

#### The Matrix and Two Selection Criteria

The model's input is written out explicitly. **MB1 — six device classes and five
operating system versions; the shares are taken as measured from member sessions, and
each axis's shares sum to one hundred; the two axes are assumed independent.** The
independence assumption does not hold in real deployments (old versions concentrate on
old devices); there, the cell share is measured directly and the computation below works
the same way.

```js
// matrix.mjs — device-version matrix and two subset selections
export const devices = [
  ['small', 22], ['medium', 26], ['large', 18], ['foldable', 6], ['tablet', 12], ['legacy', 16],
];
export const versions = [['S-6', 4], ['S-7', 9], ['S-8', 17], ['S-9', 31], ['S-10', 39]];

export const cells = devices.flatMap(([device, dp]) =>
  versions.map(([version, vp]) => ({ device, version, share: (dp * vp) / 100 })));

export const byShare = (k) => [...cells].sort((a, b) => b.share - a.share).slice(0, k);

export function byCoverage(k) {
  const selected = [];
  const seen = { device: new Set(), version: new Set() };
  while (selected.length < k) {
    const remaining = cells.filter((h) => selected.includes(h) === false);
    const gain = (h) => (seen.device.has(h.device) ? 0 : 1) + (seen.version.has(h.version) ? 0 : 1);
    remaining.sort((a, b) => gain(b) - gain(a) || b.share - a.share);
    const pick = remaining[0];
    seen.device.add(pick.device);
    seen.version.add(pick.version);
    selected.push(pick);
  }
  return selected;
}

export const subsets = { byShare: byShare(6), byCoverage: byCoverage(6) };
```

The two criteria optimize different things. Selection by share takes the cells with the
most users. Selection by coverage prefers, at every step, the cell that gains a device
class or version not yet seen, and falls back to share on a tie. Six cells are selected
each way, and the catch is counted for three defect classes: a version-dependent defect
is caught if that version is in the subset, a device-class-dependent defect if that class
is in it, and a defect specific to a single cell only if that exact cell was selected.

```js
// coverage.mjs — user share and defect-catch rate of two selections
import { devices, versions, cells, subsets } from './matrix.mjs';

const rate = (selected) => {
  const device = new Set(selected.map((h) => h.device));
  const version = new Set(selected.map((h) => h.version));
  return {
    share: selected.reduce((t, h) => t + h.share, 0),
    versionDefect: `${version.size}/${versions.length}`,
    deviceDefect: `${device.size}/${devices.length}`,
    cellDefect: `${selected.length}/${cells.length}`,
  };
};

const s = (n, g) => String(n).padStart(g);
console.log(`${'selection'.padEnd(16)}${s('cell', 6)}${s('user share', 16)}${s('version defect', 16)}${s('device defect', 15)}${s('cell defect', 13)}`);
for (const [name, selected] of [['by share', subsets.byShare], ['by coverage', subsets.byCoverage]]) {
  const r = rate(selected);
  console.log(`${name.padEnd(16)}${s(selected.length, 6)}${s(`%${r.share.toFixed(1)}`, 16)}${s(r.versionDefect, 16)}${s(r.deviceDefect, 15)}${s(r.cellDefect, 13)}`);
}
console.log(`matrix ${devices.length} devices x ${versions.length} versions = ${cells.length} cells;`
  + ` adding a version gives ${cells.length + devices.length}, adding a device gives ${cells.length + versions.length}`);
console.log('selected by share:', subsets.byShare.map((h) => `${h.device}/${h.version}`).join(' '));
console.log('selected by coverage:', subsets.byCoverage.map((h) => `${h.device}/${h.version}`).join(' '));
```

```
selection         cell      user share  version defect  device defect  cell defect
by share             6           %46.9             2/5            4/6         6/30
by coverage          6           %24.3             5/5            6/6         6/30
matrix 6 devices x 5 versions = 30 cells; adding a version gives 36, adding a device gives 35
selected by share: medium/S-10 small/S-10 medium/S-9 large/S-10 small/S-9 legacy/S-10
selected by coverage: medium/S-10 small/S-9 large/S-8 legacy/S-7 tablet/S-6 foldable/S-10
```

The numbers expose the trade-off. Selection by share represents 46.9 percent of users but touches
only two of the five versions and four of the six device classes; four of its six selected cells
are on the same version. Selection by coverage cuts the represented share in half, but in exchange
it enters every version and every device class once. On the cell-specific defect the two are equal,
and both are weak: six of thirty cells, one in five.

The matrix's growth reopens this decision every year. When a new version ships, the
matrix goes from thirty to thirty-six; when a new device class arrives, to thirty-five;
if the subset stays at six cells, cell coverage drops from a fifth to a sixth. The run
budget grows linearly while the matrix grows multiplicatively, and that is why the
question for a mobile run can never be "run everything."

#### The Dimension an Emulator Carries

The coverage table carries an assumption: that the selected cell really runs on that
device class. An **emulator** meets only half of that assumption. An emulator genuinely
runs the operating system version — it carries version-dependent defects. What it does
not carry is anything tied to the device class: the touch surface's real size, the
memory limit, network interruption, the camera and sensors, the manufacturer's interface
layer.

This arithmetic feeds straight into the table. If all six cells selected by coverage run
on an emulator, the device-class-dependent catch drops from six of six to zero; the
version-dependent catch stays at five of five. If two cells move to real hardware in a
**device farm**, the device catch rises to two of six. So the decision is not "emulator
or real device"; it is which defect class to look for in which cell.

#### The Defect It Catches: Version-Dependent Parsing

At the end of the borrowing flow, the member is shown a due date. The application is
the side that produces the date text; the platform's date parser is the side that reads
it, and that parser's behavior varies by version: older versions accept only the ordered
format.

```js
// platform.mjs — date-parsing behavior that varies by version
const WIDE_PARSER_VERSIONS = new Set(['S-9', 'S-10']);

export function parseDate(version, text) {
  if (/^\d{4}-\d{2}-\d{2}$/.test(text)) return text;
  if (WIDE_PARSER_VERSIONS.has(version) && /^\d{2}\.\d{2}\.\d{4}$/.test(text)) {
    const [day, month, year] = text.split('.');
    return `${year}-${month}-${day}`;
  }
  return null;
}
```

```js
// duedate.mjs — version 1: due date produced in a local format
export function dueDate(loanDate, days = 14) {
  const t = new Date(`${loanDate}T00:00:00Z`);
  t.setUTCDate(t.getUTCDate() + days);
  const two = (n) => String(n).padStart(2, '0');
  return `${two(t.getUTCDate())}.${two(t.getUTCMonth() + 1)}.${t.getUTCFullYear()}`;
}
```

The same flow is run over both subsets; for each cell, whether the produced date is
readable on that version is tested.

```js
// device.test.mjs — due-date flow runs across the selected subsets
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { subsets } from './matrix.mjs';
import { parseDate } from './platform.mjs';
import { dueDate } from './duedate.mjs';

const run = (selected) => {
  for (const cell of selected) {
    const text = dueDate('2026-03-04');
    assert.equal(parseDate(cell.version, text), '2026-03-18',
      `${cell.device}/${cell.version} could not read the due date: ${text}`);
  }
};

test('the due date is read across the subset selected by share', () => run(subsets.byShare));

test('the due date is read across the subset selected by coverage', () => run(subsets.byCoverage));
```

```bash
node --test --test-reporter=tap device.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the due date is read across the subset selected by share
not ok 2 - the due date is read across the subset selected by coverage
# tests 2
# pass 1
# fail 1
```

The defect is one, the subsets are two, and the result splits in two. The subset
selected by coverage turned red because it carries S-6, S-7, and S-8 cells. The subset
selected by share **stayed green with the same defect**: it holds only S-9 and S-10, and
both accept the local format. The class missed here is not one a test form misses but
one a **selection** misses, and the missed users can be counted too: the three versions
from S-6 through S-8 are thirty percent of the version axis.

The fix is to produce the date text in a platform-independent format.

```js
// duedate.mjs — version 2: due date produced in a platform-independent format
export function dueDate(loanDate, days = 14) {
  const t = new Date(`${loanDate}T00:00:00Z`);
  t.setUTCDate(t.getUTCDate() + days);
  return t.toISOString().slice(0, 10);
}
```

```bash
node --test --test-reporter=tap device.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the due date is read across the subset selected by share
ok 2 - the due date is read across the subset selected by coverage
# tests 2
# pass 2
# fail 0
```

There is also a class both subsets miss: a defect seen in only a single cell — the
interface layer of a specific device class on a specific version. Its counterpart in the
table is six of thirty; four-fifths of cell-specific defects fall outside coverage under
both selections, and that ratio does not improve without enlarging the subset.

The run-dependent side of the cost is small here — the two tests took about 33 ms,
because what runs here is a model. In a real run, the number that sets the cost is in
the table: twelve cell runs, six device sessions for each of the two subsets. Each
session means one setup, one app install, and one teardown; if the subset is expanded to
all thirty cells, that number multiplies by five, and duration grows in the same
proportion.

#### Summary

- The device and version axes multiply: six devices and five versions make thirty
  cells; a new version adds six, a new device class adds five.
- The six cells selected by share represented 46.9 percent of users but touched two of
  five versions and four of six device classes.
- The six cells selected by coverage cut the share to 24.3 percent, but in exchange
  entered every version and every device class.
- Only the subset selected by coverage caught the version-dependent date-parsing
  defect; the subset selected by share stayed green with the same defect.
- An emulator carries the version dimension, not the device-class dimension; if all six
  cells run on an emulator, device defect catches drop from six of six to zero.

#### Next Step

Up to this point, every claim was about a value: a status code, a text, a date. The
interface's real output, though, is an image, and asking "are they equal" about it
suddenly becomes hard — two screenshots are never made of exactly the same pixels. The
next lesson ties this comparison to a threshold and measures it by scanning that
threshold: how many false alarms it produces as the threshold drops, and how many real
regressions it lets through as the threshold rises.

### Visual Verification

Source: https://academia.sh/en/courses/integration-testing/visual-verification

A pixel comparison of the screenshot against the baseline image; measuring false-alarm and miss rates by scanning the threshold; the effect of excluding a variable region from the comparison; and the defect class visual verification cannot see.

Up to this point, every claim was about a value: a status code, a text, a date. The
interface's real output, though, is an image, and a button's shadow disappearing, a drop
in contrast, or a card shifting by a pixel shows up in no value claim. **Visual
regression** testing wants to fill that gap: a screenshot is compared against an
accepted **baseline image**.

The question looks like an equality question, and it is not. Two screenshots are never
made of the same pixels; font antialiasing, subpixel placement, and regions of the
screen that change produce a difference on every run. So the comparison is tied to a
threshold, and the threshold choice directly sets two rates: a **false alarm** (turning
red when there is no regression) and a **miss** (staying green when there is one).

This lesson does not take screenshots. The comparison runs over a real pixel array; the
screen is a small model of the loan card.

#### The Pixel Model of the Screen

Forty columns, twenty rows, a single tone channel. The card draws three things: the
title bar, the book's status text, and the borrow button. There is also a region at the
top right that changes on every run — the remaining-time indicator.

```js
// image.mjs — small pixel model of the card screen and a comparison
export const WIDTH = 40;
export const HEIGHT = 20;

const rect = (p, x0, y0, x1, y1, tone) => {
  for (let y = y0; y <= y1; y += 1) for (let x = x0; x <= x1; x += 1) p[y * WIDTH + x] = tone;
};

export function draw(d) {
  const p = new Uint8Array(WIDTH * HEIGHT).fill(240);
  rect(p, 2, 2, 30 + (d.titleShift ?? 0), 4, 90);
  rect(p, 2, 7, 1 + (d.statusWidth ?? 12), 8, 120);
  if (d.buttonVisible !== false) {
    const k = d.shift ?? 0;
    rect(p, 2 + k, 12, 16 + k, 16, d.buttonTone ?? 30);
  }
  if (d.roundedCorners) for (const [x, y] of [[2, 12], [16, 12], [2, 16], [16, 16]]) p[y * WIDTH + x] = 240;
  rect(p, 32, 0, 39, 1, d.clockTone ?? 40);
  return p;
}

export const excludedRegion = () => {
  const m = new Uint8Array(WIDTH * HEIGHT);
  for (let y = 0; y <= 1; y += 1) for (let x = 32; x < WIDTH; x += 1) m[y * WIDTH + x] = 1;
  return m;
};

export const compare = (a, b, pixelThreshold = 12, excluded = null) => {
  let different = 0;
  let counted = 0;
  for (let i = 0; i < a.length; i += 1) {
    if (excluded && excluded[i] === 1) continue;
    counted += 1;
    if (Math.abs(a[i] - b[i]) > pixelThreshold) different += 1;
  }
  return { different, rate: (different / counted) * 100 };
};
```

The comparison has two thresholds, and they must not be confused. The **pixel
threshold** says how much tone difference in a single pixel counts as negligible; here,
twelve. The **ratio threshold** says how many differing pixels count as a regression,
and that is what actually decides the outcome. The scan below holds the pixel threshold
fixed and scans the ratio threshold.

#### Threshold Scan

The scan produces two sets. The first set has no regression: the same accepted state is
redrawn with a changing time indicator and, with a one-in-ten probability, a title
shifted by a pixel. The second set has five real regressions. For every threshold, how
many of the first set trigger an alarm and how many of the second set are missed are
counted. The variable region at the top right is left in the comparison once and
excluded from it once, giving two measurements.

```js
// threshold.mjs — threshold scan: false-alarm and miss rates, seed 31
import { draw, compare, excludedRegion } from './image.mjs';

function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const ACCEPTED = { buttonVisible: true, buttonTone: 30, statusWidth: 12, clockTone: 40 };
const baseline = draw(ACCEPTED);

const regressions = {
  'no button': { buttonVisible: false },
  'low contrast': { buttonTone: 200 },
  'button shift': { shift: 1 },
  'status shortened': { statusWidth: 8 },
  'rounded corners': { roundedCorners: true },
};

const thresholds = [0, 0.25, 0.5, 1, 2, 5, 10];
const RUNS = 200;

const equivalent = (random) => draw({
  ...ACCEPTED,
  clockTone: 60 + Math.floor(random() * 8) * 20,
  titleShift: random() < 0.1 ? 1 : 0,
});

const measure = (excluded) => {
  const falseAlarm = thresholds.map(() => 0);
  const missed = thresholds.map(() => 0);
  const random = createRng(31);
  for (let i = 0; i < RUNS; i += 1) {
    const rate = compare(baseline, equivalent(random), 12, excluded).rate;
    thresholds.forEach((e, j) => { if (rate > e) falseAlarm[j] += 1; });
  }
  const regressionNames = Object.keys(regressions);
  for (const name of regressionNames) {
    for (let i = 0; i < RUNS / regressionNames.length; i += 1) {
      const broken = draw({
        ...ACCEPTED, ...regressions[name], clockTone: 60 + Math.floor(random() * 8) * 20,
      });
      const rate = compare(baseline, broken, 12, excluded).rate;
      thresholds.forEach((e, j) => { if (rate <= e) missed[j] += 1; });
    }
  }
  return { falseAlarm, missed };
};

const unexcluded = measure(null);
const excluded = measure(excludedRegion());
const s = (n, g) => String(n).padStart(g);
const pct = (n) => `%${((n / RUNS) * 100).toFixed(1)}`;
console.log(`${'ratio threshold'.padEnd(16)}${s('false alarm', 14)}${s('missed', 9)}`
  + `${s('excl. alarm', 17)}${s('excl. missed', 19)}`);
thresholds.forEach((e, j) => {
  console.log(`${`%${e.toFixed(2)}`.padEnd(16)}${s(pct(unexcluded.falseAlarm[j]), 14)}`
    + `${s(pct(unexcluded.missed[j]), 9)}${s(pct(excluded.falseAlarm[j]), 17)}`
    + `${s(pct(excluded.missed[j]), 19)}`);
});
for (const [name, diff] of Object.entries(regressions)) {
  const rate = compare(baseline, draw({ ...ACCEPTED, ...diff }), 12, excludedRegion()).rate;
  console.log(`${name.padEnd(18)} excluded diff rate %${rate.toFixed(2)}`);
}
```

```
ratio threshold    false alarm   missed      excl. alarm       excl. missed
%0.00                   %100.0     %0.0             %5.5               %0.0
%0.25                   %100.0     %0.0             %5.5               %0.0
%0.50                   %100.0     %0.0             %0.0               %0.0
%1.00                   %100.0     %0.0             %0.0              %20.0
%2.00                     %5.5     %0.0             %0.0              %60.0
%5.00                     %0.0    %60.0             %0.0              %60.0
%10.00                    %0.0    %60.0             %0.0             %100.0
no button          excluded diff rate %9.57
low contrast       excluded diff rate %9.57
button shift       excluded diff rate %1.28
status shortened   excluded diff rate %1.02
rounded corners    excluded diff rate %0.51
```

The first two columns show that no threshold works while the variable region stays in.
The time indicator alone is sixteen pixels, or two percent of the screen; so every
threshold below two percent alarms on every run, and every threshold above it swallows
every small regression. A two-percent threshold happens to look good by chance — zero
misses, 5.5 percent alarms — but that is the threshold settling onto the size of the
noise: once the indicator grows by one digit, the threshold has to be retuned.

Once the variable region is excluded from the comparison, the table becomes readable.
At the half-percent threshold, false alarms drop to zero and all five regressions are
caught. As the threshold rises, misses open up in order: at one percent, rounded
corners; at two percent, the status text and the button shift; at ten percent, even the
button disappearing entirely. The last lines give the reason for this order — the
difference rate each regression produces is fixed, and the moment the threshold passes
it, the defect becomes invisible.

The rule that follows is this: the threshold is kept below the smallest real regression
that could be missed; if the noise exceeds that threshold, the threshold is not raised —
**the source of the noise is removed from the comparison**. Excluding variable regions
is the cost of keeping the threshold low, and every excluded region also means that any
regression that could occur there goes unseen.

#### The Defect Caught and the Defect Missed

The class visual verification catches is a defect that shows up in no value claim. The
button tone lightening at the style layer is one of these: the tree is the same, the
selector works, the flow completes, and the button is too faint to read.

```js
// style.mjs — version 1: button tone lightened
export const style = { buttonTone: 200 };
```

```js
// visual.test.mjs — card screen compared against the baseline image
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { draw, compare, excludedRegion } from './image.mjs';
import { style } from './style.mjs';

const ACCEPTED = { buttonVisible: true, buttonTone: 30, statusWidth: 12, clockTone: 40 };
const baseline = draw(ACCEPTED);
const excluded = excludedRegion();
const THRESHOLD = 0.5;

test('the card screen matches the baseline image', () => {
  const now = draw({ ...ACCEPTED, buttonTone: style.buttonTone, clockTone: 120 });

  const { different, rate } = compare(baseline, now, 12, excluded);

  assert.ok(rate <= THRESHOLD, `${different} pixels different, rate %${rate.toFixed(2)}`);
});

test('an inactive button produces no difference in the image', () => {
  const active = draw({ ...ACCEPTED, buttonTone: style.buttonTone, active: true, clockTone: 120 });
  const inactive = draw({ ...ACCEPTED, buttonTone: style.buttonTone, active: false, clockTone: 200 });

  assert.equal(compare(active, inactive, 12, excluded).different, 0);
});
```

```bash
node --test --test-reporter=tap visual.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
not ok 1 - the card screen matches the baseline image
ok 2 - an inactive button produces no difference in the image
# tests 2
# pass 1
# fail 1
```

The second test sets up the class that is missed, and it is green in both runs: the
button being active or inactive produces the same pixels, because inactivity has no
visual counterpart in the model. Visual verification asks whether **what is seen** is
correct; it does not ask what happens when it is touched. If the borrow button looks
exactly right and sends no request at all, this test stays green, and that defect is
caught only by the flow tests from the earlier lessons.

The fix is to return the tone to the baseline image's value.

```js
// style.mjs — version 2: button tone restored to the baseline value
export const style = { buttonTone: 30 };
```

```bash
node --test --test-reporter=tap visual.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'
```

```
ok 1 - the card screen matches the baseline image
ok 2 - an inactive button produces no difference in the image
# tests 2
# pass 2
# fail 0
```

The run-dependent side of the cost is small here: the two tests took about 37 ms in this run, the
threshold scan ran eight hundred comparisons; each read 800 pixels while the variable region stayed
in, and 784 once it was excluded. The run-independent cost is the number of baseline images. If the
borrowing flow's four screens are kept across the previous lesson's six cells selected by coverage,
that stores twenty-four baseline images. A design decision that deliberately changes the button's
tone requires re-approving all twenty-four of them — visual verification's real cost is not the run
but this approval work.

#### Summary

- Visual verification compares a screenshot against an accepted baseline image, and the
  decision is tied to two thresholds: the pixel tone threshold and the
  differing-pixel ratio threshold.
- While the variable region stayed in the comparison, every threshold below two percent
  gave a 100 percent false-alarm rate, and every threshold above it missed the small
  regressions.
- Once the variable region was excluded, the half-percent threshold caught all five
  regressions with zero false alarms.
- As the threshold rises, misses open up in the order of the difference rate each
  regression produces: rounded corners at one percent, a one-pixel shift at two percent,
  the button disappearing at ten percent.
- Visual verification asks whether what is seen is correct; an inactive button produces
  the same pixels, so it passes this test green.

#### Next Step

The last three lessons saw the same fact from separate angles: wait policy shifted the
drop rate, device selection shifted defect coverage, threshold choice shifted the
false-alarm rate. All three are pieces of a single problem — end-to-end tests carry an
instability that the lower-level tests do not carry. The next lesson takes on that
instability in aggregate: what proportion the causes of drops are distributed across in
an end-to-end team, and how much cleaning the signal with quarantine raises the rate of
escaped defects.

### Flakiness Management

Source: https://academia.sh/en/courses/integration-testing/flakiness-management

Measuring flakiness's distribution across causes at the scale of an end-to-end team; how individually small drop rates compound at team scale; and the trade-off the quarantine threshold sets between red runs and escaped real defects.

The last three lessons saw the same fact from separate angles: wait policy shifted the
drop rate, device selection shifted defect coverage, threshold choice shifted the false
alarm rate. All three are pieces of a single problem — an end-to-end test carries an
instability that the lower-level tests do not carry.

The definition of a flaky test, the drop rate, and quarantine were established in the
Unit Testing and Test-Driven Development course. Here the scale changes. There, a single
test's drop rate was measured; here, the run of a forty-test team is measured, and two
new questions arise: what proportion of drops each cause accounts for, and how much
quarantining flaky tests raises the rate of escaped defects.

#### Distribution at Team Scale

The model's input is written out explicitly. **KY1 — forty end-to-end tests. Twelve are
exposed to a tight wait timeout (0.02 per run), six depend on state left behind by the
previous test (0.01), eight share the same member and book record (0.015), five make a
large number of network calls (a 0.005 per-call stall probability across six calls, that
is, 0.0296), and nine are stable. A real regression occurs in one run in four and falls
under one of forty protecting tests.** The proportions across causes come from the
mechanism: the network cause grows with the number of calls, the shared-data cause with
the number of tests using the same record.

```js
// team.mjs — two hundred runs of forty end-to-end tests and a quarantine scan
function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const tests = [];
const add = (count, cause, rate) => {
  for (let i = 0; i < count; i += 1) tests.push({ cause, rate, drops: 0 });
};
add(12, 'wait', 0.02);
add(6, 'ordering', 0.01);
add(8, 'shared data', 0.015);
add(5, 'network', 1 - (1 - 0.005) ** 6);
add(9, 'stable', 0);

const RUNS = 200;
const random = createRng(47);
for (let k = 0; k < RUNS; k += 1) {
  for (const t of tests) if (random() < t.rate) t.drops += 1;
}

const s = (n, g) => String(n).padStart(g);
console.log(`${'cause'.padEnd(17)}${s('test', 6)}${s('drops', 7)}${s('rate per test', 18)}`);
for (const cause of ['wait', 'ordering', 'shared data', 'network', 'stable']) {
  const group = tests.filter((t) => t.cause === cause);
  const drops = group.reduce((a, t) => a + t.drops, 0);
  const rate = ((drops / (group.length * RUNS)) * 100).toFixed(2);
  console.log(`${cause.padEnd(17)}${s(group.length, 6)}${s(drops, 7)}${s(`%${rate}`, 18)}`);
}

const REAL_DEFECT = 0.25;
const scenario = (threshold) => {
  const quarantined = tests.map((t) => (t.drops / RUNS) * 100 > threshold);
  const random2 = createRng(53);
  let red = 0;
  let escaped = 0;
  for (let k = 0; k < RUNS; k += 1) {
    let dropped = false;
    tests.forEach((t, i) => {
      if (quarantined[i] === false && random2() < t.rate) dropped = true;
    });
    if (dropped) red += 1;
    if (random2() < REAL_DEFECT) {
      const protector = Math.floor(random2() * tests.length);
      if (quarantined[protector]) escaped += 1;
    }
  }
  return { count: quarantined.filter(Boolean).length, red, escaped };
};

console.log(`${'quarantine threshold'.padEnd(23)}${s('quarantined', 13)}${s('red run', 15)}${s('escaped defect', 19)}`);
for (const threshold of [100, 2.5, 1.5, 0.5]) {
  const r = scenario(threshold);
  const name = threshold === 100 ? 'none' : `%${threshold.toFixed(1)}`;
  console.log(`${name.padEnd(23)}${s(r.count, 13)}${s(`%${((r.red / RUNS) * 100).toFixed(1)}`, 15)}`
    + `${s(`%${((r.escaped / RUNS) * 100).toFixed(1)}`, 19)}`);
}
```

```
cause              test  drops     rate per test
wait                 12     37             %1.54
ordering              6     10             %0.83
shared data           8     32             %2.00
network               5     28             %2.80
stable                9      0             %0.00
quarantine threshold     quarantined        red run     escaped defect
none                               0          %47.5               %0.0
%2.5                               8          %36.0               %4.5
%1.5                              13          %26.5               %9.5
%0.5                              25           %9.0              %16.5
```

The individual rows of the first table look harmless: even the most flaky group drops
only three times in a hundred runs. The first row of the second table breaks that
impression — with no quarantine, 47.5 percent of runs are red because of flakiness. That
growth comes from multiplication: for a single run to pass clean, all forty of the forty
tests have to pass, and multiplying the individual pass probabilities together yields a
43.67 percent expectation of red. The 47.5 percent measured across two hundred runs is
that expectation's sampling fluctuation.

This is where the end-to-end scale departs from the unit scale. A two-percent drop rate
is a tolerable flaw in a single unit test; in a forty-test end-to-end team, it is a
system property that turns roughly every second run red. And the causes of drops are not
fixed: the largest shares in the measurement are network and shared data, and neither
can be fixed inside a single test — one is tied to the call count, the other to the
fixture being shared.

#### The Two Faces of Quarantine

The remaining rows of the second table scan the quarantine threshold. As the threshold
drops, the number of quarantined tests rises, the red-run rate falls, and the **escaped
real defect** rate rises. A 2.5 percent threshold removes eight tests from the main run:
the red-run rate falls from 47.5 percent to 36.0 percent, while in exchange 4.5 percent
of runs let a real regression through unseen. A half-percent threshold quarantines a
quarter of the team; the signal is nearly clean — a 9.0 percent red-run rate — but 16.5
percent of runs carry a regression and still turn green.

Quarantine is therefore not a solution but a trade: it buys down noise at the cost of
coverage. The measure of that decision is visible in the table too — whichever threshold
is chosen, the escaped-defect rate approaches the share of tests taken into quarantine.

There is one more detail. The quarantine decision looks at the rate **measured** over
two hundred runs, not the true drop rate. The measured values deviate from the model's
true values: the wait group's true rate is two percent, the measured rate is 1.54
percent. That deviation grows as the sample shrinks, and stable tests start entering the
quarantine list while flaky ones start staying out of it.

#### Green Run, Standing Defect

The concrete face of this trade-off shows up in a single run. The borrowing flow has
four tests; the second checks that a warning is shown to the member when the loan limit
is exceeded. The flow's first version gives the correct status code for a rejected
request, but does not carry the warning.

```js
// flow.mjs — version 1: end-to-end visible result of a loan request
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: null };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');
```

```js
// quarantine.mjs — version 1: quarantined test names
export const quarantine = new Set();
```

```js
// team.test.mjs — the end-to-end team's four tests, checking the quarantine list
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { requestLoan, cardStatus } from './flow.mjs';
import { quarantine } from './quarantine.mjs';

const runTest = (name, body) => test(name, { skip: quarantine.has(name) }, body);

runTest('an eligible request is approved', () => {
  assert.equal(requestLoan({ openLoans: 1 }).status, 201);
});

runTest('a warning is shown when the loan limit is exceeded', () => {
  const result = requestLoan({ openLoans: 5 });

  assert.equal(result.status, 409);
  assert.equal(result.warning, 'loan-limit-exceeded');
});

runTest('the card shows loaned after approval', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 1 })), 'loaned');
});

runTest('the card stays shelved on a rejected request', () => {
  assert.equal(cardStatus(requestLoan({ openLoans: 5 })), 'shelved');
});
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
not ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 1
# skipped 0
```

The second test caught the defect. Now suppose this test makes a large number of
network calls and comes out flaky in measurement; by the rule, it gets quarantined.

```js
// quarantine.mjs — version 2: the test found to be flaky is quarantined
export const quarantine = new Set(['a warning is shown when the loan limit is exceeded']);
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded # SKIP
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 3
# fail 0
# skipped 1
```

The code has not changed, the defect stands, and the run is green. This is the class
that escapes, and it is a single instance of the "escaped real defect" column measured
in the table: every test taken into quarantine takes the defect class it was protecting
along with it. That is why it matters for a skipped test to stay visible in the output —
if how many tests a green run skipped cannot be read, the trade-off becomes invisible.

The discipline has two parts, and both are completed in this run: **the source of
flakiness is fixed** (here, the number of network calls and the timeout), and then
**the test is taken out of quarantine**. Below, both happen together: the flow is made
to carry the warning, and the list is cleared.

```js
// flow.mjs — version 2: rejected request now carries the warning too
export const LIMIT = 5;

export function requestLoan(member) {
  if (member.openLoans >= LIMIT) return { status: 409, warning: 'loan-limit-exceeded' };
  return { status: 201, warning: null };
}

export const cardStatus = (result) => (result.status === 201 ? 'loaned' : 'shelved');
```

```js
// quarantine.mjs — version 3: source of flakiness fixed, list cleared
export const quarantine = new Set();
```

```bash
node --test --test-reporter=tap team.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail|skipped))'
```

```
ok 1 - an eligible request is approved
ok 2 - a warning is shown when the loan limit is exceeded
ok 3 - the card shows loaned after approval
ok 4 - the card stays shelved on a rejected request
# tests 4
# pass 4
# fail 0
# skipped 0
```

A quarantine entry becomes permanent when it has no owner, no justification, and no
exit criterion; permanent quarantine is a slow drift toward the state in the tables'
last row — a clean signal and a 16.5 percent escaped-defect rate.

The run-dependent side of the cost is small here: the four tests took about 33 ms. The
run-independent numbers are in the measurement — forty tests across two hundred runs
produced eight thousand test executions, and every quarantine entry means one
maintenance debt, one lost defect class.

#### Summary

- At end-to-end scale, flakiness is a property of the team, not of a single test: forty
  tests each dropping around two percent per run turned 47.5 percent of runs red.
- The largest shares of measured drops were the network (2.80%) and shared-data (2.00%)
  causes; neither can be fixed inside a single test.
- As the quarantine threshold falls, red runs fall and escaped real defects rise: 36.0%
  and 4.5% at the 2.5 percent threshold; 9.0% and 16.5% at the half-percent threshold.
- The quarantine decision rests on the measured rate, not the true rate; the
  two-hundred-run sample showed the wait group's true two-percent rate as 1.54 percent.
- The single-run example made the trade-off concrete: when the quarantined test was
  skipped, the defect stayed in place while the run turned green.

#### Next Step

Every test here ran one at a time: forty tests in sequence, one finishing before the
next started. The same number always stood behind the quarantine discussion — running
time. A flaky test could have been rerun instead of quarantined, more cases could have
been added to cut the escaped-defect rate; both draw on the same budget, elapsed time.
That changes the question itself: is running tests in sequence a necessity, or a
choice? The next lesson takes on that question through the run order itself, and
measures what is paid, and what isolation requirement, in exchange for shortening the
duration.

### Parallel Execution

Source: https://academia.sh/en/courses/integration-testing/parallel-execution

Measuring the time gained and the isolation lost by splitting an end-to-end team into workers; the ceiling the serial share puts on speedup, the collisions a shared fixture produces, the race defect visible only in parallel, and isolation's cost and blind spot alike.

All of the previous lesson's tests ran one at a time, and the same number always stood
behind the quarantine discussion: running time. The question here is no longer which
test to write but whether running tests in sequence is a necessity or a choice.

It is a choice, and it has a cost. Splitting a team across **workers** shortens
duration; that same move also makes **test isolation** mandatory, because every sharing
that was harmless while running in sequence is now used at the same time. Test isolation
should not be confused with the isolation level from the Advanced SQL course: there, the
rule is about whether concurrent transactions can see each other's data; here, it is
about two tests not touching each other's fixture. This lesson counts both sides: the
time gained and the price paid for isolation.

#### Duration and Serial Share

Total duration is set not only by how the work is split but also by the part that
cannot be split: team setup, compilation, and reporting each run once, as a single
piece, on every run. This is the **serial share** established in the Horizontal and
Vertical Scaling lesson of the Introduction to System Design course; its arithmetic is
not repeated here, only applied to the end-to-end team.

**PC1 — forty tests: twenty at three units, twelve at six units, eight at fifteen
units; a serial share of twelve units.** Durations are model units, not
machine-dependent. The distribution starts from the longest test and places it on the
least-loaded worker.

```js
// scheduling.mjs — total duration of forty tests by worker count (model units)
const durations = [
  ...Array(20).fill(3), ...Array(12).fill(6), ...Array(8).fill(15),
];
const SERIAL = 12;

const distribute = (workerCount) => {
  const load = Array(workerCount).fill(0);
  for (const duration of [...durations].sort((a, b) => b - a)) {
    const least = load.indexOf(Math.min(...load));
    load[least] += duration;
  }
  return Math.max(...load);
};

const totalWork = durations.reduce((a, b) => a + b, 0);
const longest = Math.max(...durations);
const single = SERIAL + distribute(1);
const s = (n, g) => String(n).padStart(g);
console.log(`${'worker'.padEnd(6)}${s('parallel', 9)}${s('total', 8)}${s('speedup', 10)}${s('serial share', 14)}`);
for (const worker of [1, 2, 4, 8, 16, 40]) {
  const parallel = distribute(worker);
  const total = SERIAL + parallel;
  console.log(`${String(worker).padEnd(6)}${s(parallel, 9)}${s(total, 8)}`
    + `${s(`${(single / total).toFixed(2)}x`, 10)}${s(`%${((SERIAL / total) * 100).toFixed(1)}`, 14)}`);
}
console.log(`work ${totalWork} units, serial ${SERIAL} units, longest test ${longest} units, floor ${SERIAL + longest} units`);
```

```
worker parallel   total   speedup  serial share
1           252     264     1.00x          %4.5
2           126     138     1.91x          %8.7
4            63      75     3.52x         %16.0
8            33      45     5.87x         %26.7
16           18      30     8.80x         %40.0
40           15      27     9.78x         %44.4
work 252 units, serial 12 units, longest test 15 units, floor 27 units
```

The first two rows behave as expected: two workers cut duration almost in half. What
follows does not. At eight workers, speedup stays at 5.87; at forty, at 9.78 — even
given as many workers as tests, duration cannot drop below a tenth. Two limits are at
work: the serial share holds a part that climbs to forty-four percent of the total, and
no distribution can run shorter than the longest test. That is the twenty-seven-unit
floor.

Two tasks follow from this, and both come before adding workers: shrinking the serial
share and splitting the longest test. Adding more workers, without doing either of
these first, does not repay its cost.

#### The Shared Fixture

The price paid is isolation. The run below distributes four tests across workers and
breaks each test into steps, advancing them worker by worker in turn; the interleaving
that a parallel run does to tests genuinely happens here. Each test sets up a member
record, reads the open-loan count, writes a loan, verifies the limit was not exceeded,
and tears down its fixture.

Three options are given as arguments: how many workers, whether the loan decision is
made in **two steps** (read, then write) or in **one step**, and whether the fixture is
shared.

```js
// run.mjs — four tests run by worker count, service mode, and isolation mode
import { DatabaseSync } from 'node:sqlite';

const [workerCount = '1', service = 'two-step', mode = 'shared'] = process.argv.slice(2);
const WORKERS = Number(workerCount);
const TEST_COUNT = 4;
const LIMIT = 1;
const measurement = { step: 0, setupQuery: 0, collision: 0, dropped: [] };

const setupSchema = () => {
  const db = new DatabaseSync(':memory:');
  db.exec(`CREATE TABLE member (member_no TEXT PRIMARY KEY);
           CREATE TABLE loan (member_no TEXT, book_no TEXT UNIQUE)`);
  measurement.setupQuery += 2;
  return db;
};

const databases = mode === 'per-worker-schema'
  ? Array.from({ length: WORKERS }, setupSchema)
  : Array(WORKERS).fill(setupSchema());

const count = (db, memberNo) => db.prepare('SELECT COUNT(*) AS n FROM loan WHERE member_no = ?').all(memberNo)[0].n;

const makeTest = (no, worker) => {
  const db = databases[worker];
  const memberNo = mode === 'shared' ? 'U-17' : `U-${no}`;
  const bookNo = `K-${no}`;
  const local = {};
  const write = () => db.prepare('INSERT INTO loan VALUES (?, ?)').run(memberNo, bookNo);
  const steps = [
    () => {
      measurement.setupQuery += 1;
      try { db.prepare('INSERT INTO member VALUES (?)').run(memberNo); } catch { measurement.collision += 1; }
    },
    ...(service === 'two-step'
      ? [() => { local.n = count(db, memberNo); }, () => { if (local.n < LIMIT) write(); }]
      : [() => { if (count(db, memberNo) < LIMIT) write(); }]),
    () => {
      const n = count(db, memberNo);
      if (n > LIMIT) measurement.dropped.push(`test ${no}: open loans ${n}, limit ${LIMIT}`);
    },
    () => {
      db.prepare('DELETE FROM loan WHERE book_no = ?').run(bookNo);
      db.prepare('DELETE FROM member WHERE member_no = ?').run(memberNo);
    },
  ];
  return { steps, i: 0 };
};

const queues = Array.from({ length: WORKERS }, () => []);
for (let no = 1; no <= TEST_COUNT; no += 1) queues[(no - 1) % WORKERS].push(makeTest(no, (no - 1) % WORKERS));

while (queues.some((q) => q.length > 0)) {
  for (const queue of queues) {
    const t = queue[0];
    if (t === undefined) continue;
    t.steps[t.i]();
    measurement.step += 1;
    t.i += 1;
    if (t.i === t.steps.length) queue.shift();
  }
}

console.log(`worker ${WORKERS}  service ${service}  mode ${mode}`);
console.log(`step ${measurement.step}  setup query ${measurement.setupQuery}`
  + `  setup collision ${measurement.collision}  dropped test ${measurement.dropped.length}/${TEST_COUNT}`);
for (const line of measurement.dropped) console.log(line);
```

With one worker — that is, in sequence — the team is clean.

```bash
node run.mjs 1 two-step shared
```

```
worker 1  service two-step  mode shared
step 20  setup query 6  setup collision 0  dropped test 0/4
```

#### The Test That Drops Only in Parallel

Same code, same data, two workers.

```bash
node run.mjs 2 two-step shared
```

```
worker 2  service two-step  mode shared
step 20  setup query 6  setup collision 2  dropped test 4/4
test 1: open loans 2, limit 1
test 2: open loans 2, limit 1
test 3: open loans 2, limit 1
test 4: open loans 2, limit 1
```

Two separate events show up together. The first is a **setup collision**: two tests
tried to set up the same member record at the same time, and one of the two hit the
uniqueness constraint. This is a problem belonging to the test's own fixture.

The second is a product defect, and its name is the **check-then-act race**: the loan
decision is made in two steps — first the open-loan count is read, then the record is
written. Another worker writing between those two steps invalidates the count that was
read. Both tests read zero, both wrote, and the member's limit of one became two. In a
sequential run, this defect is **never** visible; the previous run's zero drops is the
proof of that. This is the class parallel execution catches.

The fix is to make the decision in a single, indivisible step.

```bash
node run.mjs 2 one-step shared
```

```
worker 2  service one-step  mode shared
step 16  setup query 6  setup collision 2  dropped test 0/4
```

The tests turned green, but the setup collision still stands at two — and that line is
a second warning. The two tests that hit the collision could not set up their own
member; their loan requests hit the limit, were silently rejected, and the assertion
still passed. Even while green, they did not test what they meant to test. Sharing the
fixture empties a test of meaning even when it does not drop the test.

#### Isolation's Cost and Blind Spot

Isolation has two levels: giving each test its own record set, or setting up a separate
schema for each worker. Both are run with the same defect present, the two-step
service.

```bash
node run.mjs 2 two-step per-test-records
```

```
worker 2  service two-step  mode per-test-records
step 20  setup query 6  setup collision 0  dropped test 0/4
```

```bash
node run.mjs 2 two-step per-worker-schema
```

```
worker 2  service two-step  mode per-worker-schema
step 20  setup query 8  setup collision 0  dropped test 0/4
```

The cost shows up in two numbers. Per-test records bring the collision to zero and add
nothing; per-worker schema raises the setup query count from six to eight, because each
worker creates its own tables. That gap grows linearly as the worker count grows: schema
setup is paid per worker, record sets are paid per test.

The class that escapes is right here too. Both runs are green, yet **the defect stands
in place** — the service still decides in two steps. Isolation removed the race's
precondition, because it kept two tests from touching the same member at the same time.
While it removes flakiness, isolation also makes invisible the defect class tied to the
very sharing that produced the flakiness. Real users share records; as long as the test
team does not, that race shows up only in production.

So the isolation decision is a balance to strike: the run order's isolation must be
**complete**, and the product's concurrency handling must be tested **separately**. The
run-dependent side of the cost is small here — each of the five configurations ran in
under a millisecond in this run, because what runs is an in-process model. The
run-independent numbers are in the measurements: twenty steps, six to eight setup
queries, two colliding records, and a serial share climbing from 4.5 percent to 44.4
percent.

#### Summary

- Adding workers shortens duration only down to the serial share and the longest test:
  with forty tests, even forty workers gave only a 9.78x speedup, and the floor stayed
  at twenty-seven units.
- The shared fixture produced two setup collisions in parallel; even when a colliding
  test does not drop, it fails to test what it meant to test.
- The check-then-act race showed up only under parallel execution: zero drops in the
  sequential run, all four of four tests dropped with two workers.
- Reducing the decision to one step turned the tests green; isolation, by contrast,
  turned the run green while the same defect stayed in place.
- Isolation's cost is measurable: per-test records did not raise the setup query count,
  per-worker schema raised it from six to eight, and that gap grows per worker.

#### Course Wrap-Up

This course's fifteen lessons answered the same three questions. Every row in the table
states what a test form sees, what it cannot see, and what it costs.

| Lesson | Defect Class Caught | Missed | Cost |
|---|---|---|---|
| The Scope of Integration Testing | schema mismatch | recipient identity mismatch | 4 queries per round, 9-line fixture |
| Test Environment Management | rejection of an unknown recipient | process-lifetime state | 1 process, 6 steps, 1 file per run |
| Test Data Management | distribution-dependent report defect | content-dependent export defect | 772 load queries, 78 kilobytes |
| Database Testing | data loss in the down migration | backfill value mismatch | 13 migration steps, 21 queries, 1 file |
| Mocking External Services | request shape drift | the frozen response going stale | 0 processes, 0 network calls on replay |
| API Testing | structural mismatch | semantic mismatch | 2 processes, at least 1 request per test, 16 hand-maintained rules |
| Consumer-Driven Contract Testing | the loss of a field written into the contract | a field read by a branch that never ran | 2 components, 100 requests, a 241-byte file |
| Schema Compatibility Checking | breaking change | a change that alters meaning without breaking shape | 0 processes, 0 requests, a 9-row class table |
| Developing With Mock Servers | shape deviation | state deviation | 2 processes, 10 requests, a 15-line fixture |
| The Test Pyramid | boundary comparison, at the unit level | status-code mapping, at the upper levels | 242 queries and 24 requests for 24 cases |
| Browser Automation | the card status not updating | the mislabeled button | 16,477 polls across 2,000 loads |
| Mobile Automation | version-dependent date parsing | 24/30 of cell-specific defects | 12 cell runs, a 30-cell matrix |
| Visual Verification | the button tone lightening | an inactive button producing the same pixels | 24 baseline images |
| Flakiness Management | the missing warning on a rejected request | the class a quarantined test was protecting | 40 tests × 200 runs, 8,000 test executions |
| Parallel Execution | the check-then-act race | the sharing defect isolation hides | setup queries from 6 to 8, serial share from 4.5% to 44.4% |

The middle column is the course's rule: a test layer's value can only be told together
with its **limit**. In none of the fifteen rows is there a "safer" option; every row has
a class caught, a class missed, and a cost paid. The only way to defend a test form is
to write out this triple, and the only justification for adding a form is a defect class
with **no counterpart in any other layer**.

The course's own limit is written by the same rule. The first topic asked whether test
doubles matched reality, the second separated out the checkable part of a contract, the
third ran the path the user sees as a whole. All three asked the same question: **does
the function work correctly?** How the system behaves under load, what it gives up
under attack, and how it degrades under failure were tested in no lesson here. The
Non-Functional Testing course takes on these three questions.