API Testing

The previous lesson set up recording and replaying an external service’s response, and it closed with a warning: a recording freezes the service’s behavior from yesterday. When the catalog service renames a field today, the test that runs against the recording still stays green, because what it compares against is not the service itself but yesterday’s copy.

This lesson builds the check that closes that gap. Its question is not API design; request methods, status codes, and versioning were established in the Web API Design course and are not repeated here. The question here is: does a live service’s response arrive in the shape the consumer expects, which check tells you that, which defect class it catches, and which one it misses.

The Silence of a Frozen Record

The library loan system has two services: the catalog service, which holds book records, and the loan service, which lends books to members. The loan service reads a book record from the catalog and looks at four of its fields: status, shelf count, loan duration, and ID. The script below feeds yesterday’s recorded body and today’s body into the same decision function.

// frozen.mjs — yesterday's recorded catalog response vs. today's, fed into the same consumer function
const RECORD = {
  isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
  publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2,
  status: "shelved", loanDays: 14,
};

// The catalog service renamed a field today: loanDays -> loanDurationDays.
const LIVE = { ...RECORD, loanDurationDays: RECORD.loanDays };
delete LIVE.loanDays;

// The loan service's decision function: it reads whatever is in the record.
const loanDecision = (book, today) =>
  book.status !== "shelved" || book.shelfCount < 1
    ? { granted: false, dueDay: null }
    : { granted: true, dueDay: today + book.loanDays };

const TODAY = 20260, common = Object.keys(RECORD).filter((a) => a in LIVE);
for (const [name, body] of [["record", RECORD], ["live", LIVE]]) {
  const decision = loanDecision(body, TODAY);
  console.log(`${name.padEnd(6)} fields ${String(Object.keys(body).length).padStart(2)}` +
    `  granted=${decision.granted}  dueDay=${decision.dueDay}`);
}
const missing = Object.keys(RECORD).filter((a) => !(a in LIVE));
console.log(`overlapping fields ${common.length}/${Object.keys(RECORD).length}; missing from live response: ${missing.join(", ")}`);
console.log(`replay test runs against the record -> dueDay 20274 expected, green`);

record fields  9  granted=true  dueDay=20274
live   fields  9  granted=true  dueDay=NaN
overlapping fields 8/9; missing from live response: loanDays
replay test runs against the record -> dueDay 20274 expected, green

Both bodies carry nine fields and eight of them overlap. But when the loan service works with the live response, it cannot compute the due day. The record-and-replay setup cannot see this, because no request ever reaches the live service; the caught defect class is zero. The missing check is one that takes the response from where it actually comes from and tests it against a written expectation.

The Response Schema

A schema puts in writing which fields the response carries and in what type. The definition below has nine fields and sixteen rules; the rules amount to type, pattern, value set, and lower bound. The checker itself is a few lines and returns a list of mismatches.

// schema.mjs — the catalog response schema and a field-by-field checker
export const BOOK_SCHEMA = {
  isbn: { type: "string", pattern: /^97[89]-\d{10}$/ },
  title: { type: "string" },
  author: { type: "string" },
  publicationYear: { type: "integer", min: 1400 },
  branch: { type: "string", set: ["central", "coast", "hill"] },
  totalCount: { type: "integer", min: 0 },
  shelfCount: { type: "integer", min: 0 },
  status: { type: "string", set: ["shelved", "loaned", "lost"] },
  loanDays: { type: "integer", min: 1 },
};

const typeHolds = (rule, d) => rule.type === "integer"
  ? Number.isInteger(d)
  : typeof d === "string";

// The returned array is empty, or each line is one mismatch.
export function checkAgainstSchema(schema, body) {
  const mismatch = [];
  for (const [name, rule] of Object.entries(schema)) {
    const d = body[name];
    if (d === undefined) { mismatch.push(`${name}: field missing`); continue; }
    if (!typeHolds(rule, d)) { mismatch.push(`${name}: not type ${rule.type} (${typeof d})`); continue; }
    if (rule.pattern && !rule.pattern.test(d)) mismatch.push(`${name}: does not match pattern`);
    if (rule.set && !rule.set.includes(d)) mismatch.push(`${name}: value outside set ${d}`);
    if (rule.min !== undefined && d < rule.min) mismatch.push(`${name}: below ${rule.min}`);
  }
  const extra = Object.keys(body).filter((a) => !(a in schema));
  if (extra.length) mismatch.push(`field not in schema: ${extra.join(", ")}`);
  return mismatch;
}

The boundary between what the schema says and what it does not say is this lesson’s axis. The schema says the shelfCount field cannot be less than zero; it does not say that number is the correct number.

Checking Against the Live Service

The catalog service is a local process and can be deliberately broken with an environment variable. Three defects are defined: type returns the shelf count as a string, meaning writes the total count in place of the shelf count, and value bumps the reference book’s seven-day loan duration up to fourteen.

// catalog.mjs — catalog service; deliberately broken by the CATALOG_DEFECT variable
// Usage: node catalog.mjs <port|0>   port 0 picks a free port.
import { createServer } from "node:http";

if (process.argv[2] === undefined) {
  console.log("usage: node catalog.mjs <port|0>");
  process.exit(0);
}

const DEFECT = process.env.CATALOG_DEFECT ?? "none";
const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
    publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2, status: "shelved", loanDays: 14 },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", author: "Knuth",
    publicationYear: 1997, branch: "hill", totalCount: 3, shelfCount: 0, status: "loaned", loanDays: 7 },
};

const output = (k) => {
  if (DEFECT === "type") return { ...k, shelfCount: String(k.shelfCount) };
  if (DEFECT === "meaning") return { ...k, shelfCount: k.totalCount };
  if (DEFECT === "value") return { ...k, loanDays: 14 };
  return { ...k };
};

const server = createServer((request, response) => {
  response.sendDate = false;
  const book = BOOKS[request.url.split("?")[0].replace("/book/", "")];
  response.writeHead(book ? 200 : 404, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(book ? output(book) : { error: "book not found" }));
});

server.listen(Number(process.argv[2]), "127.0.0.1", () =>
  console.log(`ready ${server.address().port} defect=${DEFECT}`));

The test file starts the service as a separate process, reads the chosen port from the process’s first line, and checks three things: schema conformance, an invariant, and the consumer’s decision.

// api.test.mjs — request/response check against the live catalog service
// Usage: CATALOG_DEFECT=none|type|meaning|value node --test api.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
  proc.stdout.once("data", (v) => {
    base = `http://127.0.0.1:${String(v).split(" ")[1]}`;
    resolve();
  });
}));
after(() => proc.kill());

const book = async (isbn) => (await fetch(`${base}/book/${isbn}`)).json();

test("schema: both book responses match the schema", async () => {
  for (const isbn of ["978-0262033848", "978-0201896831"]) {
    const mismatch = checkAgainstSchema(BOOK_SCHEMA, await book(isbn));
    assert.equal(mismatch.length, 0, `${isbn} -> ${mismatch.join("; ")}`);
  }
});

test("invariant: a loaned book has a shelf count of zero", async () => {
  const k = await book("978-0201896831");
  assert.equal(k.status === "loaned" && k.shelfCount > 0, false,
    `${k.isbn} -> status ${k.status} but shelfCount ${k.shelfCount}`);
});

test("consumer: the due day is computed for a shelved book", async () => {
  const k = await book("978-0262033848");
  assert.equal(k.status, "shelved");
  assert.equal(Number.isInteger(20260 + k.loanDays), true);
});

The same set of tests runs against four states of the service: the sound version and three defects.

for k in none type meaning value; do
  echo "== defect=$k =="
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs | grep -E '^(ok|not ok|# (pass|fail))'
done

== defect=none ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0
== defect=type ==
not ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=meaning ==
ok 1 - schema: both book responses match the schema
not ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=value ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0

The red-to-green turn is inside this table. When the type defect is introduced, the first test fails; when the catalog service returns the shelf count as an integer again — that is, in the defect=none row — the same test turns green. This cycle is proof that the check works.

Caught and Missed Classes

The reason a test fails can be read in a single line.

for k in type meaning value; do
  printf 'defect=%-7s ' "$k"
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs |
    grep -A1 'error: |-' | tail -1 | sed 's/^ *//' | grep . || echo "no failing test"
done

defect=type    978-0262033848 -> shelfCount: not type integer (string)
defect=meaning 978-0201896831 -> status loaned but shelfCount 3
defect=value   no failing test

The class the schema check catches is structural mismatch: a field’s absence, a change in its type, or falling outside the pattern or value set. The type defect falls into this class, and the mismatch is reported by field name. The field rename from the lesson’s opening is of the same class; a schema check that runs against the live service sees it on the very first request.

The class the schema check misses is semantic mismatch. With the meaning defect, the shelf count is still an integer greater than zero; every rule of the schema is satisfied and the first test stays green. What catches this defect is not the schema but a hand-written invariant: a loaned book’s shelf count must be zero. The invariant states the relationship between two fields; the schema, on the other hand, treats each field on its own. This is the cost of closing the missed class — every relationship has to be written by hand, one at a time.

The third defect falls outside both. The reference book’s loan duration should be seven days but comes back as fourteen; fourteen is an integer the schema accepts and violates no invariant. All three tests stay green. What’s missing here is not a check but the source of the expected result: the only place that knows this book’s duration is seven is the contract itself. The test oracle problem from the Quality and Testing Fundamentals course resurfaces here, and the next two lessons bring in that source from two different places.

The Cost of the Check

Cost is measured two ways: numbers independent of the run, and where the run’s time goes.

// cost.mjs — the cost of the API test: how many fields checked, how many requests, where the time goes
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

const ISBN = ["978-0262033848", "978-0201896831"];
const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
const base = `http://127.0.0.1:${port}`;
const REPEAT = 50, bodies = [];

const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) {
  for (const isbn of ISBN) bodies.push(await (await fetch(`${base}/book/${isbn}`)).json());
}
const t1 = performance.now();
for (const g of bodies) checkAgainstSchema(BOOK_SCHEMA, g);
const t2 = performance.now();
proc.kill();

const fields = Object.keys(BOOK_SCHEMA).length;
const rules = Object.values(BOOK_SCHEMA).reduce((n, k) => n + Object.keys(k).length, 0);
console.log(`schema: ${fields} fields, ${rules} rules`);
console.log(`run: ${bodies.length} requests, ${bodies.length * fields} field checks, 2 processes`);
console.log(`request time greater than check time : ${t1 - t0 > t2 - t1}`);
console.log(`ratio at least fifty times           : ${(t1 - t0) / (t2 - t1) >= 50}`);

schema: 9 fields, 16 rules
run: 100 requests, 900 field checks, 2 processes
request time greater than check time : true
ratio at least fifty times           : true

Absolute times are machine-dependent, so the output reports two comparisons instead of raw milliseconds. What does not change is the direction of the ratio: nearly all of the time goes into the request round trip, and the schema check itself is negligible. The cost of an API test is not the complexity of the check but the decision to step outside the process — this suite brings up two processes and sends a request over the network for every test.

The numbers independent of the run are the measure of upkeep. Nine fields and sixteen rules are hand-written, and when the catalog service adds a field, the schema has to be updated too; if it is not, the “field not in schema” line fails the test. This is a strict choice, and a deliberate one: a check that stays silent is indistinguishable from a check that does not exist.

Summary

• Record and replay never sends a request to the live service; a field rename is invisible to a test running against the recording, and visible on the very first call to a check that runs a real request.
• A schema puts the response’s fields in writing with type, pattern, value set, and bound; sixteen rules were enough for a nine-field record.
• The class the schema check catches is structural mismatch: a missing field, a type change, a value outside the set. The type defect turned the test red, and it went back to green once the defect was reverted.
• The class the schema check misses is semantic mismatch: a value that follows the rules but is wrong. Only a hand-written invariant can see the relationship between two fields, and neither one knows what the correct value actually is.
• Cost: two processes, at least one request per test, sixteen hand-maintained rules; nearly all of the run’s time goes into the request round trip, and the check itself is negligible.

Next Step

This lesson’s schema describes the shape the catalog service produces. How many of these fields the loan service actually reads is written nowhere: nine fields were checked, yet the decision function looked at four. The gap is not pointless — when the provider wants to remove a field, the schema has no answer to “who reads this.” The next lesson gathers the expectation from the consumer’s side instead: the fields the consumer’s running code reads are recorded, a contract is generated from that recording, and the contract runs in the provider’s team. The measure is the gap between the number of fields the consumer actually uses and the number of fields the contract covers.