---
title: 'API Testing'
source: 'https://academia.sh/en/courses/integration-testing/api-testing'
course: 'Integration, Contract and End-to-End Testing'
language: en
updated: '2026-08-23T14:25:13+00:00'
license: 'CC BY-SA 4.0'
---

# API Testing

Checking a live service's response against a schema: catching with a real request the field change a frozen recording cannot see, separating structural mismatch from semantic mismatch, and measuring the check's cost by the fields checked, the requests sent, and the share of time.

The previous lesson set up recording and replaying an external service's response, and it
closed with a warning: a recording freezes the service's behavior from yesterday. When the
catalog service renames a field today, the test that runs against the recording still stays
green, because what it compares against is not the service itself but yesterday's copy.

This lesson builds the check that closes that gap. Its question is not API design; request
methods, status codes, and versioning were established in the Web API Design course and are
not repeated here. The question here is: does a live service's response arrive in the shape
the consumer expects, which check tells you that, which defect class it catches, and which
one it misses.

## The Silence of a Frozen Record

The library loan system has two services: the **catalog service**, which holds book records,
and the **loan service**, which lends books to members. The loan service reads a book record
from the catalog and looks at four of its fields: status, shelf count, loan duration, and ID.
The script below feeds yesterday's recorded body and today's body into the same decision
function.

```js
// frozen.mjs — yesterday's recorded catalog response vs. today's, fed into the same consumer function
const RECORD = {
  isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
  publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2,
  status: "shelved", loanDays: 14,
};

// The catalog service renamed a field today: loanDays -> loanDurationDays.
const LIVE = { ...RECORD, loanDurationDays: RECORD.loanDays };
delete LIVE.loanDays;

// The loan service's decision function: it reads whatever is in the record.
const loanDecision = (book, today) =>
  book.status !== "shelved" || book.shelfCount < 1
    ? { granted: false, dueDay: null }
    : { granted: true, dueDay: today + book.loanDays };

const TODAY = 20260, common = Object.keys(RECORD).filter((a) => a in LIVE);
for (const [name, body] of [["record", RECORD], ["live", LIVE]]) {
  const decision = loanDecision(body, TODAY);
  console.log(`${name.padEnd(6)} fields ${String(Object.keys(body).length).padStart(2)}` +
    `  granted=${decision.granted}  dueDay=${decision.dueDay}`);
}
const missing = Object.keys(RECORD).filter((a) => !(a in LIVE));
console.log(`overlapping fields ${common.length}/${Object.keys(RECORD).length}; missing from live response: ${missing.join(", ")}`);
console.log(`replay test runs against the record -> dueDay 20274 expected, green`);
```

```
record fields  9  granted=true  dueDay=20274
live   fields  9  granted=true  dueDay=NaN
overlapping fields 8/9; missing from live response: loanDays
replay test runs against the record -> dueDay 20274 expected, green
```

Both bodies carry nine fields and eight of them overlap. But when the loan service works
with the live response, it cannot compute the due day. The record-and-replay setup cannot
see this, because **no request ever reaches the live service**; the caught defect class is
zero. The missing check is one that takes the response from where it actually comes from and
tests it against a written expectation.

## The Response Schema

A **schema** puts in writing which fields the response carries and in what type. The
definition below has nine fields and sixteen rules; the rules amount to type, pattern, value
set, and lower bound. The checker itself is a few lines and returns a list of mismatches.

```js
// schema.mjs — the catalog response schema and a field-by-field checker
export const BOOK_SCHEMA = {
  isbn: { type: "string", pattern: /^97[89]-\d{10}$/ },
  title: { type: "string" },
  author: { type: "string" },
  publicationYear: { type: "integer", min: 1400 },
  branch: { type: "string", set: ["central", "coast", "hill"] },
  totalCount: { type: "integer", min: 0 },
  shelfCount: { type: "integer", min: 0 },
  status: { type: "string", set: ["shelved", "loaned", "lost"] },
  loanDays: { type: "integer", min: 1 },
};

const typeHolds = (rule, d) => rule.type === "integer"
  ? Number.isInteger(d)
  : typeof d === "string";

// The returned array is empty, or each line is one mismatch.
export function checkAgainstSchema(schema, body) {
  const mismatch = [];
  for (const [name, rule] of Object.entries(schema)) {
    const d = body[name];
    if (d === undefined) { mismatch.push(`${name}: field missing`); continue; }
    if (!typeHolds(rule, d)) { mismatch.push(`${name}: not type ${rule.type} (${typeof d})`); continue; }
    if (rule.pattern && !rule.pattern.test(d)) mismatch.push(`${name}: does not match pattern`);
    if (rule.set && !rule.set.includes(d)) mismatch.push(`${name}: value outside set ${d}`);
    if (rule.min !== undefined && d < rule.min) mismatch.push(`${name}: below ${rule.min}`);
  }
  const extra = Object.keys(body).filter((a) => !(a in schema));
  if (extra.length) mismatch.push(`field not in schema: ${extra.join(", ")}`);
  return mismatch;
}
```

The boundary between what the schema says and what it does not say is this lesson's axis. The
schema says the `shelfCount` field cannot be less than zero; it does not say that number is
the **correct** number.

## Checking Against the Live Service

The catalog service is a local process and can be deliberately broken with an environment
variable. Three defects are defined: `type` returns the shelf count as a string, `meaning`
writes the total count in place of the shelf count, and `value` bumps the reference book's
seven-day loan duration up to fourteen.

```js
// catalog.mjs — catalog service; deliberately broken by the CATALOG_DEFECT variable
// Usage: node catalog.mjs <port|0>   port 0 picks a free port.
import { createServer } from "node:http";

if (process.argv[2] === undefined) {
  console.log("usage: node catalog.mjs <port|0>");
  process.exit(0);
}

const DEFECT = process.env.CATALOG_DEFECT ?? "none";
const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
    publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2, status: "shelved", loanDays: 14 },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", author: "Knuth",
    publicationYear: 1997, branch: "hill", totalCount: 3, shelfCount: 0, status: "loaned", loanDays: 7 },
};

const output = (k) => {
  if (DEFECT === "type") return { ...k, shelfCount: String(k.shelfCount) };
  if (DEFECT === "meaning") return { ...k, shelfCount: k.totalCount };
  if (DEFECT === "value") return { ...k, loanDays: 14 };
  return { ...k };
};

const server = createServer((request, response) => {
  response.sendDate = false;
  const book = BOOKS[request.url.split("?")[0].replace("/book/", "")];
  response.writeHead(book ? 200 : 404, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(book ? output(book) : { error: "book not found" }));
});

server.listen(Number(process.argv[2]), "127.0.0.1", () =>
  console.log(`ready ${server.address().port} defect=${DEFECT}`));
```

The test file starts the service as a separate process, reads the chosen port from the
process's first line, and checks three things: schema conformance, an **invariant**, and the
consumer's decision.

```js
// api.test.mjs — request/response check against the live catalog service
// Usage: CATALOG_DEFECT=none|type|meaning|value node --test api.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
  proc.stdout.once("data", (v) => {
    base = `http://127.0.0.1:${String(v).split(" ")[1]}`;
    resolve();
  });
}));
after(() => proc.kill());

const book = async (isbn) => (await fetch(`${base}/book/${isbn}`)).json();

test("schema: both book responses match the schema", async () => {
  for (const isbn of ["978-0262033848", "978-0201896831"]) {
    const mismatch = checkAgainstSchema(BOOK_SCHEMA, await book(isbn));
    assert.equal(mismatch.length, 0, `${isbn} -> ${mismatch.join("; ")}`);
  }
});

test("invariant: a loaned book has a shelf count of zero", async () => {
  const k = await book("978-0201896831");
  assert.equal(k.status === "loaned" && k.shelfCount > 0, false,
    `${k.isbn} -> status ${k.status} but shelfCount ${k.shelfCount}`);
});

test("consumer: the due day is computed for a shelved book", async () => {
  const k = await book("978-0262033848");
  assert.equal(k.status, "shelved");
  assert.equal(Number.isInteger(20260 + k.loanDays), true);
});
```

The same set of tests runs against four states of the service: the sound version and three
defects.

```bash
for k in none type meaning value; do
  echo "== defect=$k =="
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs | grep -E '^(ok|not ok|# (pass|fail))'
done
```

```
== defect=none ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0
== defect=type ==
not ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=meaning ==
ok 1 - schema: both book responses match the schema
not ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 2
# fail 1
== defect=value ==
ok 1 - schema: both book responses match the schema
ok 2 - invariant: a loaned book has a shelf count of zero
ok 3 - consumer: the due day is computed for a shelved book
# pass 3
# fail 0
```

The red-to-green turn is inside this table. When the type defect is introduced, the first
test fails; when the catalog service returns the shelf count as an integer again — that is,
in the `defect=none` row — the same test turns green. This cycle is proof that the check
works.

## Caught and Missed Classes

The reason a test fails can be read in a single line.

```bash
for k in type meaning value; do
  printf 'defect=%-7s ' "$k"
  CATALOG_DEFECT=$k node --test --test-reporter=tap api.test.mjs |
    grep -A1 'error: |-' | tail -1 | sed 's/^ *//' | grep . || echo "no failing test"
done
```

```
defect=type    978-0262033848 -> shelfCount: not type integer (string)
defect=meaning 978-0201896831 -> status loaned but shelfCount 3
defect=value   no failing test
```

**The class the schema check catches is structural mismatch:** a field's absence, a change in
its type, or falling outside the pattern or value set. The type defect falls into this class,
and the mismatch is reported by field name. The field rename from the lesson's opening is of
the same class; a schema check that runs against the live service sees it on the very first
request.

**The class the schema check misses is semantic mismatch.** With the meaning defect, the
shelf count is still an integer greater than zero; every rule of the schema is satisfied and
the first test stays green. What catches this defect is not the schema but a hand-written
invariant: a loaned book's shelf count must be zero. The invariant states the relationship
**between** two fields; the schema, on the other hand, treats each field on its own. This is
the cost of closing the missed class — every relationship has to be written by hand, one at a
time.

The third defect falls outside both. The reference book's loan duration should be seven days
but comes back as fourteen; fourteen is an integer the schema accepts and violates no
invariant. All three tests stay green. What's missing here is not a check but **the source of
the expected result**: the only place that knows this book's duration is seven is the
contract itself. The test oracle problem from the Quality and Testing Fundamentals course
resurfaces here, and the next two lessons bring in that source from two different places.

## The Cost of the Check

Cost is measured two ways: numbers independent of the run, and where the run's time goes.

```js
// cost.mjs — the cost of the API test: how many fields checked, how many requests, where the time goes
import { spawn } from "node:child_process";
import { BOOK_SCHEMA, checkAgainstSchema } from "./schema.mjs";

const ISBN = ["978-0262033848", "978-0201896831"];
const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
const base = `http://127.0.0.1:${port}`;
const REPEAT = 50, bodies = [];

const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) {
  for (const isbn of ISBN) bodies.push(await (await fetch(`${base}/book/${isbn}`)).json());
}
const t1 = performance.now();
for (const g of bodies) checkAgainstSchema(BOOK_SCHEMA, g);
const t2 = performance.now();
proc.kill();

const fields = Object.keys(BOOK_SCHEMA).length;
const rules = Object.values(BOOK_SCHEMA).reduce((n, k) => n + Object.keys(k).length, 0);
console.log(`schema: ${fields} fields, ${rules} rules`);
console.log(`run: ${bodies.length} requests, ${bodies.length * fields} field checks, 2 processes`);
console.log(`request time greater than check time : ${t1 - t0 > t2 - t1}`);
console.log(`ratio at least fifty times           : ${(t1 - t0) / (t2 - t1) >= 50}`);
```

```
schema: 9 fields, 16 rules
run: 100 requests, 900 field checks, 2 processes
request time greater than check time : true
ratio at least fifty times           : true
```

Absolute times are machine-dependent, so the output reports two comparisons instead of raw
milliseconds. What does not change is the direction of the ratio: nearly all of the time goes
into the request round trip, and the schema check itself is negligible. The cost of an API
test is not the complexity of the check but the decision to **step outside the process** —
this suite brings up two processes and sends a request over the network for every test.

The numbers independent of the run are the measure of upkeep. Nine fields and sixteen rules
are hand-written, and when the catalog service adds a field, the schema has to be updated
too; if it is not, the "field not in schema" line fails the test. This is a strict choice, and
a deliberate one: a check that stays silent is indistinguishable from a check that does not
exist.

## Summary

- Record and replay never sends a request to the live service; a field rename is invisible to
  a test running against the recording, and visible on the very first call to a check that
  runs a real request.
- A schema puts the response's fields in writing with type, pattern, value set, and bound;
  sixteen rules were enough for a nine-field record.
- The class the schema check catches is structural mismatch: a missing field, a type change,
  a value outside the set. The type defect turned the test red, and it went back to green
  once the defect was reverted.
- The class the schema check misses is semantic mismatch: a value that follows the rules but
  is wrong. Only a hand-written invariant can see the relationship between two fields, and
  neither one knows what the correct value actually is.
- Cost: two processes, at least one request per test, sixteen hand-maintained rules; nearly
  all of the run's time goes into the request round trip, and the check itself is negligible.

## Next Step

This lesson's schema describes the shape the catalog service **produces**. How many of these
fields the loan service actually reads is written nowhere: nine fields were checked, yet the
decision function looked at four. The gap is not pointless — when the provider wants to
remove a field, the schema has no answer to "who reads this." The next lesson gathers the
expectation from the consumer's side instead: the fields the consumer's running code reads
are recorded, a contract is generated from that recording, and the contract runs in the
provider's team. The measure is the gap between the number of fields the consumer actually
uses and the number of fields the contract covers.