Consumer-Driven Contract Testing

The previous lesson’s schema described the shape the catalog service produces: nine fields, sixteen rules. How many of these fields the loan service reads was written nowhere. The gap is not pointless; when the provider wants to remove a field, the schema has no answer to “who reads this.”

This lesson gathers the expectation from the other direction. In consumer-driven contract testing, the expectation is written not by the provider but by the consumer, and the provider runs it in its own team. The Application Layer and Service Interaction course measured this decision’s cost — the number of consumers broken, the number of units published together, the expand–contract sequence. Here that same decision is turned into a test, and two numbers are asked: how many fields does the consumer actually read, and how many of them does the contract cover.

Gathering the Expectation From the Consumer’s Code

Writing the expectation file by hand has a flaw: the person writing it has to remember what the code reads. The loan service’s decision function splits into three branches, and each branch reads different fields.

// loan.mjs — the loan service's decision function; three branches split off the catalog record
export function loanDecision(book, today) {
  if (book.status === "lost")
    return { granted: false, message: `lost record: ${book.title} / ${book.author}` };
  if (book.status !== "shelved" || book.shelfCount < 1)
    return { granted: false, message: `not on shelf: ${book.isbn}` };
  return { granted: true, dueDay: today + book.loanDays, message: `${book.isbn} loaned` };
}

The catalog service is the previous lesson’s process; in this lesson it carries three books and has two defects. The rename defect renames the loan duration field, and the missing defect removes the title field.

// catalog.mjs — catalog service; deliberately broken by the CATALOG_DEFECT variable
// Usage: node catalog.mjs <port|0>   defect: rename (loanDays is renamed)
//                                    defect: missing (title field is removed)
import { createServer } from "node:http";

if (process.argv[2] === undefined) {
  console.log("usage: node catalog.mjs <port|0>");
  process.exit(0);
}

const DEFECT = process.env.CATALOG_DEFECT ?? "none";
const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", author: "Cormen",
    publicationYear: 2009, branch: "central", totalCount: 4, shelfCount: 2, status: "shelved", loanDays: 14 },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", author: "Knuth",
    publicationYear: 1997, branch: "hill", totalCount: 3, shelfCount: 0, status: "loaned", loanDays: 7 },
  "978-0134685991": { isbn: "978-0134685991", title: "Lost Volume", author: "Bloch",
    publicationYear: 2018, branch: "coast", totalCount: 1, shelfCount: 0, status: "lost", loanDays: 14 },
};

const output = (k) => {
  if (DEFECT === "rename") { const { loanDays, ...rest } = k; return { ...rest, loanDurationDays: loanDays }; }
  if (DEFECT === "missing") { const { title, ...rest } = k; return rest; }
  return { ...k };
};

const server = createServer((request, response) => {
  response.sendDate = false;
  const book = BOOKS[request.url.split("?")[0].replace("/book/", "")];
  response.writeHead(book ? 200 : 404, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(book ? output(book) : { error: "book not found" }));
});

server.listen(Number(process.argv[2]), "127.0.0.1", () =>
  console.log(`ready ${server.address().port} defect=${DEFECT}`));

Instead of remembering the expectation, it is possible to record it. The body the consumer reads is placed behind a watcher; the watcher records every field read, by name and type. The script below runs the two scenarios from the loan service’s own test suite against the real catalog service and generates a contract file from the run.

// collect.mjs — generates a contract from the consumer's run and measures its coverage
import { spawn } from "node:child_process";
import { writeFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
const base = `http://127.0.0.1:${port}`;
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");

// The consumer reads the real response through a watcher; the watcher records the field name it read.
async function run(isbn) {
  const body = await (await fetch(`${base}/book/${isbn}`)).json();
  const read = new Map();
  const watched = new Proxy(body, {
    get: (t, field) => { if (typeof field === "string") read.set(field, typeName(t[field])); return t[field]; },
  });
  return { read, decision: loanDecision(watched, 20260) };
}

const ROUTINE = ["978-0262033848", "978-0201896831"];         // the loan service's own test scenarios
const LOST = "978-0134685991";                                 // lost-book branch: not in the suite
const contract = { consumer: "loan-service", interaction: [] };

for (const isbn of ROUTINE) {
  const { read, decision } = await run(isbn);
  contract.interaction.push({ path: `/book/${isbn}`, fields: Object.fromEntries(read) });
  console.log(`${isbn} -> ${decision.message}\n  fields read ${read.size}: ${[...read.keys()].join(", ")}`);
}
writeFileSync("contract.json", JSON.stringify(contract, null, 1));

const all = new Set(contract.interaction.flatMap((e) => Object.keys(e.fields)));
const covered = new Set(all);
for (const a of (await run(LOST)).read.keys()) all.add(a);
const produced = Object.keys(await (await fetch(`${base}/book/${LOST}`)).json());
proc.kill();

console.log(`fields produced by the provider : ${produced.length}`);
console.log(`fields covered by the contract  : ${covered.size} (${[...covered].join(", ")})`);
console.log(`fields the consumer can read    : ${all.size} (${[...all].join(", ")})`);
console.log(`left outside coverage           : ${[...all].filter((a) => !covered.has(a)).join(", ")}`);
console.log(`contract file ${JSON.stringify(contract).length} bytes, ${contract.interaction.length} interactions`);

978-0262033848 -> 978-0262033848 loaned
  fields read 4: status, shelfCount, loanDays, isbn
978-0201896831 -> not on shelf: 978-0201896831
  fields read 2: status, isbn
fields produced by the provider : 9
fields covered by the contract  : 4 (status, shelfCount, loanDays, isbn)
fields the consumer can read    : 6 (status, shelfCount, loanDays, isbn, title, author)
left outside coverage           : title, author
contract file 241 bytes, 2 interactions

The two scenarios read two different sets of fields. Only two fields being read in the second scenario is not an accident: once the condition status !== "shelved" is true, the second comparison never runs, and the shelf count is never read either. The contract writes down not what the code could read, but what it did read in that run.

The first three numbers are this lesson’s measure. The provider produces nine fields, the contract covers four of them, and the consumer’s branches together read six. The gap between nine and four is the provider’s room to maneuver: five fields that nobody reads. The gap between six and four, on the other hand, is the contract’s blind spot.

The Contract Running on the Provider’s Side

The contract file leaves the consumer’s repository and enters the provider’s team. The provider brings up its own version and actually sends every interaction in the file.

// provider.test.mjs — the consumer contract runs in the provider's team
// Usage: CATALOG_DEFECT=none|rename|missing node --test provider.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const CONTRACT = JSON.parse(readFileSync("contract.json", "utf8"));
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");
let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
  proc.stdout.once("data", (v) => { base = `http://127.0.0.1:${String(v).split(" ")[1]}`; resolve(); });
}));
after(() => proc.kill());

for (const e of CONTRACT.interaction) {
  test(`${CONTRACT.consumer} ${e.path}`, async () => {
    const body = await (await fetch(base + e.path)).json();
    const missing = Object.entries(e.fields)
      .filter(([name, type]) => body[name] === undefined || typeName(body[name]) !== type)
      .map(([name, type]) => `${name} (expected ${type})`);
    assert.equal(missing.length, 0, `unmet field: ${missing.join(", ")}`);
  });
}

for k in none rename missing; do
  echo "== defect=$k =="
  CATALOG_DEFECT=$k node --test --test-reporter=tap provider.test.mjs |
    grep -E '^(ok|not ok|# (pass|fail))|unmet' | sed 's/^ *//'
done

== defect=none ==
ok 1 - loan-service /book/978-0262033848
ok 2 - loan-service /book/978-0201896831
# pass 2
# fail 0
== defect=rename ==
not ok 1 - loan-service /book/978-0262033848
unmet field: loanDays (expected integer)
ok 2 - loan-service /book/978-0201896831
# pass 1
# fail 1
== defect=missing ==
ok 1 - loan-service /book/978-0262033848
ok 2 - loan-service /book/978-0201896831
# pass 2
# fail 0

The defect class caught is the loss of a field written into the contract. When the loan duration field was renamed, the test turned red and reported by name which field went unmet on which consumer’s path. When the field was put back under its old name — the defect=none row — the test turned green. The value of this signal is that the provider sees it before publishing, in its own repository; the side that would break is another team’s service.

The previous lesson’s schema check could not give this information. A schema check says “the response deviated from the definition”; a contract test says “the loan service’s book-reading path broke.”

The Branch Left Outside Coverage

The third column stayed silent. When the title field was removed, both tests still passed, because the title is not in the contract. Yet one of the consumer’s branches reads it.

// deviation.mjs — the consumer's lost branch when the contract stays green despite the defect
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

async function run(defect, isbn) {
  const proc = spawn("node", ["catalog.mjs", "0"], { env: { ...process.env, CATALOG_DEFECT: defect } });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const body = await (await fetch(`http://127.0.0.1:${port}/book/${isbn}`)).json();
  proc.kill();
  return loanDecision(body, 20260);
}

for (const defect of ["none", "missing"]) {
  console.log(`defect=${defect.padEnd(7)} lost branch -> ${(await run(defect, "978-0134685991")).message}`);
}
const s = JSON.parse(readFileSync("contract.json", "utf8"));
const covered = new Set(s.interaction.flatMap((e) => Object.keys(e.fields)));
console.log(`contract ${s.interaction.length} interactions, ${covered.size} fields; title covered: ${covered.has("title")}`);

defect=none    lost branch -> lost record: Lost Volume / Bloch
defect=missing lost branch -> lost record: undefined / Bloch
contract 2 interactions, 4 fields; title covered: false

This is the missed defect class: a field read by a branch that never ran. The provider’s team is green, and the consumer is broken. The generated contract is a snapshot of the consumer’s test suite; whatever branches the suite runs, the contract covers. M21/K02’s coverage measurement takes on a second meaning here — a gap in branch coverage opens up a gap not only in its own test, but also in the provider’s decision.

A missing expectation looks like a dependency that does not exist. The provider believes five fields are read by no one; the actual number of unread fields is three, because the branch that reads the title and author fields never entered the contract at all. The contract’s accuracy is exactly as good as the coverage of the run that produced it.

The Cost of the Contract

Running the same decision function in two separate fixtures shows where the cost comes from.

// cost.mjs — the cost of contract verification relative to the unit test
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";
import { loanDecision } from "./loan.mjs";

const CONTRACT = JSON.parse(readFileSync("contract.json", "utf8"));
const FAKE = [                                       // fake catalog: the contract's in-memory counterpart
  { isbn: "978-0262033848", status: "shelved", shelfCount: 2, loanDays: 14 },
  { isbn: "978-0201896831", status: "loaned", shelfCount: 0, loanDays: 7 },
];
const REPEAT = 50;

const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) for (const k of FAKE) loanDecision(k, 20260);
const t1 = performance.now();

const proc = spawn("node", ["catalog.mjs", "0"], { env: process.env });
const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
for (let i = 0; i < REPEAT; i += 1) {
  for (const e of CONTRACT.interaction) await (await fetch(`http://127.0.0.1:${port}${e.path}`)).json();
}
const t2 = performance.now();
proc.kill();

const fields = CONTRACT.interaction.reduce((n, e) => n + Object.keys(e.fields).length, 0);
console.log(`unit run     : 1 component, 0 requests, ${REPEAT * FAKE.length} decisions`);
console.log(`contract run : 2 components, ${REPEAT * CONTRACT.interaction.length} requests, ${REPEAT * fields} field checks`);
console.log(`contract run longer than unit run : ${t2 - t1 > t1 - t0}`);
console.log(`ratio at least a hundred times     : ${(t2 - t1) / (t1 - t0) >= 100}`);

unit run     : 1 component, 0 requests, 100 decisions
contract run : 2 components, 100 requests, 300 field checks
contract run longer than unit run : true
ratio at least a hundred times     : true

Absolute times are machine-dependent; the direction of the ratio does not change. The point of the comparison is this: the unit test running against the fake catalog and the contract run look at the same four fields, but one does it in-process, and the other with two components and a hundred requests. What the cost buys is a single thing: the assumption that the fake catalog’s four fields match the real service’s is no longer assumed — it is tested.

The numbers independent of the run give the maintenance side. The contract file is 241 bytes and carries two interactions; because it is not hand-written, it gets regenerated as the consumer’s tests change. In return, the provider knows that four fields are read based on a written record; for the remaining five fields, the only thing it knows is that they appear in no contract.

Summary

• In a consumer-driven contract, the consumer writes the expectation and the provider runs it in its own team; the schema says what is produced, and the contract says what is read.
• The expectation does not have to be written by hand: when the consumer’s run is read through a watcher, the fields read are recorded by name and type.
• The measure is three numbers: the provider produces nine fields, the contract covers four of them, and the consumer’s branches together read six.
• The caught class is the loss of a field written into the contract; when the loan duration field was renamed, the test turned red, and it turned green again once the name was restored.
• The missed class is a field read by a branch that never ran; when the title field was removed, the provider stayed green while the consumer’s lost-book branch broke.
• Cost: two components, a hundred requests per run, a generated file of 241 bytes; in return, the assumption that the fake catalog matches reality gets tested.

Next Step

So far, both checks have worked by running: a process comes up, a request goes out, the response gets examined. The provider’s question, though, is asked before the change is published, and it has a form that requires no run. Adding a field, removing a field, narrowing a type, and making an optional field required — which of these four change types breaks the old consumer, and which does not? The next lesson turns this question into a check that compares two schema versions, sorts the change types into compatibility classes, and shows, in that same run, a change the check counts as compatible even though it breaks the consumer.