Schema Compatibility Checking

The previous two checks both worked by running: a process comes up, a request goes out, the response gets examined. The provider’s real question, though, is asked before the change is published, and it requires no run. Adding a field, removing a field, narrowing a type, making an optional field required: which of these four change types breaks the old consumer?

The definition of a breaking change was established in the Web API Design course, and the measurement of its cost in the Application Layer and Service Interaction course; neither is repeated here. This lesson’s subject is turning that knowledge into an automatic check: two schema versions are compared, every change is placed into a type and a compatibility class, and if there is a change in the breaking class, the suite turns red.

Schema Versions and Proposals

The published schema describes the book response and the checkout request together. There are three proposals: A carries all four change types at once, B meets the same need with the first step of the expand–contract sequence, and C only widens a value set.

// schema.mjs — the published schema and three proposals; each field carries a type, requiredness, and value set
const A = (type, required, set) => (set ? { type, required, set } : { type, required });

export const PUBLISHED = {
  response: {
    isbn: A("string", true), title: A("string", true), shelfCount: A("number", true),
    status: A("string", true, ["shelved", "loaned", "lost"]),
    loanDays: A("number", true), tags: A("array", false),
  },
  request: { memberNo: A("string", true), branch: A("string", false), count: A("number", false) },
};

// A: four change types at once. B: the expand step. C: only the set widens.
export const PROPOSAL = {
  A: {
    response: { ...PUBLISHED.response, dueDay: A("string", false) },
    request: { ...PUBLISHED.request, branch: A("string", true), count: A("integer", false) },
  },
  B: { response: { ...PUBLISHED.response, dueDay: A("string", false) }, request: { ...PUBLISHED.request } },
  C: {
    response: { ...PUBLISHED.response, status: A("string", true, ["shelved", "loaned", "lost", "newly-shelved"]) },
    request: { ...PUBLISHED.request },
  },
};
delete PROPOSAL.A.response.tags;                      // field removal

The check itself has two parts: a comparison that sorts the diff into types, and a table that binds types to classes. The table having two columns is this lesson’s central distinction — the same change type falls into a different class in the request direction than in the response direction.

// compatibility.mjs — sorts the changes between two schema versions into types and classes
const NARROW = { number: "integer" };

export const CLASS = {
  "optional field addition": { response: "compatible", request: "compatible" },
  "required field addition": { response: "compatible", request: "breaking" },
  "field removal": { response: "breaking", request: "compatible" },
  "type narrowing": { response: "compatible", request: "breaking" },
  "type widening": { response: "breaking", request: "compatible" },
  "making required": { response: "compatible", request: "breaking" },
  "making optional": { response: "breaking", request: "compatible" },
  "adding a value to the set": { response: "compatible", request: "breaking" },
  "removing a value from the set": { response: "breaking", request: "compatible" },
};

export function diff(old, next) {
  const d = [];
  for (const [name, rule] of Object.entries(old)) {
    const n = next[name];
    if (n === undefined) { d.push({ name, type: "field removal" }); continue; }
    if (n.type !== rule.type) d.push({ name, type: NARROW[rule.type] === n.type ? "type narrowing" : "type widening" });
    if (n.required && !rule.required) d.push({ name, type: "making required" });
    if (rule.required && !n.required) d.push({ name, type: "making optional" });
    if ((n.set ?? []).some((v) => !(rule.set ?? []).includes(v))) d.push({ name, type: "adding a value to the set" });
    if ((rule.set ?? []).some((v) => !(n.set ?? []).includes(v))) d.push({ name, type: "removing a value from the set" });
  }
  for (const [name, n] of Object.entries(next)) {
    if (name in old) continue;
    d.push({ name, type: n.required ? "required field addition" : "optional field addition" });
  }
  return d;
}

// Evaluates a proposal in both directions; each line of the returned list is one classified change.
export const evaluate = (published, proposal) => ["response", "request"].flatMap((direction) =>
  diff(published[direction], proposal[direction]).map((x) => ({ direction, ...x, class: CLASS[x.type][direction] })));

The Distribution of Change Types

// distribution.mjs — how change types distribute over compatibility classes, and a count of the three proposals
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { CLASS, evaluate } from "./compatibility.mjs";

console.log(`${"change type".padEnd(32)}${"response".padStart(12)}${"request".padStart(12)}`);
for (const [type, s] of Object.entries(CLASS)) {
  console.log(`${type.padEnd(32)}${s.response.padStart(12)}${s.request.padStart(12)}`);
}

for (const [name, o] of Object.entries(PROPOSAL)) {
  const d = evaluate(PUBLISHED, o);
  console.log(`\nproposal ${name}: ${d.length} changes, ${d.filter((x) => x.class === "breaking").length} breaking`);
  for (const x of d) console.log(`  ${`${x.direction}.${x.name}`.padEnd(20)}${x.type.padEnd(30)}-> ${x.class}`);
}

change type                         response     request
optional field addition           compatible  compatible
required field addition           compatible    breaking
field removal                       breaking  compatible
type narrowing                    compatible    breaking
type widening                       breaking  compatible
making required                   compatible    breaking
making optional                     breaking  compatible
adding a value to the set         compatible    breaking
removing a value from the set       breaking  compatible

proposal A: 4 changes, 3 breaking
  response.tags       field removal                 -> breaking
  response.dueDay     optional field addition       -> compatible
  request.branch      making required               -> breaking
  request.count       type narrowing                -> breaking

proposal B: 1 changes, 0 breaking
  response.dueDay     optional field addition       -> compatible

proposal C: 1 changes, 0 breaking
  response.status     adding a value to the set     -> compatible

The table’s two columns mirror each other. Removing a field in the response is breaking, because the consumer is reading it; removing a field in the request is not breaking, because the old client keeps sending it and the field is ignored. In the same way, making a field required does no harm in the response — the field was always going to arrive anyway — but in the request it causes every request the old client sends to be rejected. Narrowing a type is compatible in the response, because an integer is still a number; it is breaking in the request, because the old client might send a decimal.

This symmetry shows that the eight-row table is not a list to be memorized. There is a single question, and direction decides it: is the old code on the other side still valid after the change? In the response, the other side reads; in the request, it writes.

The Acceptance Gate

The check works as the acceptance gate defined in the Quality and Testing Fundamentals course: if there is a change in the breaking class, the release stops.

// compatibility.test.mjs — pre-publish gate: if there is a breaking change, the suite turns red
// Usage: PROPOSAL=A|B|C node --test compatibility.test.mjs
import { test } from "node:test";
import assert from "node:assert/strict";
import { PUBLISHED, PROPOSAL as PROPOSALS } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

const NAME = process.env.PROPOSAL ?? "B";

test(`proposal ${NAME}: no breaking change against the published schema`, () => {
  const breaking = evaluate(PUBLISHED, PROPOSALS[NAME]).filter((x) => x.class === "breaking");
  assert.equal(breaking.length, 0,
    `breaking change: ${breaking.map((x) => `${x.direction}.${x.name} ${x.type}`).join(", ")}`);
});

for o in A B C; do
  PROPOSAL=$o node --test --test-reporter=tap compatibility.test.mjs |
    grep -E '^(ok|not ok)|breaking change:' | sed 's/^ *//'
done

not ok 1 - proposal A: no breaking change against the published schema
breaking change: response.tags field removal, request.branch making required, request.count type narrowing
ok 1 - proposal B: no breaking change against the published schema
ok 1 - proposal C: no breaking change against the published schema

The defect class caught is the contract breaking before publication. Proposal A turned red and named the three breaking changes along with their direction. The fix is to turn the same need into the first step of the expand–contract sequence: the new field is added alongside the old ones, and removal and making-required are left to the third step. Proposal B is exactly this, and the gate turns green.

What sets this check apart from the previous two lessons is that it sees the break without running a single consumer. The contract test had to run the consumer’s expectation and the provider’s version together to show a break; the check here compares two texts. In exchange, what it knows is less: it cannot say which consumer breaks, only that one might.

The Break the Check Counts as Compatible

Proposal C passed through the gate. A new value was added to the defined value set for book status: books that have newly entered the collection and are on the shelf are now reported as newly-shelved. All of the old responses are still valid, no field was lost, and no type changed.

// missed.mjs — a change the check counts as compatible breaks the consumer
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

// The loan service's decision branch: an unknown status value counts as "not on shelf".
const loanDecision = (book) => (book.status === "shelved" && book.shelfCount > 0
  ? { granted: true, message: `${book.isbn} loaned` }
  : { granted: false, message: `not on shelf (status ${book.status}, shelfCount ${book.shelfCount})` });

const d = evaluate(PUBLISHED, PROPOSAL.C);
console.log(`check: proposal C -> ${d.length} changes, ${d.filter((x) => x.class === "breaking").length} breaking, gate green`);

const ON_SHELF = { isbn: "978-0262033848", status: "shelved", shelfCount: 2 };
const NEWLY_SHELVED = { ...ON_SHELF, status: "newly-shelved" };  // freshly arrived shelf: book is shelved and loanable
for (const [label, book] of [["old value", ON_SHELF], ["new value", NEWLY_SHELVED]]) {
  const decision = loanDecision(book);
  console.log(`${label.padEnd(11)} -> granted=${String(decision.granted).padEnd(5)} ${decision.message}`);
}
console.log(`schema status values ${PROPOSAL.C.response.status.set.length}, values the consumer treats as loanable 1`);
console.log(`a unit change does not change the schema -> ${evaluate(PUBLISHED, PUBLISHED).length} changes`);

check: proposal C -> 1 changes, 0 breaking, gate green
old value   -> granted=true  978-0262033848 loaned
new value   -> granted=false not on shelf (status newly-shelved, shelfCount 2)
schema status values 4, values the consumer treats as loanable 1
a unit change does not change the schema -> 0 changes

The missed defect class is a change that alters meaning without breaking shape. A book sitting on the shelf became impossible to loan; from the schema’s point of view, no rule was violated. Adding a value to the set counts as compatible in the response direction, because it is, as far as schema validation goes — the old values keep their validity. What breaks is not the schema but the consumer’s code, which assumes it has exhausted that set. This assumption is written in no schema.

The last line shows the boundary even more plainly. The loan duration starting to be counted in weeks instead of days changes neither the field name nor its type; the two schemas stay byte-for-byte identical and the check reports zero changes. No check that only looks at the schema can see a change in unit, scale, or meaning.

The Cost of the Check

// cost.mjs — the cost of a check that only looks at the schema: zero processes, zero requests
import { createServer } from "node:http";
import { PUBLISHED, PROPOSAL } from "./schema.mjs";
import { evaluate } from "./compatibility.mjs";

const REPEAT = 200, proposalCount = Object.keys(PROPOSAL).length;
const t0 = performance.now();
for (let i = 0; i < REPEAT; i += 1) for (const o of Object.values(PROPOSAL)) evaluate(PUBLISHED, o);
const t1 = performance.now();

// Comparison point: a single local request's round trip on the same machine.
const server = createServer((_, res) => res.end("{}"));
await new Promise((c) => server.listen(0, "127.0.0.1", c));
const t2 = performance.now();
await (await fetch(`http://127.0.0.1:${server.address().port}/`)).text();
const t3 = performance.now();
server.close();

const fields = Object.keys(PUBLISHED.response).length + Object.keys(PUBLISHED.request).length;
console.log(`schema ${fields} fields (response ${Object.keys(PUBLISHED.response).length}, request ${Object.keys(PUBLISHED.request).length}), ${proposalCount} proposals`);
console.log(`check ${REPEAT * proposalCount} evaluations, 0 processes, 0 requests, 0 test data`);
console.log(`one evaluation shorter than a single local request : ${(t1 - t0) / (REPEAT * proposalCount) < t3 - t2}`);
console.log(`ratio at least fifty times                         : ${(t3 - t2) / ((t1 - t0) / (REPEAT * proposalCount)) >= 50}`);

schema 9 fields (response 6, request 3), 3 proposals
check 600 evaluations, 0 processes, 0 requests, 0 test data
one evaluation shorter than a single local request : true
ratio at least fifty times                         : true

Absolute times are machine-dependent; the direction of the ratio does not change. This check brings up no process, sends no request, and prepares no test data: its input is two texts, and its output is a classified list. Hundreds of evaluations fit inside the time of a single local request; that is why the check can run on every change proposal.

The cost independent of the run sits in two places. First, the published version of the schema has to be stored somewhere: the comparison needs two texts, and the compatibility question cannot be asked with only one. Second is the upkeep of the class table — nine rows are hand-written, and when a new constraint is added to the schema language, a row has to be added to the table too. A missing row stays silent, and a check that stays silent is indistinguishable from a check that does not exist.

Summary

• A schema compatibility check compares two schema versions, sorts the diff into change types, and places each type into a compatibility class.
• The same type falls into a different class in the request direction than in the response direction; the single question is whether the old code on the other side is still valid after the change.
• The proposal carrying four change types at once turned red with three breaking changes; once the same need was turned into the first step of the expand–contract sequence, the gate turned green.
• The missed class is a change that alters meaning without breaking shape: a value added to a value set was counted as compatible, yet a book sitting on the shelf became impossible to loan. With a unit change, the two schemas stay identical and the check reports zero changes.
• Cost: zero processes, zero requests, zero test data; in return, the published schema has to be stored and the nine-row class table maintained by hand.
• The check cannot say which consumer breaks; it can only say that one might.

Next Step

All three checks assumed the provider was ready: the schema check sent a request to the live service, the contract test brought up the provider’s version, and the compatibility check read the published schema. Yet the moment a contract is most useful is the moment the provider is not yet written. The consumer team does not have to wait, because the contract is already settled. The next lesson builds a mock server that generates responses from the contract, develops the loan service against it, and measures a single question: at how many points does the mock server deviate from the real thing, and how many of those deviations can the checks built so far close.