Developing With Mock Servers

All three previous checks assumed the provider was ready. Yet the moment a contract is most useful is the moment the provider is not yet written: the team developing the loan service does not have to wait for the catalog service to be finished, because the contract is already settled.

A mock server turns the contract into something runnable and removes the waiting time. In exchange, it creates a risk: what stands in for reality is not reality itself. This lesson turns that risk into a number — at how many points does the mock server deviate from reality, and how many of those deviations can the contract run built so far close.

Two Servers, One Contract

The contract is the knowledge produced by the previous lessons: the five fields the loan service reads, along with their types. The sample values the mock server will generate also live in the same file.

// contract.mjs — the book response contract: fields read, their types, and sample values
export const FIELD = { isbn: "string", title: "string", shelfCount: "integer", status: "string", loanDays: "integer" };
export const SAMPLE = { isbn: "978-0262033848", title: "Sample Book", shelfCount: 2, status: "shelved", loanDays: 14 };
export const SAMPLE_V1 = { ...SAMPLE, shelfCount: "2" };   // the mock server's first version: sample value is a string
export const INTERACTION = [{ path: "/book/978-0262033848", fields: FIELD }];

The real catalog service carries two books, decrements the shelf count on a checkout request, and rejects the request if there is no copy on the shelf.

// real.mjs — real catalog service: holds state, applies the business rule
// Usage: node real.mjs <port|0>
import { createServer } from "node:http";
if (process.argv[2] === undefined) { console.log("usage: node real.mjs <port|0>"); process.exit(0); }

const BOOKS = {
  "978-0262033848": { isbn: "978-0262033848", title: "Introduction to Algorithms", shelfCount: 2,
    status: "shelved", loanDays: 14, tags: ["algorithms"] },
  "978-0201896831": { isbn: "978-0201896831", title: "Reference Handbook", shelfCount: 0,
    status: "loaned", loanDays: 7, tags: [] },
};
const readBody = (i) => new Promise((c) => { let v = ""; i.on("data", (p) => (v += p)); i.on("end", () => c(v)); });

const server = createServer(async (request, response) => {
  response.sendDate = false;
  const path = request.url.split("?")[0];
  const json = (code, body) => {
    response.writeHead(code, { "content-type": "application/json; charset=utf-8" });
    response.end(JSON.stringify(body));
  };
  if (request.method === "POST" && path === "/checkout") {
    const k = BOOKS[JSON.parse((await readBody(request)) || "{}").isbn];
    if (k === undefined) return json(404, { error: "book not found" });
    if (k.shelfCount < 1) return json(409, { error: "shelf empty" });
    k.shelfCount -= 1;
    return json(201, { checkoutNo: "A-1", isbn: k.isbn });
  }
  const k = BOOKS[path.replace("/book/", "")];
  return k ? json(200, k) : json(404, { error: "book not found" });
});
server.listen(Number(process.argv[2]), "127.0.0.1", () => console.log(`real ${server.address().port}`));

The mock server is generated from the contract and does everything the contract has written down: it returns the sample body. It does nothing the contract does not write down — it holds no state and applies no rule.

// mock.mjs — mock server generated from the contract: holds no state, applies no business rule
// Usage: MOCK_VERSION=1|2 node mock.mjs <port|0>
import { createServer } from "node:http";
import { SAMPLE, SAMPLE_V1 } from "./contract.mjs";
if (process.argv[2] === undefined) { console.log("usage: node mock.mjs <port|0>"); process.exit(0); }

const BODY = process.env.MOCK_VERSION === "2" ? SAMPLE : SAMPLE_V1;
const server = createServer((request, response) => {
  request.resume();
  response.sendDate = false;
  const checkout = request.url.split("?")[0] === "/checkout";
  response.writeHead(checkout ? 201 : 200, { "content-type": "application/json; charset=utf-8" });
  response.end(JSON.stringify(checkout ? { checkoutNo: "A-1", isbn: BODY.isbn } : BODY));
});
server.listen(Number(process.argv[2]), "127.0.0.1", () => console.log(`mock ${server.address().port}`));

Counting the Deviation Points

Deviation is not an opinion but a number: the same set of requests is sent to both servers, and five metrics are compared.

// deviation.mjs — the same set of requests is sent to both servers, and deviation points are counted
import { spawn } from "node:child_process";
import { FIELD } from "./contract.mjs";

const ISBN = "978-0262033848";
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");

async function drive(file) {
  const proc = spawn("node", [file, "0"], { env: process.env });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const base = `http://127.0.0.1:${port}`;
  const call = async (path, options) => {
    const res = await fetch(base + path, options);
    return { status: res.status, body: await res.json() };
  };
  const book = await call(`/book/${ISBN}`);
  const missing = await call("/book/978-0000000000");
  const empty = await call("/checkout", { method: "POST", body: JSON.stringify({ isbn: "978-0201896831" }) });
  await call("/checkout", { method: "POST", body: JSON.stringify({ isbn: ISBN }) });
  const after = await call(`/book/${ISBN}`);
  proc.kill();
  return {
    "success field count": String(Object.keys(book.body).length),
    "shelfCount type": typeName(book.body.shelfCount),
    "unknown book code": String(missing.status),
    "checkout on empty shelf": String(empty.status),
    "shelfCount after checkout": String(after.body.shelfCount),
  };
}

const mock = await drive("mock.mjs");
const real = await drive("real.mjs");
console.log(`${"metric".padEnd(27)}${"mock".padStart(11)}${"real".padStart(11)}${"deviation".padStart(11)}`);
let deviation = 0;
for (const metric of Object.keys(mock)) {
  const different = mock[metric] !== real[metric];
  deviation += different ? 1 : 0;
  console.log(`${metric.padEnd(27)}${mock[metric].padStart(11)}${real[metric].padStart(11)}` +
    `${(different ? "yes" : "no").padStart(11)}`);
}
console.log(`mock version ${process.env.MOCK_VERSION ?? "1"}: deviation ${deviation}/${Object.keys(mock).length}, ` +
  `fields checked by the contract ${Object.keys(FIELD).length}`);

metric                            mock       real  deviation
success field count                  5          6        yes
shelfCount type                 string    integer        yes
unknown book code                  200        404        yes
checkout on empty shelf            201        409        yes
shelfCount after checkout            2          1        yes
mock version 1: deviation 5/5, fields checked by the contract 5

All five of the five metrics deviate. The deviations are not of the same kind, and that distinction is this lesson’s axis. The first two are shape deviations: field count and type. The third and fourth are decision deviations: the mock server answers every request positively, because it does not know what to return under which condition. The fifth is a state deviation: the real service decrements the shelf count after a checkout, while the mock server sees the two requests as independent of each other.

The Deviation the Contract Run Closes

The mock server has to pass the contract run too. The same test file runs against both the mock and the real server; the server name is supplied from outside.

// contract.test.mjs — the same contract run against both the mock and the real server
// Usage: SERVER=mock.mjs|real.mjs MOCK_VERSION=1|2 node --test contract.test.mjs
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { spawn } from "node:child_process";
import { INTERACTION } from "./contract.mjs";

const FILE = process.env.SERVER ?? "real.mjs";
const typeName = (d) => (Number.isInteger(d) ? "integer" : typeof d === "string" ? "string" : "other");
let proc, base;

before(() => new Promise((resolve) => {
  proc = spawn("node", [FILE, "0"], { env: process.env });
  proc.stdout.once("data", (v) => { base = `http://127.0.0.1:${String(v).split(" ")[1]}`; resolve(); });
}));
after(() => proc.kill());

for (const e of INTERACTION) {
  test(`${FILE} version ${process.env.MOCK_VERSION ?? "-"} ${e.path}`, async () => {
    const body = await (await fetch(base + e.path)).json();
    const missing = Object.entries(e.fields)
      .filter(([name, type]) => body[name] === undefined || typeName(body[name]) !== type)
      .map(([name, type]) => `${name} (expected ${type})`);
    assert.equal(missing.length, 0, `unmet field: ${missing.join(", ")}`);
  });
}

for s in "mock.mjs 1" "mock.mjs 2" "real.mjs -"; do
  set -- $s
  SERVER=$1 MOCK_VERSION=$2 node --test --test-reporter=tap contract.test.mjs |
    grep -E '^(ok|not ok)|unmet' | sed 's/^ *//'
done
MOCK_VERSION=2 node deviation.mjs | tail -1

not ok 1 - mock.mjs version 1 /book/978-0262033848
unmet field: shelfCount (expected integer)
ok 1 - mock.mjs version 2 /book/978-0262033848
ok 1 - real.mjs version - /book/978-0262033848
mock version 2: deviation 4/5, fields checked by the contract 5

The defect class caught is the mock server deviating from the contract. In the mock server’s first version, the sample value was written as a string; the contract run turned red and reported the field by name. After the sample was fixed, the same run turned green, and the deviation count dropped from five to four. The rule that follows is a single sentence: the mock server must be checked with the exact same contract the consumer is checked against; otherwise the consumer ends up developed against the mock server itself, not against the contract.

The missed defect class is the deviations outside the contract’s field of view, and it splits into three. The first is field count: the real service also returns a tags field, and the mock server does not. The contract does not cover it, because the consumer does not read it; if it ever starts reading it, the recording setup from the previous lesson adds it to the contract, and the deviation closes on its own.

The second is decision deviation: the status code for an unknown book, and a checkout request when there is no copy on the shelf. These can be closed by adding new interactions to the contract; the cost is writing one scenario and one rule in the mock server for each. With every deviation it closes, the mock server grows a little more like the real service; at the limit, the mock server turns into a second implementation of the real service, and at that point it loses its reason to exist.

The third cannot be closed. The shelf count decrementing after a checkout is not a property of a single request; it is the relationship between two requests. The unit a contract test looks at is one request and its response; state across requests falls outside this unit. Making the mock server hold state does not close this deviation, it only moves it — now the mock server’s state rule can deviate from reality, and there is no contract to check it against.

The Cost of the Mock Server

// cost.mjs — the cost of a mock server: how many lines of fixture, how many requests, what timing
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";

const lineCount = (d) => readFileSync(d, "utf8").trimEnd().split("\n").length;

async function measure(file) {
  const proc = spawn("node", [file, "0"], { env: { ...process.env, MOCK_VERSION: "2" } });
  const port = await new Promise((resolve) => proc.stdout.once("data", (v) => resolve(String(v).split(" ")[1])));
  const url = `http://127.0.0.1:${port}/book/978-0262033848`;
  for (let i = 0; i < 20; i += 1) await (await fetch(url)).json();      // warm-up
  const t = performance.now();
  for (let i = 0; i < 100; i += 1) await (await fetch(url)).json();
  const duration = performance.now() - t;
  proc.kill();
  return duration;
}

const mock = await measure("mock.mjs");
const real = await measure("real.mjs");
console.log(`fixture: mock.mjs ${lineCount("mock.mjs")} lines, real.mjs ${lineCount("real.mjs")} lines`);
console.log(`deviation run 2 processes and 10 requests, contract run 1 process and 1 interaction`);
console.log(`both servers' durations same order of magnitude : ${mock / real > 0.33 && mock / real < 3}`);
console.log(`mock server run shorter than half the real one  : ${mock < real / 2}`);

fixture: mock.mjs 15 lines, real.mjs 31 lines
deviation run 2 processes and 10 requests, contract run 1 process and 1 interaction
both servers' durations same order of magnitude : true
mock server run shorter than half the real one  : false

The last line corrects a misunderstanding. A mock server is not used for speed: both servers are local processes, and the time for a hundred requests is the same order of magnitude. What a mock server provides is not duration but timing — the ability to develop the consumer before the provider is ready. This same distinction also says that using a mock server in place of the real service on an ongoing basis has no justification.

The cost independent of the run sits in two numbers. As a fixture, the mock server is fifteen lines and the real service is thirty-one; the sixteen lines in between are exactly the work the mock server does not do, and they are the counterpart of the five rows in the deviation table. The second number is the deviation table itself: five metrics, two processes per run, and ten requests. This table is written once and run every time the contract changes; there is no other way to estimate how much the mock server actually deviates.

Summary

• A mock server turns the contract into something runnable and lets the consumer be developed without waiting for the provider; in exchange, it creates the risk of deviating from reality.
• Deviation is a number: when the same set of requests was sent to both servers, all five of the five metrics deviated.
• Deviations are of three kinds — shape (field count, type), decision (status code, rule), and state (effects across requests).
• When the mock server was checked against the same contract as the consumer, a shape deviation was caught; once the sample value was fixed, the run turned green and the deviation dropped from five to four.
• Decision deviations can be closed by adding interactions to the contract; every closure brings the mock server closer to reality, and at the limit turns it into a second implementation.
• State deviation cannot be closed: a contract test’s unit is a single request and response, and the relationship across requests falls outside this unit. Cost: 2 processes, 10 requests, a fifteen-line fixture; the gain is not duration but timing.

Next Step

This topic built four checks, and all four had the same shape: looking at a single boundary. The schema check examined one response, the contract test counted fields between one consumer and one provider, the compatibility check compared two schema texts, and the deviation table measured one mock server against one real service. Each is strong at its own boundary, and even when all of them run together, one question stays unanswered.

The path where a member looks up a book on the shelf, checks it out, and borrows it is not within the scope of any of these checks. That path uses multiple requests, multiple services, and state carried across requests, all together; the fifth metric turning out unclosable was the first sign of this. The next topic takes up tests that run this path as a whole, and its first question is a cost question: what does catching the same bug at three separate levels cost, and how many tests should be kept at which level.