The Scope of Integration Testing

The Unit Testing and Test-Driven Development course closed by naming an assumption: every test there ran in-process and without a real dependency, and it assumed that its test doubles matched reality without ever checking it anywhere. This course tests that assumption. The first question is where the testing should start.

The answer is not a virtue, it is a decision. The sentence “testing with the real thing is safer” selects nothing; running the real thing for each dependency brings its own cost and makes a separate defect class visible. This lesson makes that decision on one example and measures three things together: the defect class the chosen boundary catches, the defect class it misses, and its cost. That is the measurement axis for the whole course.

The Boundary Decision

The loan service had four dependencies: the catalog holding records, the notification channel that informs the member, the transaction log, and the clock that gives the day number. Each one has two options — the fake dependency or the real one — and when the four are chosen independently, sixteen configurations result. Not all of these are meaningful. The clock’s real version exposes no defect class in the test, it only destroys determinism. The notification channel’s real version requires bringing up a process. The catalog sits between the two: its real version is a database that can run in-process and carries its own constraints.

This lesson runs the catalog boundary for real and leaves the rest fake. This configuration’s name is the integration test: the tested code runs together with at least one dependency’s real implementation.

Two Configurations of the Same Test

The loan service stands exactly as it came from the previous course.

// service.mjs — the loan service from M21/K02: with catalog, notification, and clock dependencies
export const RULES = {
  student: { loanDays: 28, maxLoans: 10 },
  member: { loanDays: 14, maxLoans: 5 },
};

export function createLoanService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      if (catalog.memberLoanCount(member.id) >= rule.maxLoans) {
        throw new Error(`loan limit exceeded: ${rule.maxLoans}`);
      }
      const today = clock();
      const record = { memberId: member.id, bookId, borrowedDay: today, dueDay: today + rule.loanDays };
      catalog.add(record);
      notification.send(member.id, `due date: ${record.dueDay}`);
      return record;
    },
    checkIn(bookId) {
      const record = catalog.find(bookId);
      if (record === undefined) throw new Error(`no loan record: ${bookId}`);
      catalog.remove(bookId);
      return record;
    },
  };
}

Two implementations of the catalog interface are written. The first is the fake object from the previous course. The second meets the same interface on a real schema; the schema is the version used in production and comes from the team that owns the catalog.

// catalogs.mjs — two implementations of the same catalog interface: fake object and node:sqlite
import { DatabaseSync } from 'node:sqlite';

export const SCHEMA = `
CREATE TABLE loan (
  book_no    TEXT PRIMARY KEY,
  member_no  TEXT NOT NULL,
  branch     TEXT NOT NULL,
  borrow_day INTEGER NOT NULL,
  due_day    INTEGER NOT NULL,
  CHECK (due_day > borrow_day)
);`;

export function fakeCatalog() {
  const records = new Map();
  return {
    memberLoanCount: (memberId) => [...records.values()].filter((r) => r.memberId === memberId).length,
    add: (record) => records.set(record.bookId, record),
    find: (bookId) => records.get(bookId),
    remove: (bookId) => records.delete(bookId),
    queryCount: () => 0,
  };
}

export function sqliteCatalog(path = ':memory:') {
  const db = new DatabaseSync(path);
  db.exec(SCHEMA);
  let queries = 0;
  const run = (text, ...bound) => { queries += 1; return db.prepare(text).run(...bound); };
  const one = (text, ...bound) => { queries += 1; return db.prepare(text).get(...bound); };
  return {
    memberLoanCount: (memberId) => one('SELECT COUNT(*) AS n FROM loan WHERE member_no = ?', memberId).n,
    add: (record) => run(
      'INSERT INTO loan (book_no, member_no, branch, borrow_day, due_day) VALUES (?, ?, ?, ?, ?)',
      record.bookId, record.memberId, record.branch ?? null, record.borrowedDay, record.dueDay),
    find: (bookId) => {
      const s = one('SELECT book_no, member_no, borrow_day, due_day FROM loan WHERE book_no = ?', bookId);
      return s && { bookId: s.book_no, memberId: s.member_no, borrowedDay: s.borrow_day, dueDay: s.due_day };
    },
    remove: (bookId) => run('DELETE FROM loan WHERE book_no = ?', bookId),
    queryCount: () => queries,
  };
}

The fixture selects which implementation to build from a single place. The notification is a spy object, the clock is a function; both keep their shape from the previous course.

// fixture.mjs — the same test's two configurations: catalog fake or real
import { createLoanService } from './service.mjs';
import { fakeCatalog, sqliteCatalog } from './catalogs.mjs';

export function createFixture(configuration) {
  const catalog = configuration === 'real' ? sqliteCatalog() : fakeCatalog();
  const sent = [];
  const notification = { send: (memberId, text) => sent.push({ memberId, text }) };
  const service = createLoanService({ catalog, notification, clock: () => 1000 });
  return { service, catalog, sent };
}

export const MEMBER = { id: 'U-17', type: 'member', branch: 'central' };

The tests are a single file and read the configuration from an environment variable. The third test exercises the boundary check directly, so it writes five records into the catalog itself; the side writing those records knows the schema and supplies the branch field.

// loan.test.mjs — the same three tests; the CATALOG variable picks whether the boundary is real
import test from 'node:test';
import assert from 'node:assert/strict';
import { createFixture, MEMBER } from './fixture.mjs';

const configuration = process.env.CATALOG ?? 'fake';

test('a new loan appears in the catalog', () => {
  const { service, catalog } = createFixture(configuration);
  service.lendBook(MEMBER, 'K-903');
  assert.equal(catalog.find('K-903').dueDay, 1014);
});

test('a returned book drops from the catalog', () => {
  const { service, catalog } = createFixture(configuration);
  service.lendBook(MEMBER, 'K-903');
  service.checkIn('K-903');
  assert.equal(catalog.find('K-903'), undefined);
});

test('an error is raised when the loan limit is exceeded', () => {
  const { service, catalog } = createFixture(configuration);
  for (let i = 0; i < 5; i += 1) {
    catalog.add({ bookId: `K-90${i}`, memberId: MEMBER.id, branch: 'central', borrowedDay: 1000, dueDay: 1014 });
  }
  assert.throws(() => service.lendBook(MEMBER, 'K-999'), /loan limit exceeded: 5/);
});

Green at the Fake Boundary

This is the configuration the previous course left behind: the catalog is a fake object.

CATALOG=fake node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - a new loan appears in the catalog
ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 3
# fail 0

All three tests pass. This run does not show that the service is correct, it shows that the service does whatever the fake catalog accepts. A fake catalog is a map, and a map accepts a record of any shape.

Red at the Real Boundary

The one change is running the catalog boundary for real. The test file, the assertions, and the service stay exactly as they are.

CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

not ok 1 - a new loan appears in the catalog
not ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 1
# fail 2

The first two tests fail. The reason is written in the run’s detail.

CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep "error:" | sort -u

  error: 'NOT NULL constraint failed: loan.branch'

The name of the caught defect class is schema mismatch: the record the service produces does not carry a field the real schema requires. This defect was in the code and had been there through the whole previous course; the fake catalog could not see it, because a fake object has no acceptance rule. The third test staying green also carries information: the side that writes records while knowing the schema is defect-free, the defect is only in the path the service writes.

The fix is in the service’s record shape.

// service.mjs — the record now carries the branch field too
export const RULES = {
  student: { loanDays: 28, maxLoans: 10 },
  member: { loanDays: 14, maxLoans: 5 },
};

export function createLoanService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      if (catalog.memberLoanCount(member.id) >= rule.maxLoans) {
        throw new Error(`loan limit exceeded: ${rule.maxLoans}`);
      }
      const today = clock();
      const record = {
        memberId: member.id, bookId, branch: member.branch,
        borrowedDay: today, dueDay: today + rule.loanDays,
      };
      catalog.add(record);
      notification.send(member.id, `due date: ${record.dueDay}`);
      return record;
    },
    checkIn(bookId) {
      const record = catalog.find(bookId);
      if (record === undefined) throw new Error(`no loan record: ${bookId}`);
      catalog.remove(bookId);
      return record;
    },
  };
}

CATALOG=real node --test --test-reporter=tap loan.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - a new loan appears in the catalog
ok 2 - a returned book drops from the catalog
ok 3 - an error is raised when the loan limit is exceeded
# tests 3
# pass 3
# fail 0

What This Test Does Not See

The catalog boundary runs for real, the notification boundary is still a spy object. This leaves one defect class untouched. Take a version that writes the member’s type instead of the member’s id to the notification recipient.

// buggy.mjs — a version that writes the member's type instead of the member's id to the notification recipient
import { RULES } from './service.mjs';

export function createBuggyService({ catalog, notification, clock }) {
  return {
    lendBook(member, bookId) {
      const rule = RULES[member.type];
      const today = clock();
      const record = { memberId: member.id, bookId, branch: member.branch, borrowedDay: today, dueDay: today + rule.loanDays };
      catalog.add(record);
      notification.send(member.type, `due date: ${record.dueDay}`);
      return record;
    },
  };
}

The test verifies the notification too: a notification must have gone out and its text must carry the correct due date.

// missed.test.mjs — catalog is real, notification is a spy; the recipient defect is not tested
import test from 'node:test';
import assert from 'node:assert/strict';
import { sqliteCatalog } from './catalogs.mjs';
import { createBuggyService } from './buggy.mjs';

test('a loan record is written and a notification is sent', () => {
  const catalog = sqliteCatalog();
  const sent = [];
  const notification = { send: (recipient, text) => sent.push({ recipient, text }) };
  const service = createBuggyService({ catalog, notification, clock: () => 1000 });

  service.lendBook({ id: 'U-17', type: 'member', branch: 'central' }, 'K-903');

  assert.equal(catalog.find('K-903').memberId, 'U-17');
  assert.equal(sent.length, 1);
  assert.match(sent[0].text, /due date: 1014/);
  console.log(`recipient sent to: ${sent[0].recipient}`);
});

node --test --test-reporter=tap missed.test.mjs | grep -E '^ *(ok|not ok|# (recipient|tests|pass|fail))'

# recipient sent to: member
ok 1 - a loan record is written and a notification is sent
# tests 1
# pass 1
# fail 0

The test is green, the recipient is wrong. The name of the missed defect class is recipient identity mismatch: the real notification channel would have returned an error for a recipient it did not recognize, while the spy object records whatever value it is given. Catching this class also requires running the notification boundary for real; making the catalog boundary real contributes nothing to it. The rule that follows spreads across the whole course: the defect class a test sees is set by the boundary it runs for real.

Cost

The real boundary’s counterpart can be measured. Two quantities are run-independent — the query count per run and the fixture’s line count — a third, time, is machine-dependent and is reported as a comparison rather than as a raw millisecond figure.

// cost.mjs — the two configurations' query count, fixture line count, and relative time
import { createFixture, MEMBER } from './fixture.mjs';
import { fakeCatalog, sqliteCatalog } from './catalogs.mjs';

const ROUNDS = 500;
const lines = (fn) => fn.toString().split('\n').length - 2;

function measure(configuration) {
  const first = createFixture(configuration);
  first.service.lendBook(MEMBER, 'K-903');
  first.service.checkIn('K-903');
  const start = performance.now();
  for (let i = 0; i < ROUNDS; i += 1) {
    const { service } = createFixture(configuration);
    service.lendBook(MEMBER, 'K-903');
    service.checkIn('K-903');
  }
  return { time: performance.now() - start, queries: first.catalog.queryCount() };
}

const fake = measure('fake');
const real = measure('real');

console.log(`fake catalog: ${ROUNDS} rounds, ${fake.queries} queries per round, catalog ${lines(fakeCatalog)} lines`);
console.log(`real catalog: ${ROUNDS} rounds, ${real.queries} queries per round, catalog ${lines(sqliteCatalog)} lines`);
console.log(`real boundary took longer        : ${real.time > fake.time}`);
console.log(`time difference at least tenfold : ${real.time / fake.time >= 10}`);

fake catalog: 500 rounds, 0 queries per round, catalog 8 lines
real catalog: 500 rounds, 4 queries per round, catalog 17 lines
real boundary took longer        : true
time difference at least tenfold : true

One loan-and-return round runs four queries at the real boundary; the fake boundary runs none. The catalog implementation grows from eight lines to seventeen, and those nine extra lines are now fixture code that needs maintenance: if the schema changes, this place changes too. The absolute value of the time difference depends on the machine and the file system, not its direction.

These three numbers carry a decision. Four queries and a tenfold time cost were paid in exchange for one defect class — schema mismatch. Paying that same cost for the clock boundary buys no defect class; that is why the clock stays fake.

Summary

• Which boundary runs for real is a decision made dependency by dependency; four dependencies produce sixteen configurations and not all of them are meaningful.
• A fake catalog has no acceptance rule; a real schema does. The service that stayed green through the whole previous course failed two tests against the real schema, and the defect’s name is schema mismatch.
• The defect class a test cannot see is set by the boundary it does not run for real: the notification stayed a spy object, so recipient identity mismatch stayed green.
• The boundary’s cost is measured in run-independent quantities: four queries per round and nine lines of extra fixture code.
• Every lesson in this course writes the same triplet: the defect class caught, the defect class missed, and the cost.

Next Step

The catalog boundary ran for real in-process, because the catalog’s real version could live in the same process. The notification channel’s real version cannot: it is a separate process, it listens on a port, and it needs to be up before the test and torn down after. If the catalog itself sits in a file rather than in memory, having the schema already set up in that file is also a setup step. The next lesson takes up these steps as test environment management: how a dependent service is brought up, how its readiness is known, and how many processes and how many steps this fixture’s cost per run comes to.