Integration Patterns

The previous topic drew out the enterprise’s capabilities and showed which system each one lives on. That mapping never asked one question: how systems connect to each other. If a capability lives on one system, a second system that needs the same data gets it by some path, at some frequency, and under someone’s responsibility. This lesson takes a single edge, builds it four separate ways, and compares all four with the same measures.

At enterprise scale the unit of measure is the edge. Inside a single system, where a call goes is that system’s owner’s decision; at enterprise scale, two separate owners, two separate budgets, and two separate schedules stand at the two ends of an edge. So what gets counted here is not only latency, but what the edge obligates whom to.

IN1 — the enterprise is fictional and is written as a data structure in node. Systems, their owners, and the edges between them stand as an array; the edge being measured is one row of that array.

IN2 — the refresh period is modeled with a tick rather than a timer. Rationale: the period’s clock-time equivalent (a minute, an hour, a night) is set by an operating decision, not by the pattern itself; the quantity measured is “how many refresh periods until fresh,” and that number is independent of the period’s length.

IN3 — the messaging carrier is modeled as an append log. Queue mechanics, delivery guarantees, and consumer competition were measured in the Caching, Queues and Asynchronous Processing course; what is measured here is not the carrier itself, but what the edge costs the source and its owner.

The Enterprise and the Measured Edge

The regional library network is a fictional enterprise. The catalog system was acquired externally; the loan, billing, and notification services were written in-house; membership belongs to a separate unit, and identity to the municipality. The single entity measured in this lesson is copy status: whether a copy of a book is on the shelf. The catalog writes this data; five systems read it.

// enterprise.mjs — regional library network: a fictional enterprise model; prints itself when run directly
export const SYSTEM = [
  { name: "catalog", owner: "catalog unit", external: 1 },
  { name: "loan", owner: "in-house development", external: 0 },
  { name: "billing", owner: "in-house development", external: 0 },
  { name: "notification", owner: "in-house development", external: 0 },
  { name: "reporting", owner: "management unit", external: 0 },
  { name: "branch-interface", owner: "branch unit", external: 0 },
  { name: "membership", owner: "membership unit", external: 0 },
  { name: "identity", owner: "municipal IT", external: 1 },
];
// edges that read copy status from the catalog; each edge is built with one pattern (model)
export const EDGE = [
  { target: "loan", pattern: "shared-db" },
  { target: "billing", pattern: "shared-db" },
  { target: "reporting", pattern: "file" },
  { target: "branch-interface", pattern: "remote-call" },
  { target: "notification", pattern: "message" },
];
export const ownerOf = (a) => SYSTEM.find((s) => s.name === a).owner;

if (process.argv[1].endsWith("enterprise.mjs")) {
  console.log(`${SYSTEM.length} systems, ${new Set(SYSTEM.map((s) => s.owner)).size} distinct owners, ` +
    `${SYSTEM.filter((s) => s.external).length} systems acquired externally`);
  console.log(`edges out of catalog: ${EDGE.length}, owners touched: ` +
    `${new Set(EDGE.map((b) => ownerOf(b.target))).size}`);
  for (const k of ["shared-db", "file", "remote-call", "message"]) {
    console.log(`  ${k.padEnd(13)}${EDGE.filter((b) => b.pattern === k).length} edge`);
  }
}

8 systems, 6 distinct owners, 2 systems acquired externally
edges out of catalog: 5, owners touched: 3
  shared-db    2 edge
  file         1 edge
  remote-call  1 edge
  message      1 edge

The source system owns the copy table and gives access to it four ways: an endpoint that returns a single record in published format, a snapshot in the same format, a dump that exports a copy of the physical table, and a log that appends one row in published format on every change. Physical column names live inside SCHEMA and are read only in the source’s own code.

// catalog.mjs — the source system that owns the copy table; <port>; long-lived process
import { createServer } from "node:http";
import { appendFileSync, existsSync, readFileSync, writeFileSync } from "node:fs";
const [port] = process.argv.slice(2);
if (!port) { console.log("usage: node catalog.mjs <port>"); process.exit(1); }
const LATENCY_MS = 20;                             // how long the catalog takes to serve a request (ms)
const SCHEMA = { status: "status", branch: "branchCode" };   // physical column names; belongs to the catalog
const write = (s) => writeFileSync("catalog.db", s.map((r) => JSON.stringify(r)).join("\n") + "\n");
const read = () => readFileSync("catalog.db", "utf8").split("\n").filter(Boolean).map(JSON.parse);
const published = (r) => ({ copyId: r.copyId, branch: r[SCHEMA.branch],
  available: r[SCHEMA.status] === "shelved" });    // outward format; the catalog's word
if (!existsSync("catalog.db")) {
  write([...Array(200).keys()].map((i) => ({ copyId: 401 + i, branchCode: 1 + (i % 4),
    status: i % 5 === 0 ? "checked-out" : "shelved", printYear: 1990 + (i % 30) })));
}
let load = 0;                                      // number of requests the source has seen

createServer(async (request, response) => {
  const u = new URL(request.url, "http://y");
  if (u.pathname === "/load") { response.end(String(load)); return; }
  load += 1;
  await new Promise((c) => setTimeout(c, LATENCY_MS));
  const id = Number(u.searchParams.get("id"));
  if (u.pathname === "/record") {                  // remote call: single record, published format
    response.end(JSON.stringify(published(read().find((r) => r.copyId === id))));
  } else if (u.pathname === "/snapshot") {         // message consumer's first fill, published format
    response.end(JSON.stringify(read().map(published)));
  } else if (u.pathname === "/dump") {             // file transfer: a copy of the physical table
    const s = read();
    writeFileSync("catalog.dump", s.map((r) => JSON.stringify(r)).join("\n") + "\n");
    response.end(String(s.length));
  } else if (u.pathname === "/change") {           // the catalog's own write path
    const s = read(), r = s.find((x) => x.copyId === id);
    r[SCHEMA.status] = u.searchParams.get("status");
    write(s);
    appendFileSync("catalog.events", JSON.stringify(published(r)) + "\n");
    response.end("done");
  } else if (u.pathname === "/migrate") {          // the catalog renames its own column
    write(read().map((r) => { const { [SCHEMA.status]: d, ...k } = r; return { ...k, copyStatus: d }; }));
    SCHEMA.status = "copyStatus";
    response.end("done");
  } else response.end("none");
}).listen(Number(port));

The Same Edge, Four Ways

The consumer runs from a single source file and takes its pattern from the command line. Each of the four sections defines two functions: QUERY, which answers the query, and TICK, which represents the refresh period. Because the sections are marked, which pattern reads which name can be counted straight from the code.

// loan.mjs — consumer; <pattern: shared-db|file|remote-call|message> <port> <catalog-port>
import { createServer } from "node:http";
import { existsSync, readFileSync } from "node:fs";
const [pattern, port, cp] = process.argv.slice(2);
if (!port) { console.log("usage: node loan.mjs <pattern> <port> <catalog-port>"); process.exit(1); }
const C = `http://127.0.0.1:${cp}`;
const lines = (d) => readFileSync(d, "utf8").split("\n").filter(Boolean).map(JSON.parse);
const local = new Map();                           // the consumer's own copy
let read = 0;
const QUERY = {}, TICK = {};                       // TICK: the trigger modeling the refresh period

// --- shared-db --- the consumer reads the source's physical table directly
QUERY["shared-db"] = (id) => lines("catalog.db").find((r) => r.copyId === id).status === "shelved";
TICK["shared-db"] = () => 0;

// --- file --- the source dumps the physical table, the consumer reads the dump file from scratch
QUERY["file"] = (id) => local.get(id);
TICK["file"] = async () => {
  await fetch(`${C}/dump`);
  const s = lines("catalog.dump");
  for (const r of s) local.set(r.copyId, r.status === "shelved");
  return s.length;
};

// --- remote-call --- every query calls the source's published format
QUERY["remote-call"] = async (id) => (await (await fetch(`${C}/record?id=${id}`)).json()).available;
TICK["remote-call"] = () => 0;

// --- message --- the first fill is the published format, after that only change lines are read
QUERY["message"] = (id) => local.get(id);
TICK["message"] = () => {
  if (!existsSync("catalog.events")) return 0;
  const s = lines("catalog.events");
  let n = 0;
  while (read < s.length) { local.set(s[read].copyId, s[read].available); read += 1; n += 1; }
  return n;
};
if (pattern === "message") for (const r of await (await fetch(`${C}/snapshot`)).json()) local.set(r.copyId, r.available);

// --- common --- service surface independent of pattern
createServer(async (request, response) => {
  const u = new URL(request.url, "http://y");
  if (u.pathname === "/tick") { response.end(String(await TICK[pattern]())); return; }
  response.end(String(await QUERY[pattern](Number(u.searchParams.get("id")))));
}).listen(Number(port));

Measurement

The tool runs in three stages. First it sends a hundred queries to each consumer, counting the round trip per query and the number of requests that reach the source system in the process. Then it changes the status of a single copy at the source and asks the consumer twice: before the tick and after the tick. In the final stage the source renames its own column, and it is checked whether all four consumers still answer the same question correctly. The name count in consumer code is read from the source text, not from the running processes.

// measure.mjs — measures the same edge with four patterns; <catalog-port> <four consumer ports>
import { readFileSync } from "node:fs";
import { EDGE, ownerOf } from "./enterprise.mjs";
const args = process.argv.slice(2);
if (args.length < 5) { console.log("usage: node measure.mjs <catalog-port> <p1> <p2> <p3> <p4>"); process.exit(1); }
const [cp, ...port] = args;
const PATTERN = ["shared-db", "file", "remote-call", "message"];
const CONSUMER = Object.fromEntries(PATTERN.map((k, i) => [k, `http://127.0.0.1:${port[i]}`]));
const C = `http://127.0.0.1:${cp}`, LATENCY_MS = 20, QUERIES = 100;
const text = (u) => fetch(u).then((y) => y.text());
const query = (k, id) => text(`${CONSUMER[k]}/query?id=${id}`);
const tick = (k) => text(`${CONSUMER[k]}/tick`).then(Number);
const o = {};
for (const k of PATTERN) { o[k] = {}; await tick(k); }        // first fill, before measurement

for (const k of PATTERN) {                                    // latency and load seen by the source
  const l0 = Number(await text(`${C}/load`));
  const t = performance.now();
  for (let i = 0; i < QUERIES; i += 1) await query(k, 401 + (i % 200));
  o[k].roundTrip = Math.round((performance.now() - t) / QUERIES / LATENCY_MS);
  o[k].load = Number(await text(`${C}/load`)) - l0;
}
await text(`${C}/change?id=402&status=checked-out`);          // a single record changes at the source
for (const k of PATTERN) {
  const before = await query(k, 402);
  o[k].records = await tick(k);
  o[k].window = before === "false" ? 0 : (await query(k, 402)) === "false" ? 1 : -1;
}
const section = {};                                           // consumer code per pattern
for (const p of readFileSync("loan.mjs", "utf8").split(/^\/\/ --- /m).slice(1)) {
  section[p.slice(0, p.indexOf(" "))] = p;
}
const PHYSICAL = ["catalog.db", "catalog.dump", "status", "shelved"];   // the source's internal schema
const PUBLISHED = ["/record", "/snapshot", "catalog.events", "available"];  // the source's published contract
const count = (m, l) => l.filter((a) => m.includes(a)).length;
await text(`${C}/migrate`);                                   // the source renames its own column
for (const k of PATTERN) { await tick(k); o[k].correct = (await query(k, 403)) === "true"; }

console.log(`${"pattern".padEnd(13)}${"round trip".padStart(12)}${"source requests".padStart(17)}` +
  `${"freshness tick".padStart(16)}${"tick records".padStart(14)}${"physical names".padStart(16)}` +
  `${"published names".padStart(18)}${"after migrate".padStart(15)}`);
for (const k of PATTERN) {
  console.log(`${k.padEnd(13)}${String(o[k].roundTrip).padStart(12)}${String(o[k].load).padStart(17)}` +
    `${String(o[k].window).padStart(16)}${String(o[k].records).padStart(14)}` +
    `${String(count(section[k], PHYSICAL)).padStart(16)}${String(count(section[k], PUBLISHED)).padStart(18)}` +
    `${(o[k].correct ? "correct" : "wrong").padStart(15)}`);
}
const broken = EDGE.filter((b) => !o[b.pattern].correct);
console.log(`single column name changed: ${broken.length}/${EDGE.length} edges broke ` +
  `(${broken.map((b) => b.target).join(", ")}); owners to coordinate with ` +
  `${new Set(broken.map((b) => ownerOf(b.target))).size}/${new Set(EDGE.map((b) => ownerOf(b.target))).size}`);

rm -f catalog.db catalog.dump catalog.events
node catalog.mjs 8981 & K=$!
sleep 0.6
node loan.mjs shared-db   8982 8981 & A=$!
node loan.mjs file        8983 8981 & B=$!
node loan.mjs remote-call 8984 8981 & C=$!
node loan.mjs message     8985 8981 & D=$!
sleep 0.8
node measure.mjs 8981 8982 8983 8984 8985
kill $K $A $B $C $D

pattern        round trip  source requests  freshness tick  tick records  physical names   published names  after migrate
shared-db               0                0               0             0               3                 0          wrong
file                    0                0               1           200               3                 0          wrong
remote-call             1              100               0             0               0                 2        correct
message                 0                0               1             1               0                 3        correct
single column name changed: 3/5 edges broke (loan, billing, reporting); owners to coordinate with 2/3

Latency, Load, and Freshness

The first three columns show what the patterns look like on the surface. Remote call goes to the source on every query: one round trip per query, and a hundred requests for a hundred queries. In the other three patterns the query round trip is zero, because the answer comes from the consumer’s own side; the source system never sees any of the hundred queries.

The freshness column shows what this gain costs. In shared database and remote call, a change made at the source shows up in the consumer at the same instant; both patterns have a freshness window of zero ticks. In file transfer and messaging, a change is visible only after one refresh period. The difference is in the fourth column: even though only one copy’s status changed, file transfer moves all two hundred records on every refresh, while messaging moves one record. The same freshness window is bought with a two-hundred-fold difference in transfer volume. This is also where the work that falls on the source system comes from: file transfer produces zero requests per query but reads the table from start to end on every period.

No single ranking follows from this alone. Remote call zeroes the freshness window and ties the source’s load to the number of queries; file transfer ties load to the number of periods and sacrifices freshness; messaging stays tied to the number of periods but cuts transfer volume down to the number of changes. Shared database looks cheapest of all: zero round trips, zero requests, zero window. Its cost is in the next two columns.

Ownership Leak

The fifth and sixth columns measure the same edge from the ownership side. In the shared database and file transfer patterns, consumer code carries three names from the source’s physical schema: the table’s name, the column’s name, and the value that column is meaningful for. The third name is the most expensive, because it is not a name — it is a rule. The fact that the value “shelved” means available is the catalog unit’s business rule, and that rule sits in the consumer’s code as a second copy. If the catalog adds a third status value tomorrow, the consumer’s answer goes wrong even though the source code never changed.

In remote call and messaging, the physical name count is zero. The consumer does not read the source’s table, it reads the format the source hands outward; the availability decision is computed in exactly one place in the source’s code. The names in the sixth column are not a loss — it is a contract: the source system has committed to not changing those names, and in exchange it can do whatever it wants to its table.

Shared database’s real problem shows up in these rows. The table carries a single system’s name, but two systems’ code depends on it. The schema has no single owner; the catalog unit only learns it cannot change the table on its own once the side that built the edge notices the change.

The Spread of Change

The last column measures this. The source system renames a single column; it does nothing beyond keeping its own published format intact. The remote call and messaging edges are entirely unaffected by this change, because both read from the published format. The shared database and file transfer edges start answering incorrectly — silently, too, because the missing field’s counterpart is not an error, it is an undefined value.

The last line carries this result out to the whole enterprise. Three of the five edges out of the catalog break, and behind the broken ones stand two separate owners. For the source system’s owner, this means a one-line migration first requires reconciling schedules with two units, then waiting for three systems to adapt. Pattern choice stops being a latency preference here and turns into a coordination debt: whichever pattern is chosen, the data gets across, but how many owners’ permission is needed before it can change depends on the pattern.

Summary

• The same edge was built with four patterns: remote call measured 1 round trip per query and 100 source requests for a hundred queries; the other three patterns measured 0 round trips and 0 requests.
• The freshness window is 0 ticks for shared database and remote call, and 1 tick for file transfer and messaging; the same window is bought by moving 200 records in file transfer and 1 record in messaging.
• Shared database and file transfer carry 3 names from the source’s physical schema into consumer code; one of them is not a column name, but a second copy of the availability rule.
• Remote call and messaging carry no physical names; instead they bind to 2 and 3 published names, respectively, that the source has committed to.
• When the source renames a single column, 3 of the 5 edges silently start answering incorrectly, and coordination is needed with 2 of the 3 owners.

Next Step

The measurement was done with a single source and a single consumer; in the enterprise model there are five edges out of the catalog, and the catalog is not the enterprise’s only system. It was never asked how the edge count grows with the number of systems when every system connects to every other system with its own chosen pattern, how much work adding a new system creates for how many parties, and what routing these edges through one common point makes cheaper and what it collects in a single place. The next lesson builds point-to-point edges and a central point side by side in the same enterprise model, and counts the edge count, the cost of a new system, how many edges break the instant the hub goes down, and who owns the business logic that accumulates at the hub.