--- title: 'Enterprise Context and Integration' source: 'https://academia.sh/en/courses/enterprise-context' language: en updated: '2026-08-23T07:01:05+00:00' license: 'CC BY-SA 4.0' lessons: 10 outcomes: - 'Explains the purpose and the limits of enterprise architecture frameworks' - 'Chooses integration patterns between systems inside an enterprise' - 'Makes the build-or-buy decision against criteria' --- # Enterprise Context and Integration The scale at which a system does not stand alone: enterprise architecture as business–IT alignment, the shared structure and the limits of frameworks, business process modeling and automation boundaries, traceability from business capability to system, the file transfer, shared database, remote call, and messaging integration patterns, the pros and cons of centralized integration, master data management with a system of record and a golden record, transformation pipelines that separate the analytical load, replacing legacy systems in stages by wrapping them, and the build-or-buy decision resting on total cost of ownership and dependency risk. ## Enterprise Architecture The level above a single system: what business–IT alignment means, the shared structure of enterprise architecture frameworks and where it falls short, modeling business processes and drawing the automation boundary, and building traceability from a business capability to the system that covers it. ### The Enterprise Architecture Concept Source: https://academia.sh/en/courses/enterprise-context/enterprise-architecture-concept The level that steps outside a single system: turning the enterprise's systems, owners, budgets, and business capabilities into a data structure, and tying business–IT alignment to three numbers — a capability no system covers, a capability more than one system covers, and a system that cannot be tied to any capability. The previous course turned a decision into a record and a system into a view, and measured every document by the number of questions it answered. All those records and views shared one boundary: they stood inside a single system. But a system does not stand on its own. Beside it are other systems that serve the same member on the same day; some are bought from outside, some are paid for out of another unit's budget line, and for some, no employee of that enterprise can write a line of code. This lesson moves the scale up by one step. The Architecture Levels lesson in The Architect's Role course defined the enterprise level by scope: the number of deployment units a decision binds. That lesson's count of uninformed units is not repeated here; it is taken as an input. What changes is the unit itself: it is no longer a separately released deployment unit, but **a system with its own owner and its own budget**. Enterprise architecture is the name for this scale, and its first question is this: how well do the jobs the enterprise must do and the systems it has actually match. #### The Enterprise Is a Data Structure **Business–IT alignment** is, in words, an unmeasurable claim. For it to become measurable, two lists must be placed side by side: the **business capabilities** the enterprise must cover, and the systems that claim to cover them. The capability list is not derived from the systems; it comes from the enterprise's own business (**EA1**). This distinction carries the whole measurement: if the list were derived from the systems, an uncovered capability would never appear. The model used is a regional library network. It is fictional; no real institution, vendor, product, or person is described. The network has branch systems, a catalog system bought from outside, in-house loan and billing services, a separate membership system, an identity verification service that sits with the municipality, and a management unit that wants reporting. Owners and budgets are separate. A system is software paid for out of a separate budget, with a separate owner, and released separately (**EA2**). The coverage list in a system's record is a **claim**; whether the claim holds up in code is not tested in this lesson (**EA3**). An **edge** (integration edge) is the directed pair running from the system that writes to the system that reads across the same data asset (**EA4**) — this is the course's unit of measure. ```js // enterprise/model.mjs — MODEL regional library network: systems, owners, capabilities, data. // Fictional; no real institution, vendor, product, or person is described. export const SYSTEM = { catalog: { owner: "external-provider", budget: "service-fee" }, loan: { owner: "it-department", budget: "internal-development" }, billing: { owner: "it-department", budget: "internal-development" }, membership: { owner: "member-services", budget: "member-services" }, identity: { owner: "municipal-it", budget: "municipality" }, "branch-local": { owner: "branch-management", budget: "branch" }, kiosk: { owner: "branch-management", budget: "branch" }, reporting: { owner: "management-unit", budget: "management" }, archive: { owner: "none", budget: "none" }, }; // CAPABILITY — the business capabilities the enterprise must cover (a list independent of system) export const CAPABILITY = ["material-search", "loan-issuance", "return-intake", "reservation", "membership-enrollment", "member-verification", "fee-calculation", "fee-collection", "inter-branch-transfer", "asset-count", "usage-reporting", "purchase-suggestion", "overdue-notification"]; // COVERS[s] = the capabilities system s claims to cover export const COVERS = { catalog: ["material-search", "asset-count"], loan: ["loan-issuance", "return-intake", "reservation", "overdue-notification"], billing: ["fee-calculation", "overdue-notification"], membership: ["membership-enrollment", "member-verification"], identity: ["member-verification"], "branch-local": ["return-intake", "inter-branch-transfer", "asset-count"], kiosk: ["loan-issuance", "material-search"], reporting: ["usage-reporting"], archive: [], }; // DATA[v] = data asset; which system writes it, which ones read it export const DATA = { "member-record": { writes: ["membership"], reads: ["loan", "billing", "kiosk", "reporting"] }, "identity-match": { writes: ["identity", "membership"], reads: ["loan", "kiosk"] }, "material-record": { writes: ["catalog"], reads: ["loan", "kiosk", "branch-local", "reporting"] }, "copy-status": { writes: ["catalog", "loan", "branch-local"], reads: ["kiosk", "reporting"] }, "loan-transaction": { writes: ["loan", "kiosk"], reads: ["billing", "reporting"] }, "fee-record": { writes: ["billing"], reads: ["membership", "reporting", "kiosk"] }, "penalty-rule": { writes: ["billing"], reads: ["loan", "kiosk"] }, "transfer-request": { writes: ["branch-local"], reads: ["loan", "catalog"] }, "legacy-record": { writes: [], reads: ["archive", "reporting"] }, "count-discrepancy": { writes: ["branch-local"], reads: [] }, }; export const SYSTEMS = Object.keys(SYSTEM); export const owner = (s) => SYSTEM[s].owner; // edge = the directed pair from the system that writes a data asset to the system that reads it export function edges() { const e = new Map(); for (const [v, d] of Object.entries(DATA)) for (const w of d.writes) for (const r of d.reads) if (w !== r) e.set(`${w}->${r}`, [...(e.get(`${w}->${r}`) ?? []), v]); return e; } ``` #### Measurement The second file reduces alignment to three numbers and reads the same model in two inventory formats. ```js // enterprise/alignment.mjs — gap, overlap, orphan system; what two inventory formats see import { SYSTEM, CAPABILITY, COVERS, DATA, SYSTEMS, owner, edges } from "./model.mjs"; const col = (s, n) => String(s).padEnd(n); const owners = [...new Set(SYSTEMS.map(owner))].filter((o) => o !== "none"); const covering = (y) => SYSTEMS.filter((s) => COVERS[s].includes(y)); const E = edges(); const bidirectional = [...E.keys()].filter((k) => { const [a, b] = k.split("->"); return E.has(`${b}->${a}`) && a < b; }); const crossOwner = [...E.keys()].filter((k) => { const [a, b] = k.split("->"); return owner(a) !== owner(b); }); console.log(`MODEL enterprise: ${SYSTEMS.length} systems, ${owners.length} owners, ` + `${CAPABILITY.length} business capabilities, ${Object.keys(DATA).length} data assets`); console.log(`edges: ${E.size} directed edges, ${bidirectional.length} bidirectional, ` + `${crossOwner.length} cross an ownership boundary\n`); console.log(col("business capability", 24) + col("system", 8) + col("owner", 7) + "status"); console.log("-".repeat(60)); const status = (n, os) => (n === 0 ? "gap" : n === 1 ? "single" : os > 1 ? "overlap+conflict" : "overlap"); for (const y of CAPABILITY) { const k = covering(y); const os = new Set(k.map(owner)).size; console.log(col(y, 24) + col(k.length, 8) + col(os, 7) + status(k.length, os)); } const gap = CAPABILITY.filter((y) => covering(y).length === 0); const overlap = CAPABILITY.filter((y) => covering(y).length > 1); const conflict = overlap.filter((y) => new Set(covering(y).map(owner)).size > 1); const orphan = SYSTEMS.filter((s) => COVERS[s].length === 0); const settlement = overlap.reduce((t, y) => t + new Set(covering(y).map(owner)).size - 1, 0); console.log(`\ngap=${gap.length} (${gap.join(", ")})`); console.log(`overlap=${overlap.length}, ${conflict.length} of which are ownership conflicts`); console.log(`orphan system=${orphan.length} (${orphan.join(", ")})`); console.log(`alignment ratio = ${CAPABILITY.length - gap.length}/${CAPABILITY.length} = ` + `${(((CAPABILITY.length - gap.length) / CAPABILITY.length) * 100).toFixed(1)}%`); console.log(`settlements needed for overlapping capabilities = ${settlement}`); // ---- two inventory formats are derived from the same model ---- // A: system inventory — record unit is the system; capability names appear inside the system record // B: capability inventory — record unit is the capability; system names appear inside the capability record const finding = [...gap.map((y) => ["gap", y]), ...overlap.map((y) => ["overlap", y]), ...orphan.map((s) => ["orphan", s])]; const universeA = new Set(Object.values(COVERS).flat()); // A knows only the claimed capabilities const seesA = (t) => t !== "gap"; // an orphan system sits in A as an empty record const seesB = (t) => t !== "orphan"; // a capability record never names a system const countA = finding.filter(([t]) => seesA(t)).length; const countB = finding.filter(([t]) => seesB(t)).length; console.log(`\nsystem inventory A's capability universe: ${universeA.size}/${CAPABILITY.length} capabilities`); console.log(col("format", 24) + col("findings seen", 16) + "misses"); console.log("-".repeat(64)); console.log(col("A system inventory", 24) + col(`${countA}/${finding.length}`, 16) + `${gap.length} gaps`); console.log(col("B capability inventory", 24) + col(`${countB}/${finding.length}`, 16) + `${orphan.length} orphan systems`); console.log(col("enterprise model (A+B)", 24) + col(`${finding.length}/${finding.length}`, 16) + "-"); ``` ``` MODEL enterprise: 9 systems, 6 owners, 13 business capabilities, 10 data assets edges: 23 directed edges, 4 bidirectional, 20 cross an ownership boundary business capability system owner status ------------------------------------------------------------ material-search 2 2 overlap+conflict loan-issuance 2 2 overlap+conflict return-intake 2 2 overlap+conflict reservation 1 1 single membership-enrollment 1 1 single member-verification 2 2 overlap+conflict fee-calculation 1 1 single fee-collection 0 0 gap inter-branch-transfer 1 1 single asset-count 2 2 overlap+conflict usage-reporting 1 1 single purchase-suggestion 0 0 gap overdue-notification 2 1 overlap gap=2 (fee-collection, purchase-suggestion) overlap=6, 5 of which are ownership conflicts orphan system=1 (archive) alignment ratio = 11/13 = 84.6% settlements needed for overlapping capabilities = 5 system inventory A's capability universe: 11/13 capabilities format findings seen misses ---------------------------------------------------------------- A system inventory 7/9 2 gaps B capability inventory 8/9 1 orphan systems enterprise model (A+B) 9/9 - ``` #### The Three Numbers of Alignment Alignment is not one number but three separate numbers, and each says something different. A **gap** is a business capability no system covers. There are two in the model: fee collection and purchase suggestion. Both happen — fees are taken at the counter, purchase suggestions are collected on hand-kept lists — but neither appears in any system's record. A gap does not mean the work is not done; it means **the work is done outside any system**. In this model the alignment ratio is 11/13, or 84.6%. An **overlap** is a case where more than one system covers the same capability: there are six. Overlap itself is not a flaw; it is a requirement that both the kiosk and the loan service be able to issue a loan. What matters is **how many owners** the overlap spreads across. In five of the six overlaps the covering systems have different owners; these are **ownership conflicts**. Overdue notification is covered by two systems, but both are owned by the same unit, so a rule change is settled at a single table. In the other five, a rule change requires reconciling two separate budgets, two separate schedules, and two separate priorities. The settlement count in the model is five, and this number is the lower bound on the enterprise's coordination load. An **orphan system** is a system that cannot be tied to any capability: the archive. It has no owner either. An orphan system is not automatically something to shut down — reporting still reads the archive's record — but it is a line item with no defender: in a budget discussion, who will speak for it is unclear. The edge count sits on top of these three numbers. Across ten data assets, twenty-three directed edges emerge; four are bidirectional, and twenty cross an ownership boundary. That twenty of the twenty-three edges change owner puts a number on something at enterprise scale: integration is an organizational operation, not a technical one — in 87% of the edges, a change made by one party concerns another budget's owner. #### The Inventory's Format Determines the Finding The same model can be written in two formats. In the **system inventory**, the record unit is the system; each system's record lists the capabilities it covers. In the **capability inventory**, the record unit is the capability; each capability's record lists the systems that cover it. Both are written from the same model, but the set of findings they see is not the same. There are nine findings in the model in total: two gaps, six overlaps, one orphan system. The system inventory shows seven of them. What it cannot see is the gaps, because the system inventory's capability universe is derived from system records: of the thirteen capabilities, only eleven appear in any record at all, and the two that never appear are absent from the inventory. The capability inventory shows eight findings; what it cannot see is the orphan system, because a system whose name never appears in any capability's record never enters that inventory at all. Only the two-way mapping — the enterprise model — shows all nine. Enterprise architecture's first product is therefore not a list but a **mapping**: where the two lists fail to touch carries more information than either list on its own. #### Summary - Enterprise architecture's unit is not the deployment unit but the system with its own owner and its own budget; the model enterprise has 9 systems, 6 owners, 13 business capabilities, and 10 data assets. - Business–IT alignment is not one number but three: 2 gaps, 6 overlaps, 1 orphan system; alignment ratio 11/13 (84.6%). - The number of owners determines overlap's cost: 5 of the 6 overlaps spread across two separate owners and require 5 settlements in total; an overlap under the same owner is resolved at a single table. - The model enterprise has 23 directed edges; 4 are bidirectional and 20 cross an ownership boundary. - The system inventory shows 7 of the 9 findings, the capability inventory 8; all nine are read only from the mapping of both lists. #### Next Step This lesson built the enterprise model once and produced three numbers. How the model **should be documented** was never asked: how many layers it splits into, how many separate views it is written with, who reviews it and how often, and how much upkeep the document itself will demand are all still open. Enterprise architecture frameworks claim to answer exactly this question. The next lesson documents the same enterprise model in two framework formats — detailed-layered and lightweight — and counts how many questions each answers, how many items each requires to maintain, and when a framework's own maintenance cost exceeds the value of the question it answers. ### Frameworks Source: https://academia.sh/en/courses/enterprise-context/frameworks The shared structure of enterprise architecture frameworks — layer separation, view set, governance cycle, and maturity level — and documenting the same enterprise model in a detailed-layered and a lightweight format: how many questions each format answers, how many items it carries, how many items are touched per change, and the questions no format answers at all. The previous lesson built the enterprise model and tied alignment to three numbers. The model was produced once, but where it should be written down was never discussed: which layers it splits into, how many separate views it is kept in, who reviews it and how often are all unsettled. Enterprise architecture frameworks claim to fill this gap, and the claim usually becomes a debate — which framework to pick, whether the heavy one or the light one is right. This lesson turns the debate into a measurement. No brand or framework name is written; frameworks are referred to by their shared structure. #### The Shared Structure of Frameworks Whatever their names and details, enterprise architecture frameworks are built from four parts. **Layer separation** divides the enterprise into planes that are distinct from one another but mapped onto each other: business capabilities and their owners, systems, data assets, and the infrastructure that carries the systems. The **view set** is the same model's cross-sections, each prepared for a different reader. The **governance cycle** ties down which record is reviewed by whom and how often. The **maturity level** is a ladder ranking how much of these records the enterprise keeps. What these four parts share is this: none of them produces information, each tells you **where information gets written**. A framework can therefore only be measured through the record types it carries. Measurement needs three definitions. An **item** is a record row looked at independently; fields that fall on the same row are not counted as separate items (**EA5**). The **question set** is the twelve questions someone asks later; the measure is borrowed from the Architectural Decisions and Documentation course — a document's value is the number of questions it answers (**EA6**). A **maturity level** is not a title but a predicate tied to the presence of specific record types (**EA7**). The first file gives the previous lesson's enterprise model again; the block is repeated so it can run on its own. ```js // enterprise/model.mjs — MODEL regional library network: systems, owners, capabilities, data. // Fictional; no real institution, vendor, product, or person is described. export const SYSTEM = { catalog: { owner: "external-provider", budget: "service-fee" }, loan: { owner: "it-department", budget: "internal-development" }, billing: { owner: "it-department", budget: "internal-development" }, membership: { owner: "member-services", budget: "member-services" }, identity: { owner: "municipal-it", budget: "municipality" }, "branch-local": { owner: "branch-management", budget: "branch" }, kiosk: { owner: "branch-management", budget: "branch" }, reporting: { owner: "management-unit", budget: "management" }, archive: { owner: "none", budget: "none" }, }; // CAPABILITY — the business capabilities the enterprise must cover (a list independent of system) export const CAPABILITY = ["material-search", "loan-issuance", "return-intake", "reservation", "membership-enrollment", "member-verification", "fee-calculation", "fee-collection", "inter-branch-transfer", "asset-count", "usage-reporting", "purchase-suggestion", "overdue-notification"]; // COVERS[s] = the capabilities system s claims to cover export const COVERS = { catalog: ["material-search", "asset-count"], loan: ["loan-issuance", "return-intake", "reservation", "overdue-notification"], billing: ["fee-calculation", "overdue-notification"], membership: ["membership-enrollment", "member-verification"], identity: ["member-verification"], "branch-local": ["return-intake", "inter-branch-transfer", "asset-count"], kiosk: ["loan-issuance", "material-search"], reporting: ["usage-reporting"], archive: [], }; // DATA[v] = data asset; which system writes it, which ones read it export const DATA = { "member-record": { writes: ["membership"], reads: ["loan", "billing", "kiosk", "reporting"] }, "identity-match": { writes: ["identity", "membership"], reads: ["loan", "kiosk"] }, "material-record": { writes: ["catalog"], reads: ["loan", "kiosk", "branch-local", "reporting"] }, "copy-status": { writes: ["catalog", "loan", "branch-local"], reads: ["kiosk", "reporting"] }, "loan-transaction": { writes: ["loan", "kiosk"], reads: ["billing", "reporting"] }, "fee-record": { writes: ["billing"], reads: ["membership", "reporting", "kiosk"] }, "penalty-rule": { writes: ["billing"], reads: ["loan", "kiosk"] }, "transfer-request": { writes: ["branch-local"], reads: ["loan", "catalog"] }, "legacy-record": { writes: [], reads: ["archive", "reporting"] }, "count-discrepancy": { writes: ["branch-local"], reads: [] }, }; export const SYSTEMS = Object.keys(SYSTEM); export const owner = (s) => SYSTEM[s].owner; // edge = the directed pair from the system that writes a data asset to the system that reads it export function edges() { const e = new Map(); for (const [v, d] of Object.entries(DATA)) for (const w of d.writes) for (const r of d.reads) if (w !== r) e.set(`${w}->${r}`, [...(e.get(`${w}->${r}`) ?? []), v]); return e; } ``` #### Measurement The second file defines the two formats. The detailed-layered format keeps each record type as a separate item in its own layer; the lightweight format consists of nothing more than a single system table and a capability list, folding owner, budget, and coverage information into the system row's columns. ```js // enterprise/framework.mjs — same enterprise model in two framework formats: answer, item, maintenance import { SYSTEM, CAPABILITY, COVERS, DATA, SYSTEMS, owner, edges } from "./model.mjs"; const col = (s, n) => String(s).padEnd(n); const E = edges(); // ITEM[type] = number of rows of that record type in the model enterprise const ITEM = { capability: CAPABILITY.length, owner: new Set(SYSTEMS.map(owner)).size - 1, budget: new Set(SYSTEMS.map((s) => SYSTEM[s].budget)).size - 1, system: SYSTEMS.length, "capability-mapping": Object.values(COVERS).flat().length, data: Object.keys(DATA).length, edge: E.size, carrier: 4, "system-carrier": SYSTEMS.length, view: 4, governance: 4, maturity: 1, }; // FORMAT[f][type] = the independent item that carries that record type; null means the format does not hold it. // Records that fall on the same item do not need separate maintenance (this describes shared structure, not a brand). const own = (l) => Object.fromEntries(l.map((k) => [k, k])); const FORMAT = { "detailed-layered": own(Object.keys(ITEM)), lightweight: { system: "system", owner: "system", budget: "system", "capability-mapping": "system", capability: "capability" }, }; const holds = (f, type) => Boolean(FORMAT[f][type]); const itemCount = (f) => [...new Set(Object.values(FORMAT[f]))].reduce((t, k) => t + ITEM[k], 0); // QUESTION[i] = a question someone asks later; needs = the record types required to answer it const QUESTION = [ ["which system covers which capability", ["capability-mapping"]], ["who owns a system", ["owner"]], ["which capability is covered by no system", ["capability", "capability-mapping"]], ["if a system is shut down, which capability is left uncovered", ["capability-mapping"]], ["which budget pays for a system's cost", ["budget"]], ["which system writes a given data asset", ["data"]], ["which data passes between two systems", ["edge"]], ["which carrier does a system run on", ["system-carrier"]], ["which section of the document was reviewed when", ["governance"]], ["which view documents a given topic", ["view"]], ["how often does a given data asset pass", ["edge-frequency"]], ["who is notified if an edge breaks", ["edge-owner"]], ]; const answers = (f, needs) => needs.every((n) => holds(f, n)); // CHANGE[i][1] = the records that must be updated when the change happens const CHANGE = [ ["a new system goes live", { system: 1, "capability-mapping": 1, edge: 2, carrier: 1, "system-carrier": 1, view: 2, governance: 1 }], ["a system is transferred to another unit", { system: 1, owner: 1, budget: 1, view: 1, governance: 1 }], ["a new reader is added to a data asset", { data: 1, edge: 1, view: 1, governance: 1 }], ]; function touched(f, d) { const group = new Map(); // records that fall on the same item count as one maintenance for (const [type, n] of Object.entries(d)) { const k = FORMAT[f][type]; if (k) group.set(k, Math.max(group.get(k) ?? 0, n)); } return [...group.values()].reduce((t, n) => t + n, 0); } // MATURITY — rungs; each rung requires certain record types const MATURITY = [ ["a system inventory exists", ["system"]], ["every system's owner is recorded", ["system", "owner"]], ["a capability mapping exists", ["capability", "capability-mapping"]], ["data ownership is recorded", ["data"]], ["the impact of a change can be computed", ["data", "edge"]], ]; const level = (f) => { let n = 0; for (const [, needs] of MATURITY) { if (!needs.every((g) => holds(f, g))) break; n++; } return n; }; const formats = Object.keys(FORMAT); console.log(`MODEL enterprise: ${SYSTEMS.length} systems, ${CAPABILITY.length} capabilities, ` + `${Object.keys(DATA).length} data, ${E.size} edges; question set ${QUESTION.length} questions\n`); console.log(col("question", 64) + formats.map((f) => col(f, 20)).join("")); console.log("-".repeat(104)); for (const [q, needs] of QUESTION) console.log(col(q, 64) + formats.map((f) => col(answers(f, needs) ? "answer" : "-", 20)).join("")); console.log(""); console.log(col("format", 20) + col("answer", 8) + col("item", 8) + col("level", 9) + col("item/answer", 13) + "items touched per change"); console.log("-".repeat(100)); for (const f of formats) { const y = QUESTION.filter(([, g]) => answers(f, g)).length; const k = itemCount(f); const d = CHANGE.map(([, o]) => touched(f, o)); console.log(col(f, 20) + col(`${y}/${QUESTION.length}`, 8) + col(k, 8) + col(`${level(f)}/5`, 9) + col((k / y).toFixed(1), 13) + d.join(" + ") + ` = ${d.reduce((t, n) => t + n, 0)}`); } const [d, l] = formats; const ansD = QUESTION.filter(([, g]) => answers(d, g)).length; const ansL = QUESTION.filter(([, g]) => answers(l, g)).length; console.log(`\ndiff: ${itemCount(d) - itemCount(l)} more items -> ${ansD - ansL} more answers = ` + `${((itemCount(d) - itemCount(l)) / (ansD - ansL)).toFixed(1)} items/extra answer`); const untouched = CHANGE.filter(([, o]) => touched(l, o) === 0); console.log(`changes where the lightweight format touches no item: ${untouched.length} ` + `(${untouched.map(([name]) => name).join("; ")})`); const unanswered = QUESTION.filter(([, g]) => formats.every((f) => !answers(f, g))); console.log(`questions no format answers: ${unanswered.length} -> ` + unanswered.map(([q]) => q).join(" / ")); console.log("annual review rounds (capacity sensitivity):"); for (const cap of [20, 40, 60]) console.log(" per round " + col(cap, 4) + "items -> " + formats.map((f) => `${f}=${Math.ceil(itemCount(f) / cap)}`).join(", ")); ``` ``` MODEL enterprise: 9 systems, 13 capabilities, 10 data, 23 edges; question set 12 questions question detailed-layered lightweight -------------------------------------------------------------------------------------------------------- which system covers which capability answer answer who owns a system answer answer which capability is covered by no system answer answer if a system is shut down, which capability is left uncovered answer answer which budget pays for a system's cost answer answer which system writes a given data asset answer - which data passes between two systems answer - which carrier does a system run on answer - which section of the document was reviewed when answer - which view documents a given topic answer - how often does a given data asset pass - - who is notified if an edge breaks - - format answer item level item/answer items touched per change ---------------------------------------------------------------------------------------------------- detailed-layered 10/12 106 5/5 10.6 9 + 5 + 4 = 18 lightweight 5/12 22 3/5 4.4 1 + 1 + 0 = 2 diff: 84 more items -> 5 more answers = 16.8 items/extra answer changes where the lightweight format touches no item: 1 (a new reader is added to a data asset) questions no format answers: 2 -> how often does a given data asset pass / who is notified if an edge breaks annual review rounds (capacity sensitivity): per round 20 items -> detailed-layered=6, lightweight=2 per round 40 items -> detailed-layered=3, lightweight=1 per round 60 items -> detailed-layered=2, lightweight=1 ``` #### The Difference Between the Two Formats The detailed-layered format answers ten of the twelve questions and carries 106 items. The lightweight format answers five and carries 22 items. The difference comes down to a single number: **84 more items, 5 more answers** — 16.8 items per extra answer. The framework choice is made against this ratio. If an enterprise finds the answer to "which system writes a given data asset" worth the ongoing maintenance of 16.8 items, the detailed format wins; if it does not, the lightweight format wins. This is a calculation, not a preference, and the numbers depend on the size of the enterprise model: as the edge count grows, the detailed format's item count grows fast, because 23 of its 106 items are edges. The per-change cost difference is sharper: the same three changes update 18 items in the detailed format and 2 in the lightweight one. There is a caveat inside this. The change "a system is transferred to another unit" touches five items in the detailed format; the system itself does not change, only its owner and budget do. At enterprise scale, most maintenance comes not from technical change but from **ownership change**. The third change makes the distinction visible: when a new reader is added to a data asset, the lightweight format touches **no item at all**. This is the source of the lightweight format's cheapness, and at the same time its limit. The document does not go stale, because it never made any claim on the matter; it stays silent. A new edge forms in the enterprise, and no one looking at the document can see it. The maturity level is a shorthand for the same difference: the detailed format reaches five of the five levels, the lightweight format three. A level is not a ranking of success; it is which questions can be answered, reduced to a single number. The lightweight format stops at the fourth level, because it does not keep data ownership at all — and the fifth level, whether the impact of a change can be computed, depends exactly on this course's unit of measure. #### The Question a Framework Does Not Answer Two questions stay unanswered in either format: how often a given data asset passes, and who is notified when an edge breaks. The reason for this is not the framework choice. The model never carries these fields; because it does not carry them, no layer separation and no view set can produce them. A framework debate does not produce information, it organizes information that already exists — and if there is no information to organize, a more detailed framework only produces more empty items. The governance cycle's cost is read from the same place. The number of items that can be worked through per round is a threshold, and the source of that threshold should be written down: this model chose forty items per round (**EA8**). At this threshold, the detailed format is reviewed in three rounds, the lightweight format in one. When the threshold drops to twenty, the detailed format rises to six rounds; when it rises to sixty, it falls to two. If the review capacity stays below the item count, the document loses currency — that is, the framework itself becomes the source of the drift it measures. #### Summary - Enterprise architecture frameworks are built from four parts regardless of brand differences: layer separation, view set, governance cycle, and maturity level; none of them produces information, each determines where it belongs. - The same enterprise model answers 10 of 12 questions and carries 106 items in the detailed-layered format; it answers 5 and carries 22 items in the lightweight format. - The number that decides is items per extra answer: 84 more items buy 5 more answers, that is, 16.8 items/extra answer; maintenance per change is 18 items against 2. - One change touches no item at all in the lightweight format — a new reader being added. This is the source of the cheapness, and it is also where the document stays silent. - Two questions go unanswered in either format (a data asset's frequency and an edge's owner), because the model never carries those fields; choosing a framework does not fill a missing field. #### Next Step Both lessons treated the enterprise as a static structure: systems, owners, capabilities, records. But a business capability is not a list row; it is a process that unfolds step by step. Between a member walking through the door and taking a material in hand there is a sequence of operations, and not all of these steps happen inside software. The next lesson models a business process step by step and sorts every step into three classes — fully automatic, requiring human judgment, outside the system. What gets measured is where the automation boundary runs, and how many rollbacks come from forcing that boundary, that is, from automating a step that requires human judgment. ### Business Process Modeling Source: https://academia.sh/en/courses/enterprise-context/business-process-modeling Modeling a business process step by step and sorting every step into three classes — fully automatic, requiring human judgment, outside the system: deriving the automation boundary from the model, the number of actors and owners the process passes through, and the number of wrong decisions and rollbacks produced by automating a step that requires human judgment by forcing the boundary. The previous two lessons treated the enterprise as a static structure: systems, owners, capabilities, records. But a business capability is a single row in the inventory, yet as it happens it is a process that unfolds step by step. Behind the "loan issuance" row are dozens of operations lined up from the card scan to the security tag, and not all of them happen inside software. This lesson opens up that row and ties down where automation stops with a rule. #### A Process Is a Step Sequence Process notations differ in their symbol set; no brand or notation name is written here. What they share is this: every step has an **actor**, a set of **inputs**, and an **output**. Once these three fields are written down, the notation itself becomes secondary, because the fields are what can be measured. An actor is not always a system. The fee being collected at the counter, and the security tag being deactivated, are both steps of the process; both have an actor outside software. A process model's first job is not to delete these steps but to **classify** them. Three classes are used: fully automatic, requiring human judgment, outside the system. Classification is not left to opinion, it is derived from the model (**EA9**): a step is automatic only if every one of its inputs comes from a data asset that **has a writer**, and no information item the decision needs falls outside the data model. This rule separates two different obstacles from each other. The first is that the information exists nowhere — a material's damage degree is in no data asset. The second is that the information appears to exist as a data asset, but no system writes it; the legacy record in the enterprise model is like this, and it is called **ownerless data**. ```js // enterprise/model.mjs — MODEL regional library network: systems, owners, capabilities, data. // Fictional; no real institution, vendor, product, or person is described. export const SYSTEM = { catalog: { owner: "external-provider", budget: "service-fee" }, loan: { owner: "it-department", budget: "internal-development" }, billing: { owner: "it-department", budget: "internal-development" }, membership: { owner: "member-services", budget: "member-services" }, identity: { owner: "municipal-it", budget: "municipality" }, "branch-local": { owner: "branch-management", budget: "branch" }, kiosk: { owner: "branch-management", budget: "branch" }, reporting: { owner: "management-unit", budget: "management" }, archive: { owner: "none", budget: "none" }, }; // CAPABILITY — the business capabilities the enterprise must cover (a list independent of system) export const CAPABILITY = ["material-search", "loan-issuance", "return-intake", "reservation", "membership-enrollment", "member-verification", "fee-calculation", "fee-collection", "inter-branch-transfer", "asset-count", "usage-reporting", "purchase-suggestion", "overdue-notification"]; // COVERS[s] = the capabilities system s claims to cover export const COVERS = { catalog: ["material-search", "asset-count"], loan: ["loan-issuance", "return-intake", "reservation", "overdue-notification"], billing: ["fee-calculation", "overdue-notification"], membership: ["membership-enrollment", "member-verification"], identity: ["member-verification"], "branch-local": ["return-intake", "inter-branch-transfer", "asset-count"], kiosk: ["loan-issuance", "material-search"], reporting: ["usage-reporting"], archive: [], }; // DATA[v] = data asset; which system writes it, which ones read it export const DATA = { "member-record": { writes: ["membership"], reads: ["loan", "billing", "kiosk", "reporting"] }, "identity-match": { writes: ["identity", "membership"], reads: ["loan", "kiosk"] }, "material-record": { writes: ["catalog"], reads: ["loan", "kiosk", "branch-local", "reporting"] }, "copy-status": { writes: ["catalog", "loan", "branch-local"], reads: ["kiosk", "reporting"] }, "loan-transaction": { writes: ["loan", "kiosk"], reads: ["billing", "reporting"] }, "fee-record": { writes: ["billing"], reads: ["membership", "reporting", "kiosk"] }, "penalty-rule": { writes: ["billing"], reads: ["loan", "kiosk"] }, "transfer-request": { writes: ["branch-local"], reads: ["loan", "catalog"] }, "legacy-record": { writes: [], reads: ["archive", "reporting"] }, "count-discrepancy": { writes: ["branch-local"], reads: [] }, }; export const SYSTEMS = Object.keys(SYSTEM); export const owner = (s) => SYSTEM[s].owner; // edge = the directed pair from the system that writes a data asset to the system that reads it export function edges() { const e = new Map(); for (const [v, d] of Object.entries(DATA)) for (const w of d.writes) for (const r of d.reads) if (w !== r) e.set(`${w}->${r}`, [...(e.get(`${w}->${r}`) ?? []), v]); return e; } ``` #### Measurement The second file models the loan checkout process as sixteen steps, derives classification from the rule, and runs both boundaries over the same two hundred transactions. How often steps requiring human judgment occur, how many steps a wrong decision rolls back, and how many systems the fix requires writing to are chosen values (**EA10**). The probability of a wrong decision per missing information item was taken as 0.15; that is the source of the threshold, and its sensitivity is printed separately (**EA11**). ```js // enterprise/process.mjs — loan checkout process: step classes, automation boundary, cost of forcing it import { DATA, owner } from "./model.mjs"; const col = (s, n) => String(s).padEnd(n); // STEP — MODEL loan checkout process; system null means the step happens outside software (counter, shelf). // missing = the number of information items the step needs but that appear in no data asset. const STEP = [ { name: "member scans card", system: "kiosk", input: [] }, { name: "identity is verified", system: "identity", input: ["identity-match"] }, { name: "membership status is queried", system: "membership", input: ["member-record"] }, { name: "unpaid fee is checked", system: "billing", input: ["fee-record"] }, { name: "accrued fee is collected at the counter", system: null, input: [] }, { name: "material barcode is scanned", system: "kiosk", input: ["material-record"] }, { name: "copy status is queried", system: "catalog", input: ["copy-status"] }, { name: "transfer is opened for a copy at another branch", system: "branch-local", input: ["copy-status"], missing: 2, frequency: 0.12, rollbackSteps: 3, fix: ["branch-local", "catalog", "loan"] }, { name: "material's damage degree is recorded", system: "loan", input: ["material-record"], missing: 1, frequency: 0.06, rollbackSteps: 2, fix: ["loan", "billing"] }, { name: "loan period is calculated", system: "loan", input: ["penalty-rule", "member-record"] }, { name: "researcher extension is granted", system: "loan", input: ["member-record"], missing: 2, frequency: 0.04, rollbackSteps: 2, fix: ["loan", "membership"] }, { name: "debt in the legacy record is queried", system: "loan", input: ["legacy-record"], frequency: 0.09, rollbackSteps: 4, fix: ["loan", "billing", "reporting"] }, { name: "loan record is written", system: "loan", input: ["member-record"] }, { name: "copy status is updated", system: "loan", input: ["copy-status"] }, { name: "security tag is deactivated", system: null, input: [] }, { name: "overdue notification is scheduled", system: "loan", input: ["loan-transaction", "penalty-rule"] }, ]; // Automation boundary rule: a step can only be automated if every one of its inputs is a data // asset with a writer, and no information item it needs falls outside the data model. const unsourced = (a) => a.input.filter((v) => DATA[v].writes.length === 0); function classify(a) { if (a.system === null) return ["non-system", "happens outside any system"]; if (a.missing) return ["human-decision", `${a.missing} information item(s) in no data asset`]; if (unsourced(a).length) return ["human-decision", `data with no writer: ${unsourced(a)[0]}`]; return ["automatic", "-"]; } console.log(col("step", 50) + col("actor", 15) + col("class", 16) + "reason"); console.log("-".repeat(102)); for (const a of STEP) { const [s, n] = classify(a); console.log(col(a.name, 50) + col(a.system ?? "counter/shelf", 15) + col(s, 16) + n); } const byClass = (s) => STEP.filter((a) => classify(a)[0] === s); let longestRun = 0, streak = 0; for (const a of STEP) { streak = classify(a)[0] === "automatic" ? streak + 1 : 0; longestRun = Math.max(longestRun, streak); } const actor = (a) => a.system ?? "counter/shelf"; const stepOwner = (a) => (a.system ? owner(a.system) : "branch-management"); const handoff = STEP.slice(1).filter((a, i) => actor(a) !== actor(STEP[i])); const ownerCrossing = STEP.slice(1).filter((a, i) => stepOwner(a) !== stepOwner(STEP[i])); console.log(`\n${STEP.length} steps: ${byClass("automatic").length} automatic, ` + `${byClass("human-decision").length} human decision, ${byClass("non-system").length} non-system`); console.log(`longest unbroken automatic run: ${longestRun} steps`); console.log(`handoffs: ${handoff.length}, ${ownerCrossing.length} of which change owner; ` + `the process passes through ${new Set(STEP.map(actor)).size} actors and ${new Set(STEP.map(stepOwner)).size} owners`); // ---- forcing the boundary: if steps that require human judgment are automated ---- const P = 0.15; // chosen rule: probability of a wrong decision per missing information item const wrongProb = (a) => 1 - (1 - P) ** (a.missing ?? 1); const makeRandom = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32; }; const rand = makeRandom(90210); // seed is visible; the same seed gives the same numbers const N = 200; const humanSteps = byClass("human-decision"); let touches = 0, wrong = 0, rolledBack = 0, fixWrites = 0, crossOwnerFixes = 0; for (let i = 0; i < N; i++) for (const a of humanSteps) { if (rand() >= a.frequency) continue; touches++; if (rand() >= wrongProb(a)) continue; wrong++; rolledBack += a.rollbackSteps; fixWrites += a.fix.length; if (new Set(a.fix.map(owner)).size > 1) crossOwnerFixes++; } console.log(`\n${N} loan transactions, seed 90210, wrong-decision probability per item ${P}`); console.log(col("boundary", 18) + col("human touches", 16) + col("wrong decisions", 17) + col("rolled-back steps", 19) + "fix writes"); console.log("-".repeat(84)); console.log(col("current", 18) + col(touches, 16) + col(0, 17) + col(0, 19) + 0); console.log(col("forced", 18) + col(0, 16) + col(wrong, 17) + col(rolledBack, 19) + fixWrites); console.log(`\nthe forced boundary avoids ${touches} human touches, and produces ${wrong} wrong ` + `decisions in exchange`); console.log(`${crossOwnerFixes} of these ${wrong} rollbacks concern more than one owner`); console.log(`break-even: if a rollback costs more than ${(touches / wrong).toFixed(2)} human ` + `touches, the current boundary wins`); console.log("probability sensitivity (expected value):"); for (const p of [0.1, 0.15, 0.2]) { const expWrong = humanSteps.reduce((t, a) => t + N * a.frequency * (1 - (1 - p) ** (a.missing ?? 1)), 0); const expTouches = humanSteps.reduce((t, a) => t + N * a.frequency, 0); console.log(` p=${p.toFixed(2)} -> expected wrong ${expWrong.toFixed(1)}, ` + `break-even ${(expTouches / expWrong).toFixed(2)} touches`); } ``` ``` step actor class reason ------------------------------------------------------------------------------------------------------ member scans card kiosk automatic - identity is verified identity automatic - membership status is queried membership automatic - unpaid fee is checked billing automatic - accrued fee is collected at the counter counter/shelf non-system happens outside any system material barcode is scanned kiosk automatic - copy status is queried catalog automatic - transfer is opened for a copy at another branch branch-local human-decision 2 information item(s) in no data asset material's damage degree is recorded loan human-decision 1 information item(s) in no data asset loan period is calculated loan automatic - researcher extension is granted loan human-decision 2 information item(s) in no data asset debt in the legacy record is queried loan human-decision data with no writer: legacy-record loan record is written loan automatic - copy status is updated loan automatic - security tag is deactivated counter/shelf non-system happens outside any system overdue notification is scheduled loan automatic - 16 steps: 10 automatic, 4 human decision, 2 non-system longest unbroken automatic run: 4 steps handoffs: 10, 9 of which change owner; the process passes through 8 actors and 5 owners 200 loan transactions, seed 90210, wrong-decision probability per item 0.15 boundary human touches wrong decisions rolled-back steps fix writes ------------------------------------------------------------------------------------ current 62 0 0 0 forced 0 10 30 27 the forced boundary avoids 62 human touches, and produces 10 wrong decisions in exchange 8 of these 10 rollbacks concern more than one owner break-even: if a rollback costs more than 6.20 human touches, the current boundary wins probability sensitivity (expected value): p=0.10 -> expected wrong 9.1, break-even 6.83 touches p=0.15 -> expected wrong 13.4, break-even 4.63 touches p=0.20 -> expected wrong 17.5, break-even 3.54 touches ``` #### Where the Boundary Runs Ten of the sixteen steps are automatic, four require human judgment, two are outside the system. These three numbers alone give the automation rate but not the boundary's **location**. The number that gives the location is the length of the longest unbroken automatic run: four steps. The process runs four steps automatically, then falls to the counter; two more steps run automatically, then it snags on two human decisions. The value of automation is not in the step count but in the length of the unbroken run — every break means a wait, a handoff, and starting over. At enterprise scale, the number that matters is the handoff. Across sixteen steps the actor changes ten times, and in nine of those ten changes **the owner changes too**. The process passes through eight actors and five separate owners. A process's enterprise cost is read from here: technically a sixteen-step flow, organizationally nine boundary crossings. The reasons behind the four steps requiring human judgment are not the same. In three of them, the information is in no data asset. The fourth is different: querying the debt in the legacy record rests on a data asset, but no system writes that data. Ownerless data comes back inside the process as a human decision — a gap in enterprise architecture turns into a staff step in the process. #### The Cost of Forcing the Boundary The second table runs the same two hundred loan transactions through two boundaries. Under the **current boundary**, four steps are left to staff; across two hundred transactions these steps occur 62 times and spend 62 human touches. Under the **forced boundary**, all four are automated; human touches drop to zero, and wrong decisions begin in exchange. With the generator seeded at 90210, this run produced 10 wrong decisions; since the expected value is 13.4, the number depends on the run and changes if the seed changes. Ten wrong decisions roll back 30 steps, and the fix requires a system write 27 times. The number that matters at enterprise scale is the last one: **eight of the ten rollbacks** concern more than one owner. The cost of a wrongly automated step is not fixing a single screen, it is two or three separate budget owners reconciling the same record. The decision is read from here: if reconciling a rollback costs more than 6.20 human touches, the current boundary wins. This number is sensitive to probability — when the probability of a wrong decision per missing information item is 0.10, the break-even rises to 6.83 touches; at 0.20 it falls to 3.54. Forcing the automation boundary is not a matter of courage, it is the comparison of these two numbers: touches avoided and rollbacks produced. #### Summary - Three fields of a process step are measurable: actor, input set, output; the notation's symbol set is secondary next to these fields. - The automation boundary is derived from a rule: a step is automatic only if all its inputs are in a data asset that has a writer and no information item falls outside it. - In the model process, 10 of 16 steps are automatic, 4 require human judgment, 2 are outside the system; the longest unbroken automatic run is 4 steps. - The process passes through 8 actors and 5 owners; the owner also changes in 9 of the 10 handoffs. - When the boundary is forced, 62 human touches are avoided; in exchange this run produces 10 wrong decisions, 30 rolled-back steps, and 27 fix writes; 8 of the rollbacks concern more than one owner, and the break-even point is 6.20 touches. #### Next Step This lesson opened up a single process for a single capability and counted which systems that process passes through. The enterprise has thirteen capabilities, and a similar chain stands behind each one. When a capability changes — the loan-period rule is renewed, a membership condition changes — which systems will be touched, what data those systems write, and who reads that data is not known unless it is traced one by one. The next lesson builds a traceability chain between capability, system, and data; it counts the number of systems touched when a capability changes and the points where the chain breaks. ### Capability Mapping Source: https://academia.sh/en/courses/enterprise-context/capability-mapping Building a traceability chain that runs from a business capability to the system that covers it and the data that system writes: the number of systems and owners touched when a capability changes, the difference between the short chain and the full chain, and the points where the chain breaks — untraceable capability, orphan system, ownerless data, unread data. The previous lesson opened up a single process for a single capability and counted that the process passes through eight actors and five owners. The enterprise has thirteen capabilities, and a similar chain stands behind each one. When a capability changes — the loan-period rule is renewed, a membership condition changes — and someone checks whether anyone asked who should be notified, there is no list to look at: the inventory says which system covers which capability, but it does not say who reads the data that system writes. This lesson adds the missing link. #### The Traceability Chain **Traceability** is being able to travel without a break from a business capability to the pieces of software that carry it out, and to the data those pieces produce. The chain is built from three links (**EA12**): the capability, the systems that claim to cover it, and the data assets those systems write. One more link is added: the systems that **read** the written data. This last link is kept to a single hop; it does not extend to the data those readers themselves write. What the chain measures is this: the set of systems touched when a capability's definition changes. The direct link is the systems that cover that capability; they are the ones who carry out the change. The indirect link is the readers of the changed data; they may not change code, but they need to see the change, because the field they read is transformed. A coordination round is taken as one less than the number of distinct affected owners (**EA13**): an owner does not negotiate with itself, and every additional owner adds one round of reconciliation. ```js // enterprise/model.mjs — MODEL regional library network: systems, owners, capabilities, data. // Fictional; no real institution, vendor, product, or person is described. export const SYSTEM = { catalog: { owner: "external-provider", budget: "service-fee" }, loan: { owner: "it-department", budget: "internal-development" }, billing: { owner: "it-department", budget: "internal-development" }, membership: { owner: "member-services", budget: "member-services" }, identity: { owner: "municipal-it", budget: "municipality" }, "branch-local": { owner: "branch-management", budget: "branch" }, kiosk: { owner: "branch-management", budget: "branch" }, reporting: { owner: "management-unit", budget: "management" }, archive: { owner: "none", budget: "none" }, }; // CAPABILITY — the business capabilities the enterprise must cover (a list independent of system) export const CAPABILITY = ["material-search", "loan-issuance", "return-intake", "reservation", "membership-enrollment", "member-verification", "fee-calculation", "fee-collection", "inter-branch-transfer", "asset-count", "usage-reporting", "purchase-suggestion", "overdue-notification"]; // COVERS[s] = the capabilities system s claims to cover export const COVERS = { catalog: ["material-search", "asset-count"], loan: ["loan-issuance", "return-intake", "reservation", "overdue-notification"], billing: ["fee-calculation", "overdue-notification"], membership: ["membership-enrollment", "member-verification"], identity: ["member-verification"], "branch-local": ["return-intake", "inter-branch-transfer", "asset-count"], kiosk: ["loan-issuance", "material-search"], reporting: ["usage-reporting"], archive: [], }; // DATA[v] = data asset; which system writes it, which ones read it export const DATA = { "member-record": { writes: ["membership"], reads: ["loan", "billing", "kiosk", "reporting"] }, "identity-match": { writes: ["identity", "membership"], reads: ["loan", "kiosk"] }, "material-record": { writes: ["catalog"], reads: ["loan", "kiosk", "branch-local", "reporting"] }, "copy-status": { writes: ["catalog", "loan", "branch-local"], reads: ["kiosk", "reporting"] }, "loan-transaction": { writes: ["loan", "kiosk"], reads: ["billing", "reporting"] }, "fee-record": { writes: ["billing"], reads: ["membership", "reporting", "kiosk"] }, "penalty-rule": { writes: ["billing"], reads: ["loan", "kiosk"] }, "transfer-request": { writes: ["branch-local"], reads: ["loan", "catalog"] }, "legacy-record": { writes: [], reads: ["archive", "reporting"] }, "count-discrepancy": { writes: ["branch-local"], reads: [] }, }; export const SYSTEMS = Object.keys(SYSTEM); export const owner = (s) => SYSTEM[s].owner; // edge = the directed pair from the system that writes a data asset to the system that reads it export function edges() { const e = new Map(); for (const [v, d] of Object.entries(DATA)) for (const w of d.writes) for (const r of d.reads) if (w !== r) e.set(`${w}->${r}`, [...(e.get(`${w}->${r}`) ?? []), v]); return e; } ``` #### Measurement The second file builds the chain for every capability, compares the two chain formats, and counts the break points. ```js // enterprise/tracing.mjs — capability -> system -> data -> reader chain; systems touched by a change import { SYSTEM, CAPABILITY, COVERS, DATA, SYSTEMS, owner } from "./model.mjs"; const col = (s, n) => String(s).padEnd(n); const covering = (y) => SYSTEMS.filter((s) => COVERS[s].includes(y)); const writes = (s) => Object.entries(DATA).filter(([, d]) => d.writes.includes(s)).map(([v]) => v); const ownerSet = (l) => new Set(l.map(owner).filter((o) => o !== "none")); function chain(y) { const direct = covering(y); const data = [...new Set(direct.flatMap(writes))]; const indirect = [...new Set(data.flatMap((v) => DATA[v].reads))].filter((s) => !direct.includes(s)); return { direct, data, indirect, all: [...direct, ...indirect] }; } console.log(col("business capability", 24) + col("system", 8) + col("data", 6) + col("readers", 8) + col("touched", 11) + col("owner", 7) + "coordination rounds"); console.log("-".repeat(88)); let shortSystems = 0, fullSystems = 0, shortRounds = 0, fullRounds = 0; const untraceable = []; for (const y of CAPABILITY) { const z = chain(y); if (z.direct.length === 0) { untraceable.push(y); continue; } const fullOwners = ownerSet(z.all).size, shortOwners = ownerSet(z.direct).size; shortSystems += z.direct.length; fullSystems += z.all.length; shortRounds += Math.max(0, shortOwners - 1); fullRounds += Math.max(0, fullOwners - 1); console.log(col(y, 24) + col(z.direct.length, 8) + col(z.data.length, 6) + col(z.indirect.length, 8) + col(z.all.length, 11) + col(fullOwners, 7) + (fullOwners - 1)); } const traced = CAPABILITY.length - untraceable.length; console.log(`\n${traced}/${CAPABILITY.length} capabilities are traceable; untraceable: ${untraceable.join(", ")}`); console.log(col("chain format", 28) + col("systems touched", 18) + col("per capability", 16) + "coordination rounds"); console.log("-".repeat(80)); console.log(col("short (capability->system)", 28) + col(shortSystems, 18) + col((shortSystems / traced).toFixed(2), 16) + shortRounds); console.log(col("full (->data->readers)", 28) + col(fullSystems, 18) + col((fullSystems / traced).toFixed(2), 16) + fullRounds); console.log(`diff: ${fullSystems - shortSystems} systems and ${fullRounds - shortRounds} coordination rounds ` + `do not show up in the short chain (${((fullSystems - shortSystems) / fullSystems * 100).toFixed(0)}%)`); const widest = CAPABILITY.filter((y) => covering(y).length) .map((y) => [y, chain(y).all.length]).sort((a, b) => b[1] - a[1])[0]; console.log(`widest chain: ${widest[0]} -> ${widest[1]} systems`); // ---- where the chain breaks ---- const brk = [ ["untraceable capability", untraceable], ["orphan system (cannot connect to a capability)", SYSTEMS.filter((s) => COVERS[s].length === 0)], ["ownerless system", SYSTEMS.filter((s) => SYSTEM[s].owner === "none")], ["ownerless data (no writer)", Object.keys(DATA).filter((v) => DATA[v].writes.length === 0)], ["unread data", Object.keys(DATA).filter((v) => DATA[v].reads.length === 0)], ]; console.log(""); console.log(col("break type", 50) + col("count", 6) + "items"); console.log("-".repeat(78)); for (const [t, l] of brk) console.log(col(t, 50) + col(l.length, 6) + (l.join(", ") || "-")); const total = brk.reduce((t, [, l]) => t + l.length, 0); const reached = new Set(CAPABILITY.filter((y) => covering(y).length).flatMap((y) => chain(y).data)); console.log(`\n${total} break points in total; ` + `${reached.size} of the ${Object.keys(DATA).length} data assets are reachable from some capability`); ``` ``` business capability system data readers touched owner coordination rounds ---------------------------------------------------------------------------------------- material-search 2 3 4 6 4 3 loan-issuance 2 2 2 4 3 2 return-intake 2 4 4 6 4 3 reservation 1 2 3 4 3 2 membership-enrollment 1 2 4 5 4 3 member-verification 2 2 4 6 5 4 fee-calculation 1 2 4 5 4 3 inter-branch-transfer 1 3 4 5 4 3 asset-count 2 4 3 5 4 3 usage-reporting 1 0 0 1 1 0 overdue-notification 2 4 3 5 4 3 11/13 capabilities are traceable; untraceable: fee-collection, purchase-suggestion chain format systems touched per capability coordination rounds -------------------------------------------------------------------------------- short (capability->system) 17 1.55 5 full (->data->readers) 52 4.73 29 diff: 35 systems and 24 coordination rounds do not show up in the short chain (67%) widest chain: material-search -> 6 systems break type count items ------------------------------------------------------------------------------ untraceable capability 2 fee-collection, purchase-suggestion orphan system (cannot connect to a capability) 1 archive ownerless system 1 archive ownerless data (no writer) 1 legacy-record unread data 1 count-discrepancy 6 break points in total; 9 of the 10 data assets are reachable from some capability ``` #### The Number of Systems a Change Touches The short chain shows only the covering systems: 17 systems in total for the eleven traceable capabilities, 1.55 per capability. The full chain adds the data and its readers too: 52 systems, 4.73 per capability. The 35 systems in between are systems that someone looking only at the short chain would never think to notify — **67% of the systems** that will be touched never show up in the first link. The coordination-round difference is sharper: 5 in total in the short chain, 29 in the full chain. What produces the cost at enterprise scale is not the size of the code change but these 24 extra rounds; every round is a reconciliation with a separate budget's owner. Two rows of the table show the extremes. The member-verification capability is covered by two systems, but its chain spreads across six systems and five owners; this is the most expensive change in the enterprise, because even the two covering systems have different owners. At the other end is usage reporting: one system, zero data, zero readers, zero coordination rounds. Reporting writes no data, it only reads; that is why a change made to it spreads nowhere. What determines a system's change cost is not its size, it is **how many readers the data it writes has**. #### Where the Chain Breaks The chain cannot be built everywhere; there are six break points, and each has a different consequence. There are two **untraceable capabilities**: fee collection and purchase suggestion. Both happen, but neither can be connected to any system, so the list of systems to touch when they change comes out empty. This does not mean the change is free; it means the cost is paid outside any system, at the counter and on hand-kept lists. **Orphan system** and **ownerless system** land on the same item, the archive: it cannot be connected to any capability, and its owner is not recorded. Such a system never appears in the traceability table at all; both the decision to shut it down and the decision to keep it are left undefended. The **ownerless data** is the legacy record — it is read but has no writer; nine of the ten data assets are reachable from a capability, and this is the one that is not. As seen in the previous lesson, this gap came back as a human decision inside the process. **Unread data**, on the other hand, is the count discrepancy: it is written, no one reads it. These are breaks at the two ends of the chain, and both say the same thing — a data asset's value lies in both of its ends being connected. #### Summary - The traceability chain has four links: capability, covering systems, the data those systems write, and the systems that read that data; a coordination round is one less than the number of distinct affected owners. - 11 of the 13 capabilities in the model enterprise are traceable; the remaining two cannot be connected to any system. - The short chain shows 1.55 systems per capability, the full chain 4.73; 67% of the systems that will be touched do not show up in the first link, and the coordination rounds rise from 5 to 29. - The chain needing the most owners and rounds is member verification: 6 systems, 5 owners, 4 rounds. The narrowest chain is usage reporting: 1 system, 0 readers, 0 rounds — because it writes no data. - The chain breaks at 6 points: 2 untraceable capabilities, 1 orphan system, 1 ownerless system, 1 ownerless data asset, 1 unread data asset; 9 of the 10 data assets are reachable from a capability. #### Next Step This topic modeled the enterprise in four steps: systems were mapped to capabilities, the same model was documented in two framework formats, a business process was classified step by step, and traceability from capability to data was built. What is now visible is which system a capability sits in and how many systems a change concerns. One thing remains invisible, and it is the subject of the second half of this course: **how** systems connect to each other. Behind every link in the chain that reads "this system reads that data" there is a transport form — which route the data travels, how often it is refreshed, who is responsible if it breaks. It was not for nothing that two questions in the frameworks lesson could not be answered in any format: the model never carried those fields. The next lesson turns the edge itself into an object of choice, and builds the same set of edges with different transport forms to count its cost. ## Integration Connecting the systems inside an enterprise to each other: the per-edge cost of file transfer, shared database, remote call, and messaging; the benefits and bottleneck risk of a central integration point; master data management with a system of record and a golden record; transformation pipelines that separate the resolution load from the operating load; replacing legacy systems that cannot be changed by wrapping them in stages; and the build-or-buy decision weighing total cost of ownership against dependency risk. ### Integration Patterns Source: https://academia.sh/en/courses/enterprise-context/integration-patterns Building and comparing the same edge with four patterns: round trips and requests to the source system per query, the freshness window, records moved per refresh, the number of physical schema names leaking into consumer code, and the edges broken and owners to coordinate with when the source renames a single column. The previous topic drew out the enterprise's capabilities and showed which system each one lives on. That mapping never asked one question: how systems connect to each other. If a capability lives on one system, a second system that needs the same data gets it by some path, at some frequency, and under someone's responsibility. This lesson takes a single edge, builds it four separate ways, and compares all four with the same measures. At enterprise scale the unit of measure is the edge. Inside a single system, where a call goes is that system's owner's decision; at enterprise scale, two separate owners, two separate budgets, and two separate schedules stand at the two ends of an edge. So what gets counted here is not only latency, but what the edge obligates whom to. **IN1 — the enterprise is fictional and is written as a data structure in `node`.** Systems, their owners, and the edges between them stand as an array; the edge being measured is one row of that array. **IN2 — the refresh period is modeled with a `tick` rather than a timer.** Rationale: the period's clock-time equivalent (a minute, an hour, a night) is set by an operating decision, not by the pattern itself; the quantity measured is "how many refresh periods until fresh," and that number is independent of the period's length. **IN3 — the messaging carrier is modeled as an append log.** Queue mechanics, delivery guarantees, and consumer competition were measured in the Caching, Queues and Asynchronous Processing course; what is measured here is not the carrier itself, but what the edge costs the source and its owner. #### The Enterprise and the Measured Edge The regional library network is a fictional enterprise. The catalog system was acquired externally; the loan, billing, and notification services were written in-house; membership belongs to a separate unit, and identity to the municipality. The single entity measured in this lesson is copy status: whether a copy of a book is on the shelf. The catalog writes this data; five systems read it. ```js // enterprise.mjs — regional library network: a fictional enterprise model; prints itself when run directly export const SYSTEM = [ { name: "catalog", owner: "catalog unit", external: 1 }, { name: "loan", owner: "in-house development", external: 0 }, { name: "billing", owner: "in-house development", external: 0 }, { name: "notification", owner: "in-house development", external: 0 }, { name: "reporting", owner: "management unit", external: 0 }, { name: "branch-interface", owner: "branch unit", external: 0 }, { name: "membership", owner: "membership unit", external: 0 }, { name: "identity", owner: "municipal IT", external: 1 }, ]; // edges that read copy status from the catalog; each edge is built with one pattern (model) export const EDGE = [ { target: "loan", pattern: "shared-db" }, { target: "billing", pattern: "shared-db" }, { target: "reporting", pattern: "file" }, { target: "branch-interface", pattern: "remote-call" }, { target: "notification", pattern: "message" }, ]; export const ownerOf = (a) => SYSTEM.find((s) => s.name === a).owner; if (process.argv[1].endsWith("enterprise.mjs")) { console.log(`${SYSTEM.length} systems, ${new Set(SYSTEM.map((s) => s.owner)).size} distinct owners, ` + `${SYSTEM.filter((s) => s.external).length} systems acquired externally`); console.log(`edges out of catalog: ${EDGE.length}, owners touched: ` + `${new Set(EDGE.map((b) => ownerOf(b.target))).size}`); for (const k of ["shared-db", "file", "remote-call", "message"]) { console.log(` ${k.padEnd(13)}${EDGE.filter((b) => b.pattern === k).length} edge`); } } ``` ``` 8 systems, 6 distinct owners, 2 systems acquired externally edges out of catalog: 5, owners touched: 3 shared-db 2 edge file 1 edge remote-call 1 edge message 1 edge ``` The source system owns the copy table and gives access to it four ways: an endpoint that returns a single record in published format, a snapshot in the same format, a dump that exports a copy of the physical table, and a log that appends one row in published format on every change. Physical column names live inside `SCHEMA` and are read only in the source's own code. ```js // catalog.mjs — the source system that owns the copy table; ; long-lived process import { createServer } from "node:http"; import { appendFileSync, existsSync, readFileSync, writeFileSync } from "node:fs"; const [port] = process.argv.slice(2); if (!port) { console.log("usage: node catalog.mjs "); process.exit(1); } const LATENCY_MS = 20; // how long the catalog takes to serve a request (ms) const SCHEMA = { status: "status", branch: "branchCode" }; // physical column names; belongs to the catalog const write = (s) => writeFileSync("catalog.db", s.map((r) => JSON.stringify(r)).join("\n") + "\n"); const read = () => readFileSync("catalog.db", "utf8").split("\n").filter(Boolean).map(JSON.parse); const published = (r) => ({ copyId: r.copyId, branch: r[SCHEMA.branch], available: r[SCHEMA.status] === "shelved" }); // outward format; the catalog's word if (!existsSync("catalog.db")) { write([...Array(200).keys()].map((i) => ({ copyId: 401 + i, branchCode: 1 + (i % 4), status: i % 5 === 0 ? "checked-out" : "shelved", printYear: 1990 + (i % 30) }))); } let load = 0; // number of requests the source has seen createServer(async (request, response) => { const u = new URL(request.url, "http://y"); if (u.pathname === "/load") { response.end(String(load)); return; } load += 1; await new Promise((c) => setTimeout(c, LATENCY_MS)); const id = Number(u.searchParams.get("id")); if (u.pathname === "/record") { // remote call: single record, published format response.end(JSON.stringify(published(read().find((r) => r.copyId === id)))); } else if (u.pathname === "/snapshot") { // message consumer's first fill, published format response.end(JSON.stringify(read().map(published))); } else if (u.pathname === "/dump") { // file transfer: a copy of the physical table const s = read(); writeFileSync("catalog.dump", s.map((r) => JSON.stringify(r)).join("\n") + "\n"); response.end(String(s.length)); } else if (u.pathname === "/change") { // the catalog's own write path const s = read(), r = s.find((x) => x.copyId === id); r[SCHEMA.status] = u.searchParams.get("status"); write(s); appendFileSync("catalog.events", JSON.stringify(published(r)) + "\n"); response.end("done"); } else if (u.pathname === "/migrate") { // the catalog renames its own column write(read().map((r) => { const { [SCHEMA.status]: d, ...k } = r; return { ...k, copyStatus: d }; })); SCHEMA.status = "copyStatus"; response.end("done"); } else response.end("none"); }).listen(Number(port)); ``` #### The Same Edge, Four Ways The consumer runs from a single source file and takes its pattern from the command line. Each of the four sections defines two functions: `QUERY`, which answers the query, and `TICK`, which represents the refresh period. Because the sections are marked, which pattern reads which name can be counted straight from the code. ```js // loan.mjs — consumer; import { createServer } from "node:http"; import { existsSync, readFileSync } from "node:fs"; const [pattern, port, cp] = process.argv.slice(2); if (!port) { console.log("usage: node loan.mjs "); process.exit(1); } const C = `http://127.0.0.1:${cp}`; const lines = (d) => readFileSync(d, "utf8").split("\n").filter(Boolean).map(JSON.parse); const local = new Map(); // the consumer's own copy let read = 0; const QUERY = {}, TICK = {}; // TICK: the trigger modeling the refresh period // --- shared-db --- the consumer reads the source's physical table directly QUERY["shared-db"] = (id) => lines("catalog.db").find((r) => r.copyId === id).status === "shelved"; TICK["shared-db"] = () => 0; // --- file --- the source dumps the physical table, the consumer reads the dump file from scratch QUERY["file"] = (id) => local.get(id); TICK["file"] = async () => { await fetch(`${C}/dump`); const s = lines("catalog.dump"); for (const r of s) local.set(r.copyId, r.status === "shelved"); return s.length; }; // --- remote-call --- every query calls the source's published format QUERY["remote-call"] = async (id) => (await (await fetch(`${C}/record?id=${id}`)).json()).available; TICK["remote-call"] = () => 0; // --- message --- the first fill is the published format, after that only change lines are read QUERY["message"] = (id) => local.get(id); TICK["message"] = () => { if (!existsSync("catalog.events")) return 0; const s = lines("catalog.events"); let n = 0; while (read < s.length) { local.set(s[read].copyId, s[read].available); read += 1; n += 1; } return n; }; if (pattern === "message") for (const r of await (await fetch(`${C}/snapshot`)).json()) local.set(r.copyId, r.available); // --- common --- service surface independent of pattern createServer(async (request, response) => { const u = new URL(request.url, "http://y"); if (u.pathname === "/tick") { response.end(String(await TICK[pattern]())); return; } response.end(String(await QUERY[pattern](Number(u.searchParams.get("id"))))); }).listen(Number(port)); ``` #### Measurement The tool runs in three stages. First it sends a hundred queries to each consumer, counting the round trip per query and the number of requests that reach the source system in the process. Then it changes the status of a single copy at the source and asks the consumer twice: before the tick and after the tick. In the final stage the source renames its own column, and it is checked whether all four consumers still answer the same question correctly. The name count in consumer code is read from the source text, not from the running processes. ```js // measure.mjs — measures the same edge with four patterns; import { readFileSync } from "node:fs"; import { EDGE, ownerOf } from "./enterprise.mjs"; const args = process.argv.slice(2); if (args.length < 5) { console.log("usage: node measure.mjs "); process.exit(1); } const [cp, ...port] = args; const PATTERN = ["shared-db", "file", "remote-call", "message"]; const CONSUMER = Object.fromEntries(PATTERN.map((k, i) => [k, `http://127.0.0.1:${port[i]}`])); const C = `http://127.0.0.1:${cp}`, LATENCY_MS = 20, QUERIES = 100; const text = (u) => fetch(u).then((y) => y.text()); const query = (k, id) => text(`${CONSUMER[k]}/query?id=${id}`); const tick = (k) => text(`${CONSUMER[k]}/tick`).then(Number); const o = {}; for (const k of PATTERN) { o[k] = {}; await tick(k); } // first fill, before measurement for (const k of PATTERN) { // latency and load seen by the source const l0 = Number(await text(`${C}/load`)); const t = performance.now(); for (let i = 0; i < QUERIES; i += 1) await query(k, 401 + (i % 200)); o[k].roundTrip = Math.round((performance.now() - t) / QUERIES / LATENCY_MS); o[k].load = Number(await text(`${C}/load`)) - l0; } await text(`${C}/change?id=402&status=checked-out`); // a single record changes at the source for (const k of PATTERN) { const before = await query(k, 402); o[k].records = await tick(k); o[k].window = before === "false" ? 0 : (await query(k, 402)) === "false" ? 1 : -1; } const section = {}; // consumer code per pattern for (const p of readFileSync("loan.mjs", "utf8").split(/^\/\/ --- /m).slice(1)) { section[p.slice(0, p.indexOf(" "))] = p; } const PHYSICAL = ["catalog.db", "catalog.dump", "status", "shelved"]; // the source's internal schema const PUBLISHED = ["/record", "/snapshot", "catalog.events", "available"]; // the source's published contract const count = (m, l) => l.filter((a) => m.includes(a)).length; await text(`${C}/migrate`); // the source renames its own column for (const k of PATTERN) { await tick(k); o[k].correct = (await query(k, 403)) === "true"; } console.log(`${"pattern".padEnd(13)}${"round trip".padStart(12)}${"source requests".padStart(17)}` + `${"freshness tick".padStart(16)}${"tick records".padStart(14)}${"physical names".padStart(16)}` + `${"published names".padStart(18)}${"after migrate".padStart(15)}`); for (const k of PATTERN) { console.log(`${k.padEnd(13)}${String(o[k].roundTrip).padStart(12)}${String(o[k].load).padStart(17)}` + `${String(o[k].window).padStart(16)}${String(o[k].records).padStart(14)}` + `${String(count(section[k], PHYSICAL)).padStart(16)}${String(count(section[k], PUBLISHED)).padStart(18)}` + `${(o[k].correct ? "correct" : "wrong").padStart(15)}`); } const broken = EDGE.filter((b) => !o[b.pattern].correct); console.log(`single column name changed: ${broken.length}/${EDGE.length} edges broke ` + `(${broken.map((b) => b.target).join(", ")}); owners to coordinate with ` + `${new Set(broken.map((b) => ownerOf(b.target))).size}/${new Set(EDGE.map((b) => ownerOf(b.target))).size}`); ``` ```bash rm -f catalog.db catalog.dump catalog.events node catalog.mjs 8981 & K=$! sleep 0.6 node loan.mjs shared-db 8982 8981 & A=$! node loan.mjs file 8983 8981 & B=$! node loan.mjs remote-call 8984 8981 & C=$! node loan.mjs message 8985 8981 & D=$! sleep 0.8 node measure.mjs 8981 8982 8983 8984 8985 kill $K $A $B $C $D ``` ``` pattern round trip source requests freshness tick tick records physical names published names after migrate shared-db 0 0 0 0 3 0 wrong file 0 0 1 200 3 0 wrong remote-call 1 100 0 0 0 2 correct message 0 0 1 1 0 3 correct single column name changed: 3/5 edges broke (loan, billing, reporting); owners to coordinate with 2/3 ``` #### Latency, Load, and Freshness The first three columns show what the patterns look like on the surface. Remote call goes to the source on every query: one round trip per query, and a hundred requests for a hundred queries. In the other three patterns the query round trip is zero, because the answer comes from the consumer's own side; the source system never sees any of the hundred queries. The freshness column shows what this gain costs. In shared database and remote call, a change made at the source shows up in the consumer at the same instant; both patterns have a freshness window of zero ticks. In file transfer and messaging, a change is visible only after one refresh period. The difference is in the fourth column: even though only one copy's status changed, file transfer moves all two hundred records on every refresh, while messaging moves one record. The same freshness window is bought with a two-hundred-fold difference in transfer volume. This is also where the work that falls on the source system comes from: file transfer produces zero requests per query but reads the table from start to end on every period. No single ranking follows from this alone. Remote call zeroes the freshness window and ties the source's load to the number of queries; file transfer ties load to the number of periods and sacrifices freshness; messaging stays tied to the number of periods but cuts transfer volume down to the number of changes. Shared database looks cheapest of all: zero round trips, zero requests, zero window. Its cost is in the next two columns. #### Ownership Leak The fifth and sixth columns measure the same edge from the ownership side. In the shared database and file transfer patterns, consumer code carries three names from the source's physical schema: the table's name, the column's name, and the value that column is meaningful for. The third name is the most expensive, because it is not a name — it is a rule. The fact that the value "shelved" means available is the catalog unit's business rule, and that rule sits in the consumer's code as a second copy. If the catalog adds a third status value tomorrow, the consumer's answer goes wrong even though the source code never changed. In remote call and messaging, the physical name count is zero. The consumer does not read the source's table, it reads the format the source hands outward; the availability decision is computed in exactly one place in the source's code. The names in the sixth column are not a loss — it is a contract: the source system has committed to not changing those names, and in exchange it can do whatever it wants to its table. Shared database's real problem shows up in these rows. The table carries a single system's name, but two systems' code depends on it. The schema has no single owner; the catalog unit only learns it cannot change the table on its own once the side that built the edge notices the change. #### The Spread of Change The last column measures this. The source system renames a single column; it does nothing beyond keeping its own published format intact. The remote call and messaging edges are entirely unaffected by this change, because both read from the published format. The shared database and file transfer edges start answering incorrectly — silently, too, because the missing field's counterpart is not an error, it is an undefined value. The last line carries this result out to the whole enterprise. Three of the five edges out of the catalog break, and behind the broken ones stand two separate owners. For the source system's owner, this means a one-line migration first requires reconciling schedules with two units, then waiting for three systems to adapt. Pattern choice stops being a latency preference here and turns into a coordination debt: whichever pattern is chosen, the data gets across, but how many owners' permission is needed before it can change depends on the pattern. #### Summary - The same edge was built with four patterns: remote call measured 1 round trip per query and 100 source requests for a hundred queries; the other three patterns measured 0 round trips and 0 requests. - The freshness window is 0 ticks for shared database and remote call, and 1 tick for file transfer and messaging; the same window is bought by moving 200 records in file transfer and 1 record in messaging. - Shared database and file transfer carry 3 names from the source's physical schema into consumer code; one of them is not a column name, but a second copy of the availability rule. - Remote call and messaging carry no physical names; instead they bind to 2 and 3 published names, respectively, that the source has committed to. - When the source renames a single column, 3 of the 5 edges silently start answering incorrectly, and coordination is needed with 2 of the 3 owners. #### Next Step The measurement was done with a single source and a single consumer; in the enterprise model there are five edges out of the catalog, and the catalog is not the enterprise's only system. It was never asked how the edge count grows with the number of systems when every system connects to every other system with its own chosen pattern, how much work adding a new system creates for how many parties, and what routing these edges through one common point makes cheaper and what it collects in a single place. The next lesson builds point-to-point edges and a central point side by side in the same enterprise model, and counts the edge count, the cost of a new system, how many edges break the instant the hub goes down, and who owns the business logic that accumulates at the hub. ### The Enterprise Service Bus Source: https://academia.sh/en/courses/enterprise-context/enterprise-service-bus Building and comparing the same flow set with point-to-point edges and with a central hub: the number of translators and adapters written, how many places a rule appears, the number of messages the hub processes, the share of flows that break when the hub goes down, the number of owners touched by adding a new system, and the ownership of the business logic that piles up at the hub. The previous lesson built a single edge with four patterns and showed what one pattern costs. In the enterprise, though, every system talks to every other system by whatever path it chose. This lesson builds the same set of data flows two ways — a direct edge for each flow, or a single central point that all flows pass through — and compares the two with the same measures. **IN4 — the enterprise is fictional; the flow set is written as an array in `node`.** Every quantity measured is read either from this array, from the running processes, or from the integration code's own text. **IN5 — the receiver systems are modeled in a single process, each on its own port.** Rationale: the quantities measured (units written, hops crossed, messages processed, flows broken) are independent of whether the systems sit on separate machines; separate ports are enough to preserve the system boundary. **IN6 — the hub is not a product, it is a location.** What is measured is not which capabilities the hub carries, but what routing every flow through a single point does to edge count, failure surface, and ownership. #### The Flow Set The regional library network has six systems, nine data flows, and five separate owners. The loan service reports every loan to four systems, the membership system reports member changes to three systems, and the catalog reports copy status to two systems. Nine flows are a small subset of the thirty directed pairs six systems could form; the graph is sparse, and this sparseness determines most of the numbers that follow. ```js // enterprise.mjs — fictional enterprise model: systems, their owners, and the data flows between them export const OWNER = { catalog: "catalog unit", loan: "in-house development", billing: "in-house development", membership: "membership unit", reporting: "management unit", "branch-interface": "branch unit", reservation: "branch unit", hub: "integration unit" }; export const RECEIVER = ["billing", "reporting", "catalog", "branch-interface", "loan"]; export const FLOW = [ // [source, target]; each row is an edge ["loan", "billing"], ["loan", "reporting"], ["loan", "catalog"], ["loan", "branch-interface"], ["membership", "loan"], ["membership", "billing"], ["membership", "reporting"], ["catalog", "branch-interface"], ["catalog", "reporting"], ]; export const NEW = [ // flows born when the seventh system is added ["loan", "reservation"], ["membership", "reservation"], ["catalog", "reservation"], ["reservation", "loan"], ["reservation", "branch-interface"], ]; export const degree = (a) => FLOW.filter((b) => b.includes(a)).length; if (process.argv[1].endsWith("enterprise.mjs")) { const s = [...new Set(FLOW.flat())]; console.log(`${s.length} systems, ${FLOW.length} flows, ${new Set(s.map((a) => OWNER[a])).size} owners; ` + `possible directed pairs ${s.length * (s.length - 1)}`); console.log(`system touching the most flows: ` + `${s.map((a) => `${a} ${degree(a)}`).sort((x, y) => y.split(" ")[1] - x.split(" ")[1])[0]}`); console.log(`seventh system added: flows ${FLOW.length} -> ${FLOW.length + NEW.length}`); } ``` ``` 6 systems, 9 flows, 5 owners; possible directed pairs 30 system touching the most flows: loan 5 seventh system added: flows 9 -> 14 ``` The receiver systems sit in a single process on five separate ports and write the record they receive to their own log; the two arrangements' results are compared from these logs. ```js // systems.mjs — five receiver systems, each on its own port; ; long-lived process import { createServer } from "node:http"; import { appendFileSync } from "node:fs"; import { RECEIVER } from "./enterprise.mjs"; const [base] = process.argv.slice(2); if (!base) { console.log("usage: node systems.mjs "); process.exit(1); } RECEIVER.forEach((name, i) => createServer(async (request, response) => { let s = ""; for await (const p of request) s += p; appendFileSync(`received-${name}.log`, `${s}\n`); response.end("done"); }).listen(Number(base) + i)); ``` #### Point-to-Point Edges In the point-to-point arrangement, one translator is written for each flow, and the translator lives inside the source system. Every system has its own vocabulary: the loan service says `memberId`, the membership system says `member_no`, billing says `memberNo`. The translator closes this gap and produces the fields the target expects. Four rules (late fee, report period, branch region, member class) are called inside the translators. ```js // point-to-point.mjs — point-to-point integration: one translator per flow; the translator lives in the source system export const lateFee = (days) => days * 5; // late fee rule export const period = (time) => time.slice(0, 7); // report period rule export const region = (code) => (code <= 2 ? "north" : "south"); // branch region rule export const memberClass = (statusCode) => (statusCode === 1 ? "full" : "restricted"); // member class rule export const TRANSLATOR = { "loan->billing": (o) => ({ memberNo: o.memberId, copy: o.copyId, debt: lateFee(o.daysLate) }), "loan->reporting": (o) => ({ event: "loan", key: o.copyId, period: period(o.time) }), "loan->catalog": (o) => ({ copyId: o.copyId, available: false }), "loan->branch-interface": (o) => ({ copy: o.copyId, region: region(o.branchCode) }), "membership->loan": (o) => ({ memberId: o.member_no, memberClass: memberClass(o.status_code) }), "membership->billing": (o) => ({ memberNo: o.member_no, memberClass: memberClass(o.status_code) }), "membership->reporting": (o) => ({ event: "member", key: o.member_no, period: period(o.time) }), "catalog->branch-interface": (o) => ({ copy: o.copyId, region: region(o.branchCode) }), "catalog->reporting": (o) => ({ event: "copy", key: o.copyId, period: period(o.time) }), }; ``` #### The Hub In the hub arrangement, the producing system hands its own format to a single place. The hub converts that format to the common format, finds the targets by looking at the flow table, and for each target converts to and sends that system's format. There are two adapters per system: inbound and outbound. The four rules now live in a single place, and the `RULE` array records who actually owns each rule's work. ```js // hub.mjs — the central point; ; long-lived process import { createServer } from "node:http"; import { FLOW, RECEIVER } from "./enterprise.mjs"; const [port, base] = process.argv.slice(2); if (!base) { console.log("usage: node hub.mjs "); process.exit(1); } const lateFee = (days) => days * 5, period = (time) => time.slice(0, 7); // rules collected at the hub const region = (code) => (code <= 2 ? "north" : "south"), memberClass = (statusCode) => (statusCode === 1 ? "full" : "restricted"); export const RULE = [ // rule at the hub and the actual owner of the work { name: "lateFee", owner: "in-house development" }, { name: "period", owner: "management unit" }, { name: "region", owner: "branch unit" }, { name: "memberClass", owner: "membership unit" }, ]; const INBOUND = { // system format -> common format loan: (o) => ({ type: "loan", member: o.memberId, copy: o.copyId, branch: o.branchCode, late: o.daysLate, time: o.time }), membership: (o) => ({ type: "member", member: o.member_no, status: o.status_code, time: o.time }), catalog: (o) => ({ type: "copy", copy: o.copyId, branch: o.branchCode, time: o.time }), }; const OUTBOUND = { // common format -> system format billing: (c) => (c.type === "loan" ? { memberNo: c.member, copy: c.copy, debt: lateFee(c.late) } : { memberNo: c.member, memberClass: memberClass(c.status) }), reporting: (c) => ({ event: c.type, key: c.copy ?? c.member, period: period(c.time) }), catalog: (c) => ({ copyId: c.copy, available: false }), "branch-interface": (c) => ({ copy: c.copy, region: region(c.branch) }), loan: (c) => ({ memberId: c.member, memberClass: memberClass(c.status) }), }; let incoming = 0, outgoing = 0; // number of messages the hub has processed createServer(async (request, response) => { if (request.url === "/counts") { response.end(`${incoming} ${outgoing}`); return; } if (request.url === "/shutdown") { response.end("closed"); process.exit(0); } let s = ""; for await (const p of request) s += p; const { source, body } = JSON.parse(s); incoming += 1; const common = INBOUND[source](body); for (const t of FLOW.filter((b) => b[0] === source).map((b) => b[1])) { outgoing += 1; await fetch(`http://127.0.0.1:${Number(base) + RECEIVER.indexOf(t)}/receive`, { method: "POST", body: JSON.stringify(OUTBOUND[t](common)) }); } response.end("done"); }).listen(Number(port)); ``` #### Measurement The tool runs the same six events first through the point-to-point arrangement, then through the hub arrangement, and compares the receiver logs. The number of units written and the number of places a rule appears are read from the text of the two integration files, not from the running processes. In the final step the hub is shut down and the same six events are sent again. ```js // measure.mjs — measures two arrangements on the same flow set; import { existsSync, readFileSync, rmSync } from "node:fs"; import { FLOW, RECEIVER, OWNER, NEW } from "./enterprise.mjs"; import { TRANSLATOR } from "./point-to-point.mjs"; const [base, hp] = process.argv.slice(2); if (!hp) { console.log("usage: node measure.mjs "); process.exit(1); } const EVENT = [ // events in the producing systems' own formats ["loan", { memberId: 7, copyId: 401, branchCode: 1, daysLate: 3, time: "2026-04-11" }], ["loan", { memberId: 9, copyId: 402, branchCode: 3, daysLate: 0, time: "2026-04-12" }], ["membership", { member_no: 7, status_code: 1, time: "2026-04-11" }], ["membership", { member_no: 9, status_code: 2, time: "2026-04-13" }], ["catalog", { copyId: 401, branchCode: 1, time: "2026-04-11" }], ["catalog", { copyId: 403, branchCode: 4, time: "2026-04-14" }], ]; const DELIVERIES = FLOW.length * 2; // each flow carries two events const clear = () => RECEIVER.forEach((a) => rmSync(`received-${a}.log`, { force: true })); const collect = () => RECEIVER.flatMap((a) => existsSync(`received-${a}.log`) ? readFileSync(`received-${a}.log`, "utf8").split("\n").filter(Boolean).map((s) => `${a} ${s}`) : []); const runPointToPoint = async () => { // the translator runs in the source system, sends directly for (const [source, native] of EVENT) { for (const t of FLOW.filter((b) => b[0] === source).map((b) => b[1])) { await fetch(`http://127.0.0.1:${Number(base) + RECEIVER.indexOf(t)}/receive`, { method: "POST", body: JSON.stringify(TRANSLATOR[`${source}->${t}`](native)) }); } } }; const runHub = async () => { // the producer hands only its own format to the hub let accepted = 0; for (const [source, native] of EVENT) { try { await fetch(`http://127.0.0.1:${hp}/event`, { method: "POST", body: JSON.stringify({ source, body: native }) }); accepted += 1; } catch { accepted += 0; } // producer cannot deliver while the hub is down } return accepted; }; clear(); await runPointToPoint(); const a = collect().sort(); clear(); await runHub(); const b = collect().sort(); const [incoming, outgoing] = (await (await fetch(`http://127.0.0.1:${hp}/counts`)).text()).split(" ").map(Number); const hubSrc = readFileSync("hub.mjs", "utf8"), p2pSrc = readFileSync("point-to-point.mjs", "utf8"); const section = (m, name) => m.split(`const ${name} = {`)[1].split("\n};")[0]; const unitCount = (m) => m.match(/^ {2}"?[\w>-]+"?: \(/gm).length; // units written per pattern const calls = (m) => ["lateFee", "period", "region", "memberClass"] // number of places the rule appears .reduce((s, r) => s + (m.match(new RegExp(`\\b${r}\\(`, "g")) || []).length, 0); const rule = hubSrc.split("RULE = [")[1].split("];")[0].match(/owner: "([^"]+)"/g); const setSize = (l) => new Set(l).size; const HUB_UNITS = ["reservation", "hub"]; // units touched in the hub arrangement const row = (name, x, y) => console.log(`${name.padEnd(41)}${String(x).padStart(7)}${String(y).padStart(9)}`); row("measure", "p2p", "hub"); row("flow (edge) count", FLOW.length, FLOW.length); row("translators / adapters written", unitCount(section(p2pSrc, "TRANSLATOR")), unitCount(section(hubSrc, "INBOUND")) + unitCount(section(hubSrc, "OUTBOUND"))); row("places the rule appears", calls(p2pSrc), calls(hubSrc)); row("hops per delivery", 1, 2); row("messages the busiest unit handles", 2, incoming + outgoing); row("flows broken if one unit goes down", Math.max(...FLOW.flat() .map((s) => FLOW.filter((b) => b.includes(s)).length)), FLOW.length); row("seventh system: units to write", NEW.length, HUB_UNITS.length); row("seventh system: owners touched", setSize(NEW.map((b) => OWNER[b[0]])), setSize(HUB_UNITS.map((s) => OWNER[s]))); row("rules at the hub owned by someone else", 0, rule.filter((s) => !s.includes(OWNER.hub)).length); console.log(`both arrangements' deliveries: ${a.length}/${b.length} records, ` + `identical: ${JSON.stringify(a) === JSON.stringify(b) ? "yes" : "no"}`); await fetch(`http://127.0.0.1:${hp}/shutdown`).catch(() => 0); // the hub is shut down clear(); const accepted = await runHub(); const downCount = collect().length; clear(); await runPointToPoint(); console.log(`hub down: ${accepted}/${EVENT.length} events accepted, ` + `${downCount}/${DELIVERIES} delivered; point-to-point delivered ${collect().length}/${DELIVERIES} at the same time`); ``` ```bash node systems.mjs 8991 & S=$! node hub.mjs 8990 8991 & sleep 0.8 node measure.mjs 8991 8990 kill $S ``` ``` measure p2p hub flow (edge) count 9 9 translators / adapters written 9 8 places the rule appears 8 5 hops per delivery 1 2 messages the busiest unit handles 2 24 flows broken if one unit goes down 5 9 seventh system: units to write 5 2 seventh system: owners touched 4 2 rules at the hub owned by someone else 0 4 both arrangements' deliveries: 18/18 records, identical: yes hub down: 0/6 events accepted, 0/18 delivered; point-to-point delivered 18/18 at the same time ``` #### How Much the Edge Count Drops The comparison before the last line establishes the measurement's validity: both arrangements deliver eighteen records, and the delivered records are exactly identical. The difference is not in the outcome of the work, but in where the work sits. The second row does not give the expected payoff. Nine translators become eight adapters for nine flows; the gain is a single unit. The classic justification for hub-based integration is that edge count grows with the square of the system count, but that justification rests on the assumption that every system talks to every other system. In this enterprise, only nine of the thirty directed pairs six systems could form actually exist. Translator count grows with the number of flows, adapter count grows with the number of systems; as long as the graph is sparse the two stay close together, and the hub arrangement's advantage on this row is small. The real gain is in the third row. The same four rules are called eight times in the point-to-point arrangement, and five times at the hub. The report period rule appears in three separate translators, because all three systems that send data to reporting have to do that calculation in their own code. If the period definition changes, the point-to-point arrangement requires three separate systems to change; the hub arrangement requires one unit. What the hub sells is not a lower edge count, but the removal of repetition. #### The Hub's Cost The fifth row turns the bottleneck into a number. For six events the hub processes twenty-four messages: six inbound, eighteen outbound. In the point-to-point arrangement, the busiest single edge processes two messages. The hub does twelve times the work of the busiest point-to-point edge, and that ratio grows with the number of flows; on top of that, it adds one hop per delivery. The sixth row gives the failure surface. In the point-to-point arrangement, the system touching the most flows is the loan service, and when it goes down, five of the nine flows break. When the hub goes down, all nine of nine break. The output's last line shows this by measurement: when the hub is shut down, none of the six events are accepted and zero of eighteen deliveries happen; at the same time, the point-to-point arrangement completes eighteen of eighteen deliveries. When the loan service goes down, the enterprise has already stopped, because it cannot lend books anyway; when the hub goes down, a component that does no business work of its own stops the entire enterprise. This is the unsettling part of a single point of failure: the component with zero business value has the highest share of failure. #### A New System and the Logic That Piles Up at the Hub The seventh and eighth rows are the hub arrangement's strongest defense. Adding the reservation system creates five new flows. In the point-to-point arrangement, five translators are written for these five flows, and because the translators live inside the source systems, the loan, membership, catalog, and reservation code all get touched: schedules have to be reconciled with four separate owners. In the hub arrangement only two units are written — reservation's inbound and outbound adapters — and the number of owners touched drops to two. The new system's cost decouples from the system count. The last row shows what this costs in return. Four rules sit at the hub, and all four have a different actual owner: late fee is billing's business, report period is management's, region definition is the branch unit's, member class is membership's. The integration unit that owns the hub has none of the expertise to write any of these rules, but it carries all of their code. This has two consequences. First, when the member class definition changes, the change ships in the hub's release, not the membership unit's own release; second, that release redeploys all nine flows at once. As the rule's repetition disappears, the rule's ownership blurs, and the deployment units become coupled to each other. The choice depends on how tight the flow graph is: the hub lowers repetition and the coordination cost of a new system, and in exchange adds one hop, a shared point of failure, and a pile of rules with unclear ownership. #### Summary - The same nine flows were built in both arrangements, and both delivered the same 18 records with exactly identical content; the comparison was built on top of this verification. - The number of units written is 9 versus 8; since only 9 of six systems' 30 possible directed pairs exist, the hub's gain in edge count stayed at a single unit. - The same four rules appear in 8 places in the point-to-point arrangement and 5 at the hub; the report period rule is repeated in three separate systems. - The hub processes 24 messages for six events, versus 2 for the busiest point-to-point edge; when the hub is shut down, 0 of 18 deliveries happen while the point-to-point arrangement completes all 18 at the same time. - The seventh system costs 5 units and 4 owners in the point-to-point arrangement, and 2 units and 2 owners in the hub arrangement; in exchange, 4 rules owned by someone else pile up at the hub. #### Next Step Across two lessons, data always flowed in one direction: one system wrote, the others read. The source was fixed, the edge's direction was fixed, and there was no such thing as conflict. In the enterprise, this assumption does not hold for most entities. A member record lives in the membership system, in the loan service, and in the municipality's identity service; all three are written from their own user interface, and all three carry different values in the address field for the same member. The next lesson holds the same entity in three systems and counts: how many systems write, how many fields conflict, how many records a reconciliation round fixes, and how many inconsistencies remain; how these numbers change when one system is chosen as the source, and what that choice lengthens in the write path. ### Master Data Management Source: https://academia.sh/en/courses/enterprise-context/master-data-management Measuring the same member record held in three systems: how many systems write it, the number of conflicting field pairs, the fields corrected and the inconsistency left after a reconciliation round, how these numbers change when a golden record is chosen, and the write-step and availability cost the choice imposes on the write path. Across two lessons, data always flowed in one direction: one system wrote, the others read. The source was fixed, the edge's direction was fixed, and there was no such problem as two copies carrying a different truth. In the enterprise, this assumption does not hold for most entities. A member record lives in the membership system, in the loan service, and in the municipality's identity service; all three are written from their own screen, and the same member's address can carry three different values in three places. This lesson builds that situation and counts it: how many systems write, how many fields conflict, how many values a reconciliation round fixes, and what is left over. Then one of the systems is chosen as the system of record, the same event stream is run again, and the same numbers are read again. **IN7 — the enterprise is fictional; the systems, their fields, owners, and write events are a data structure in `node`.** Randomness is generated from a single visible seed, and the generator is inside the code. **IN8 — replication mechanics are not modeled.** Rationale: keeping copies, replication lag, and eventual consistency were measured in a separate course. What is measured here is not how the data gets copied, but who writes it and the fact that two copies carry a different truth; a write is treated as visible the instant it happens. **IN9 — the reconciliation rule is modeled as "last writer wins."** Rationale: the quantity measured is not the rule itself, but the surface the rule can be applied to. In a system that cannot be written from outside, no correction can be written no matter which rule is chosen; changing the rule only changes the winner, not where the remaining inconsistency sits. #### The Same Entity, Three Systems Three systems hold the same member, and the three have different field sets. The membership system carries four fields, the loan service three, the municipality's identity service two. All three have a different owner, and all three have their own users. The determining difference is not in the field count, but in the last column: the identity service cannot be written from outside. The enterprise can read that system, but it cannot send it a correction. ```js // enterprise.mjs — the three systems holding the member record: fictional enterprise model and event generator export const SYSTEM = [ { name: "membership", owner: "membership unit", externallyWritable: 1, field: ["name", "address", "phone", "email"] }, { name: "loan", owner: "in-house development", externallyWritable: 1, field: ["name", "address", "phone"] }, { name: "identity", owner: "municipal IT", externallyWritable: 0, field: ["name", "address"] }, ]; export const FIELD = ["name", "address", "phone", "email"]; export const MEMBERS = 120, WRITES = 6, SEED = 20260411; // member count, writes per member, visible seed const makeRng = (t) => () => ((t = (t * 1103515245 + 12345) % 2147483648) / 2147483648); // each system's own user writes from their own screen; events are generated from a single seed export const events = () => { const r = makeRng(SEED), o = []; for (let u = 1; u <= MEMBERS; u += 1) { for (let w = 1; w <= WRITES; w += 1) { const s = SYSTEM[Math.floor(r() * SYSTEM.length)]; const f = s.field[Math.floor(r() * s.field.length)]; o.push({ member: u, system: s.name, field: f, value: `d${Math.floor(r() * 4)}`, time: w }); } } return o; }; if (process.argv[1].endsWith("enterprise.mjs")) { const O = events(); console.log(`${SYSTEM.length} systems hold the same entity, ` + `${new Set(SYSTEM.map((s) => s.owner)).size} distinct owners; ` + `externally writable ${SYSTEM.filter((s) => s.externallyWritable).length}`); for (const s of SYSTEM) { console.log(` ${s.name.padEnd(12)}${String(s.field.length).padStart(2)} fields ` + `${String(O.filter((o) => o.system === s.name).length).padStart(3)} writes ${s.owner}`); } console.log(`total ${O.length} writes, ${MEMBERS} members, ${FIELD.length} fields`); } ``` ``` 3 systems hold the same entity, 3 distinct owners; externally writable 2 membership 4 fields 224 writes membership unit loan 3 fields 258 writes in-house development identity 2 fields 238 writes municipal IT total 720 writes, 120 members, 4 fields ``` #### Two Write Arrangements In the distributed arrangement, a write lands wherever it came from and stays there. In the golden record arrangement, every write that comes from an externally writable system's screen goes to the system of record first, then spreads out to the other writable systems; a write coming from the municipality's own screen stays outside this arrangement. The reconciliation round runs with the same function in both arrangements: for every conflicting field, the most recently written value is found and applied to the writable systems. ```js // arrangement.mjs — two write arrangements and a reconciliation round; store format: store[system][member][field] import { SYSTEM } from "./enterprise.mjs"; const find = (s) => SYSTEM.find((x) => x.name === s); export const empty = () => Object.fromEntries(SYSTEM.map((s) => [s.name, {}])); export const put = (store, sys, member, field, value, time) => { // if it is not carried, the write step is not spent if (!find(sys).field.includes(field)) return 0; (store[sys][member] ??= {})[field] = { value, time }; return 1; }; // distributed arrangement: a write lands wherever the system's screen it came from export const distributed = (store, o, down) => (down.includes(o.system) ? { step: 0, first: null } : { step: put(store, o.system, o.member, o.field, o.value, o.time), first: o.system }); // golden record arrangement: every externally writable write goes to the system of record first, then spreads out export const golden = (store, o, down, recordSystem = "membership") => { if (!find(o.system).externallyWritable) return distributed(store, o, down); if (down.includes(recordSystem)) return { step: 0, first: null }; let step = put(store, recordSystem, o.member, o.field, o.value, o.time); for (const s of SYSTEM.filter((x) => x.externallyWritable && x.name !== recordSystem)) { step += put(store, s.name, o.member, o.field, o.value, o.time); } return { step, first: recordSystem }; }; // conflict: more than one different value stands across systems for the same (member, field) export const conflict = (store, members, fields) => members.flatMap((m) => fields.flatMap((f) => { const v = SYSTEM.map((s) => store[s.name][m]?.[f]).filter(Boolean); return new Set(v.map((x) => x.value)).size > 1 ? [{ m, f }] : []; })); // reconciliation round: the last writer wins, and the winner is applied to the writable systems export const reconcile = (store, c) => { let corrected = 0; for (const { m, f } of c) { const winner = SYSTEM.map((s) => ({ ...store[s.name][m]?.[f] })).filter((x) => x.value) .sort((x, y) => y.time - x.time)[0]; for (const s of SYSTEM.filter((x) => x.externallyWritable)) { const now = store[s.name][m]?.[f]; if (now && now.value !== winner.value) { put(store, s.name, m, f, winner.value, winner.time); corrected += 1; } } } return corrected; }; ``` #### Measurement The tool runs seven hundred twenty write events first through the distributed arrangement, then through the golden record arrangement. After each run, conflicting field pairs are counted, a reconciliation round runs, and the conflict is counted again afterward. The final measure repeats the same event stream with the membership system down and counts how many writes are accepted. ```js // measure.mjs — runs the same event stream through both arrangements and counts the inconsistency import { FIELD, SYSTEM, MEMBERS, events } from "./enterprise.mjs"; import { golden, empty, conflict, distributed, reconcile } from "./arrangement.mjs"; const EVENTS = events(), MEMBER_IDS = [...Array(MEMBERS).keys()].map((i) => i + 1); const run = (write, down = []) => { const store = empty(), writers = new Set(); let step = 0, accepted = 0; for (const o of EVENTS) { const { step: n, first } = write(store, o, down); step += n; if (first) { accepted += 1; writers.add(first); } } return { store, step, accepted, writers: writers.size }; }; const measure = (write) => { const t = run(write); const before = conflict(t.store, MEMBER_IDS, FIELD); const corrected = reconcile(t.store, before); const after = conflict(t.store, MEMBER_IDS, FIELD); return { ...t, before, corrected, after, down: run(write, ["membership"]).accepted }; }; const DIST = measure(distributed), GOLD = measure(golden); const row = (name, x, y) => console.log(`${name.padEnd(45)}${String(x).padStart(13)}${String(y).padStart(15)}`); row("measure", "distributed", "golden record"); row("systems receiving user writes", DIST.writers, GOLD.writers); row("write steps per event", (DIST.step / EVENTS.length).toFixed(2), (GOLD.step / EVENTS.length).toFixed(2)); row("conflicting (member, field) pairs", DIST.before.length, GOLD.before.length); row("fields corrected in reconciliation", DIST.corrected, GOLD.corrected); row("inconsistencies remaining after reconciliation", DIST.after.length, GOLD.after.length); row("writes accepted while membership is down", `${DIST.down}/${EVENTS.length}`, `${GOLD.down}/${EVENTS.length}`); console.log(); console.log(`${"field".padEnd(9)}${"carriers".padStart(9)}${"dist. conflict".padStart(16)}` + `${"dist. remaining".padStart(17)}${"golden conflict".padStart(17)}${"golden remaining".padStart(18)}`); for (const f of FIELD) { const n = (l) => l.filter((x) => x.f === f).length; console.log(`${f.padEnd(9)}${String(SYSTEM.filter((s) => s.field.includes(f)).length).padStart(9)}` + `${String(n(DIST.before)).padStart(16)}${String(n(DIST.after)).padStart(17)}` + `${String(n(GOLD.before)).padStart(17)}${String(n(GOLD.after)).padStart(18)}`); } ``` ``` measure distributed golden record systems receiving user writes 3 2 write steps per event 1.00 1.61 conflicting (member, field) pairs 131 95 fields corrected in reconciliation 95 94 inconsistencies remaining after reconciliation 48 48 writes accepted while membership is down 496/720 238/720 field carriers dist. conflict dist. remaining golden conflict golden remaining name 3 62 32 53 32 address 3 54 16 42 16 phone 2 15 0 0 0 email 1 0 0 0 0 ``` #### Where Conflict Comes From The first column of the second table is explanatory on its own. The email field lives only in the membership system and never conflicts in either arrangement. Phone lives in two systems, address and name in three; the conflict counts follow the same order. Conflict is born not from a field's importance, how often it gets filled, or data quality, but from how many systems write that field. For four fields across a hundred twenty members, the distributed arrangement measures a hundred thirty-one conflicting pairs; all of them are in the three fields more than one system writes. The reconciliation round corrects ninety-five field values out of a hundred thirty-one conflicts, and forty-eight conflicts remain. Where the remainder falls is in the fourth column of the second table: thirty-two in name, sixteen in address. These two fields are exactly the two fields the identity service carries. All fifteen conflicts in the phone field are corrected, because both systems that carry phone are writable. The source of the remaining inconsistency is not weakness in the reconciliation rule. The last-writer-wins rule determines a winner in every conflict; the problem is that the winner cannot be written to the other systems. When the winning value turns out different from the value in the municipality's system, the enterprise cannot correct that system. The number forty-eight is not a measure of data quality — it is a measure of an authority boundary. #### What the Golden Record Changes When the membership system is chosen as the system of record, the table changes in four places. The number of systems receiving user writes drops from three to two — two, not three, because the third writer sits outside the enterprise's authority boundary. The conflicting-pair count drops from a hundred thirty-one to ninety-five. Conflict in the phone field disappears entirely: since the two writable systems are now fed from the same write path, that field can no longer produce two different truths. Conflict in the name and address fields drops but does not reach zero — from sixty-two to fifty-three, and from fifty-four to forty-two. All of the remaining conflict sits between the enterprise's own copies and the municipality's copy. The fifth row is this lesson's harshest number: the remaining inconsistency is exactly forty-eight in both arrangements. This is not a coincidence — it is a structural consequence of the mechanism. Reconciliation's winner is the most recently written value for that field; which screen it came from and which system it landed in does not change the winner. Whenever the winner did not come from the identity service, that service's copy stays wrong, and because it cannot be written from outside, it cannot be corrected. The golden record choice does not reduce even one of these forty-eight conflicts. This is where master data management's real limit shows up. A golden record produces consistency on the surface that write authority reaches. For every copy that sits outside that surface, the only thing it can produce is knowing where the inconsistency is. What can be done for the forty-eight records is not to correct them, but to make them reportable and to write down, in the enterprise's own decisions, which copy to trust. #### What the Choice Costs The second and sixth rows give the cost. Write steps per event go from 1.00 to 1.61. The increase is not twofold, because only the fields more than one writable system carries get spread out; email writes stay at a single step. The write path's lengthening also comes back as a measurable delay: the user now writes not to their own system's database, but to another unit's system, and waits for that write to succeed. The sixth row is a harsher form of this. With the membership system down, four hundred ninety-six of seven hundred twenty writes are accepted in the distributed arrangement, while only two hundred thirty-eight are accepted in the golden record arrangement — and those are only the ones coming from the municipality's own screen. All of the enterprise's member writes are now tied to a single unit's system. In the distributed arrangement, the membership system being down did not stop the loan service from updating a member's address; in the golden record arrangement, it does. The cost paid is not only technical. Choosing a system of record turns one unit's system into a precondition for other units' operation: availability target, maintenance window, and release calendar stop being decisions that unit can make on its own. At enterprise scale, the golden record decision looks like a data decision, but what it pays for is a dependency and a coordination obligation. #### Summary - When the same member record is held in three systems, 720 write events produce 131 conflicting (member, field) pairs in the distributed arrangement; conflict is born only in fields more than one system writes. - The email field, which lives in a single system, never conflicts in either arrangement; the phone field, which lives in two systems, drops from 15 conflicts to 0 in the golden record arrangement. - A reconciliation round corrects 95 field values in the distributed arrangement and 94 in the golden record arrangement; the remaining inconsistency is 48 in both arrangements, and all of it sits in the two fields the non-externally-writable system carries. - The golden record choice drops conflict from 131 to 95 but does not reduce the remaining inconsistency by even one: correction only works on the surface write authority reaches. - The choice costs a rise in write steps per event from 1.00 to 1.61, and a drop in writes accepted while the system of record is down from 496/720 to 238/720. #### Next Step Across three lessons, every measurement stayed on the operating path: a record was written, passed through an edge, was read in a system. The management unit, though, does not ask about any of these one at a time; it wants the period's loan count, the per-branch late rate, and membership movements together, and answering these questions requires laying three separate systems' data side by side. Running these questions on the operating systems was never measured: the load a reporting query puts on the source system, bringing three systems' different field names into a single resolved format, and the delay that transformation adds to the result's freshness. The next lesson separates the resolution load from the operating path and counts these three quantities. ### The Data Warehouse and Transformation Pipelines Source: https://academia.sh/en/courses/enterprise-context/data-warehouse-and-transformation-pipelines Separating analytical load from operational load: five reports imposing 163 queries and 5,354,308 records read per day on the operational system, the warehouse taking on that load in exchange for a freshness lag, an eight-step transformation pipeline moving 523,341 records in one cycle, and a single field change in the source schema breaking one step and stopping four more, dropping all five of the five reports. The previous lesson dealt with a case where multiple systems wrote the same entity: a source system was chosen and the inconsistency between records was counted. There, every party was **writing** the data. In this lesson there is a party that never writes at all. The regional library network's management unit only asks questions: which branch is filling up, which item is overdue, how membership is flowing. A question does not change the data, but it is not free either; every question becomes a query, every query becomes records read, and those records sit on the disk of the system that runs loan transactions. This lesson measures the separation of analytical load from operational load. The network is a **fictional model** whose owners and budgets are separate. #### The Questioning Party Brings a Countable Load The model has four sources, each with a separate owner: the loan service, the membership system, the externally sourced catalog, and the branch systems. The management unit's five questions read these. **IN10: the record counts, run frequencies, and freshness requirements are fictional** and are not taken from any real institution. **IN11: a report touching one entity is one query; the event entity is read by window, the dimension entity is scanned in full.** **IN14: a step that cannot find the field it reads breaks; a step whose input is not intact stops.** ```js // warehouse/model.mjs — the regional library network's fictional model and transformation pipeline. // Records are actually generated; the numbers in the next blocks are measured by scanning these arrays. export const SEED = 7391, DAYS = 180, END = DAYS * 24 - 1; export const OWNER = { loan: "loan service (in-house)", member: "membership system", item: "catalog (external)", branch: "branch systems" }; export function generate(t = SEED) { let s = t; const r = () => (s = (s * 48271) % 2147483647) / 2147483647, n = (k) => Math.floor(r() * k); const array = (k, f) => Array.from({ length: k }, (_, i) => f(i)); return { branch: array(12, (i) => ({ id: i, region: `b${i % 4}` })), item: array(128_000, (i) => ({ id: i, type: ["book", "periodical", "audio"][n(3)], changeStamp: n(END) })), member: array(46_000, (i) => ({ id: i, regDay: n(DAYS), branchId: n(12), changeStamp: n(END) })), loan: array(240_000, (i) => ({ id: i, memberId: n(46_000), itemId: n(128_000), branchId: r() < 0.008 ? null : n(12), day: n(DAYS), hour: n(24) })), }; } // Reports: entity -> window (days) or full scan; runs per day; largest accepted lag (hours); // basis in the warehouse; field that determines breakage. export const REPORTS = { occupancy: { reads: { loan: 1, branch: "full" }, runsPerDay: 48, freshness: 1, basis: "branch-summary", field: "loan.branchId" }, "overdue-item": { reads: { loan: 30, item: "full" }, runsPerDay: 24, freshness: 4, basis: "enriched", field: "item.type" }, "new-member": { reads: { member: "full" }, runsPerDay: 12, freshness: 14, basis: "member-summary", field: "member.regDay" }, activity: { reads: { loan: 90, member: "full" }, runsPerDay: 2, freshness: 36, basis: "member-summary", field: "member.regDay" }, turnover: { reads: { loan: 180, item: "full", branch: "full" }, runsPerDay: 1, freshness: 72, basis: "enriched", field: "item.type" }, }; // Records read and queries opened in a single run. export function readReport(data, r) { let recordsRead = 0, queries = 0; for (const [entity, window] of Object.entries(r.reads)) { queries += 1; if (window === "full") { recordsRead += data[entity].length; continue; } for (const k of data[entity]) if (k.day >= DAYS - window) recordsRead += 1; } return { recordsRead, queries }; } // The slice pulled in one cycle: from the event records within the period, from the dimension // either everything or records whose change stamp falls within the period (IN13). export function extract(data, period, dimension) { const threshold = END - period; const changed = (d) => dimension === "full" ? d : d.filter((k) => k.changeStamp > threshold); const slice = { loan: data.loan.filter((k) => k.day * 24 + k.hour > threshold), member: changed(data.member), item: changed(data.item), branch: data.branch }; return { slice, recordsRead: Object.values(slice).reduce((t, d) => t + d.length, 0), queries: 4 }; } // The pipeline's eight steps; each step declares the field it reads. export const STEPS = [ { name: "extract-loan", source: "loan", inputs: [], reads: ["memberId", "itemId", "branchId", "day"] }, { name: "extract-member", source: "member", inputs: [], reads: ["regDay", "branchId"] }, { name: "extract-item", source: "item", inputs: [], reads: ["type"] }, { name: "extract-branch", source: "branch", inputs: [], reads: ["region"] }, { name: "clean", inputs: ["extract-loan"], reads: ["branchId", "day"] }, { name: "enriched", inputs: ["clean", "extract-item", "extract-branch"], reads: ["itemId", "type", "region"] }, { name: "branch-summary", inputs: ["enriched"], reads: ["region", "day"] }, { name: "member-summary", inputs: ["clean", "extract-member"], reads: ["memberId", "regDay"] }, ]; const group = (d, f) => [...new Set(d.map(f))].map((a) => ({ a })); const TRANSFORM = { clean: ([l]) => l.filter((k) => k.branchId !== null), enriched: ([l, i, b]) => { const t = new Map(i.map((k) => [k.id, k.type])), r = new Map(b.map((k) => [k.id, k.region])); return l.map((k) => ({ ...k, type: t.get(k.itemId) ?? "unknown", region: r.get(k.branchId) ?? "unknown" })); }, "branch-summary": ([l]) => group(l, (k) => `${k.region}|${k.day}`), "member-summary": ([l]) => group(l, (k) => `${k.memberId}|${Math.floor(k.day / 30)}`), }; // Run the pipeline: a step whose transform throws is "broken", a step whose input is not intact is "stopped". export function run(slice) { const output = new Map(), status = new Map(), counts = new Map(); let moved = 0; for (const s of STEPS) { if (s.inputs.some((d) => status.get(d) !== "done")) { status.set(s.name, "stopped"); continue; } const feed = s.source ? [slice[s.source]] : s.inputs.map((d) => output.get(d)); const sample = feed.filter((d) => d.length > 0).map((d) => d[0]); try { for (const field of s.reads) if (sample.every((k) => (field in k) === false)) throw new Error(`missing field ${field}`); const result = s.source ? feed[0] : TRANSFORM[s.name](feed); const inCount = feed.reduce((t, d) => t + d.length, 0); output.set(s.name, result); status.set(s.name, "done"); counts.set(s.name, { inCount, outCount: result.length }); moved += inCount + result.length; } catch (e) { status.set(s.name, `broken: ${e.message}`); } } return { status, counts, moved }; } ``` #### The Warehouse Takes On the Load, in Exchange for a Lag In one cycle the warehouse reads the sources, transforms them, and leaves behind the summaries the reports will read. A report gets its answer from the warehouse only if the lag it accepts is larger than the warehouse's own lag. **IN12: the pipeline moves 4,000 records per second; the freshness lag is the sum of the period and the run duration.** **IN13: incremental dimension loading assumes an index on the change stamp.** ```js // warehouse/load.mjs — queries and records read that land on the operational system: without a warehouse and with one import { generate, REPORTS, readReport, extract, run, DAYS } from "./model.mjs"; const data = generate(); const SPEED = 4000; // IN12: the pipeline moves 4,000 records per second const measurements = Object.fromEntries(Object.entries(REPORTS).map(([name, r]) => [name, readReport(data, r)])); console.log(`model: ${Object.entries(data).map(([name, d]) => `${name} ${d.length}`).join(", ")}; ${DAYS} days`); console.log("\nreport runs/day queries/run records/run records/day freshness (hrs)"); console.log("-------------- ---------- ------------ ------------- ----------- ---------------"); let rawRead = 0, rawQueries = 0; for (const [name, r] of Object.entries(REPORTS)) { const o = measurements[name]; rawRead += o.recordsRead * r.runsPerDay; rawQueries += o.queries * r.runsPerDay; console.log(`${name.padEnd(14)} ${String(r.runsPerDay).padStart(10)} ${String(o.queries).padStart(12)} ` + `${String(o.recordsRead).padStart(13)} ${String(o.recordsRead * r.runsPerDay).padStart(11)} ${String(r.freshness).padStart(15)}`); } console.log(`without warehouse, total: ${rawQueries} queries/day, ${rawRead} records read/day`); console.log("\nperiod dimension run (s) lag in whs direct queries/day records/day % of raw"); console.log("------ ------------ --------- ------ ------- ------- ------------ ----------- ----------"); const remainingByPeriod = {}; for (const dimension of ["full", "incremental"]) for (const period of [24, 12, 3, 1]) { const c = extract(data, period, dimension); const duration = run(c.slice).moved / SPEED; const lag = period + duration / 3600, cycles = 24 / period; let queries = c.queries * cycles, recordsRead = c.recordsRead * cycles; const remaining = []; for (const [name, r] of Object.entries(REPORTS)) { if (r.freshness >= lag) continue; remaining.push(name); queries += measurements[name].queries * r.runsPerDay; recordsRead += measurements[name].recordsRead * r.runsPerDay; } remainingByPeriod[period] = remaining; console.log(`${String(period).padStart(6)} ${dimension.padEnd(12)} ${duration.toFixed(1).padStart(9)} ` + `${lag.toFixed(2).padStart(6)} ${String(5 - remaining.length).padStart(7)} ${String(remaining.length).padStart(7)} ` + `${String(Math.round(queries)).padStart(12)} ${String(Math.round(recordsRead)).padStart(11)} ` + `${`${(recordsRead / rawRead * 100).toFixed(1)}%`.padStart(10)}`); } console.log(`\nreports still operational at the 24-hour period: ${remainingByPeriod[24].join(", ")}`); console.log(`reports still operational at the 3-hour period: ${remainingByPeriod[3].join(", ")}`); ``` ``` model: branch 12, item 128000, member 46000, loan 240000; 180 days report runs/day queries/run records/run records/day freshness (hrs) -------------- ---------- ------------ ------------- ----------- --------------- occupancy 48 2 1326 63648 1 overdue-item 24 2 168293 4039032 4 new-member 12 1 46000 552000 14 activity 2 2 165808 331616 36 turnover 1 3 368012 368012 72 without warehouse, total: 163 queries/day, 5354308 records read/day period dimension run (s) lag in whs direct queries/day records/day % of raw ------ ------------ --------- ------ ------- ------- ------------ ----------- ---------- 24 full 133.4 24.04 2 3 160 4830006 90.2% 12 full 131.9 12.04 3 2 152 4451992 83.1% 3 full 130.8 3.04 4 1 128 1456912 27.2% 1 full 130.6 1.04 4 1 192 4241160 79.2% 24 incremental 3.6 24.00 2 3 160 4656902 87.0% 12 incremental 1.8 12.00 3 2 152 4104854 76.7% 3 incremental 0.4 3.00 4 1 128 65392 1.2% 1 incremental 0.1 1.00 4 1 192 65160 1.2% reports still operational at the 24-hour period: occupancy, overdue-item, new-member reports still operational at the 3-hour period: occupancy ``` The first table shows where the load sits. There are 163 queries and 5,354,308 records read per day; 4,039,032 of that, or seventy-five percent, comes from **a single report**. The reason is not the number of runs but the records read per run: the overdue-item list scans the catalog's 128,000 rows on every run. The occupancy report, which runs 48 times a day, does not even reach one percent of the total. The second table counts the share the warehouse takes on, period by period, and it draws a two-way curve. As the period shortens, the lag drops, and as the lag drops, more reports can get their answer from the warehouse: from two to four. But the cycle count also rises, and every cycle re-reads the sources. With full dimension loading, records read come to 90.2 percent at the 24-hour period, 27.2 percent at the 3-hour period, and back up to 79.2 percent at the 1-hour period. The cheapest point is not at either end, it is in the middle. The column that actually decides is **dimension**. At the same period, incremental loading brings 27.2 percent down to 1.2 percent, because the catalog's 128,000 rows are read only for their changed portion, not on every cycle. Whether the operational load drops is not decided by the warehouse's existence but by how it reads the sources. One report never moves to the warehouse at any period: the occupancy report accepts a lag of one hour, and the shortest period's lag is 1.04 hours. The freshness requirement defines the path the warehouse cannot cover. #### The Pipeline's Steps and a Change in the Source Schema The pipeline is eight steps, and each step declares the field it reads. A field change in the source schema is actually applied: the field is renamed or removed, the pipeline is run, and the broken step is counted. ```js // warehouse/breakage.mjs — records moved step by step, and the step a source-schema change breaks import { generate, REPORTS, extract, run, STEPS, OWNER } from "./model.mjs"; const data = generate(), PERIOD = 3; const { slice } = extract(data, PERIOD, "full"); const baseline = run(slice); const lag = PERIOD + baseline.moved / 4000 / 3600; console.log("step input in out"); console.log("--------------- ----------------------------------- -------- --------"); for (const s of STEPS) { const c = baseline.counts.get(s.name); console.log(`${s.name.padEnd(15)} ${(s.source ?? s.inputs.join(",")).padEnd(35)} ` + `${String(c.inCount).padStart(8)} ${String(c.outCount).padStart(8)}`); } console.log(`one cycle moved ${STEPS.length} steps, ${baseline.moved} records; lag ${lag.toFixed(2)} hours`); // Three schema changes applied: a field is renamed or removed. const CHANGES = { "item.type -> item.typeCode": { entity: "item", field: "type", newName: "typeCode" }, "loan.branchId -> loan.branch": { entity: "loan", field: "branchId", newName: "branch" }, "member.regDay removed": { entity: "member", field: "regDay", newName: null }, }; const apply = (d, c) => ({ ...d, [c.entity]: d[c.entity].map((k) => { const y = { ...k }; delete y[c.field]; if (c.newName) y[c.newName] = k[c.field]; return y; }) }); console.log("\nchange owner broken stopped intact dropped in whs broken w/o whs"); console.log("----------------------------- --------------------------- ------ ------- ------- -------------- --------------"); for (const [name, c] of Object.entries(CHANGES)) { const d = [...run(apply(slice, c)).status]; const count = (f) => d.filter(([, v]) => f(v)).length; const broken = new Set(d.filter(([, v]) => v !== "done").map(([k]) => k)); const dropped = Object.values(REPORTS).filter((r) => broken.has(r.basis)).length; const direct = Object.values(REPORTS).filter((r) => r.field === `${c.entity}.${c.field}`).length; console.log(`${name.padEnd(29)} ${OWNER[c.entity].padEnd(27)} ${String(count((v) => v.startsWith("broken"))).padStart(6)} ` + `${String(count((v) => v === "stopped")).padStart(7)} ${String(count((v) => v === "done")).padStart(7)} ` + `${String(dropped).padStart(14)} ${String(direct).padStart(14)}`); } // Edge: without a warehouse, a report connects to every source it reads; with one, only the pipeline does. const rawEdges = Object.values(REPORTS).flatMap((r) => Object.keys(r.reads)); const uncovered = Object.values(REPORTS).filter((r) => r.freshness < lag); const uncoveredEdges = uncovered.flatMap((r) => Object.keys(r.reads)); console.log(`\nedges: without warehouse ${rawEdges.length} (report -> source); with warehouse ${4 + (5 - uncovered.length) + uncoveredEdges.length} ` + `(4 pulls + ${5 - uncovered.length} warehouse reads + ${uncoveredEdges.length} uncovered direct reads)`); console.log(`owners the management unit talks to: without warehouse ${new Set(rawEdges).size}, ` + `with warehouse ${new Set(["warehouse", ...uncoveredEdges]).size}`); ``` ``` step input in out --------------- ----------------------------------- -------- -------- extract-loan loan 146 146 extract-member member 46000 46000 extract-item item 128000 128000 extract-branch branch 12 12 clean extract-loan 146 144 enriched clean,extract-item,extract-branch 128156 144 branch-summary enriched 144 4 member-summary clean,extract-member 46144 143 one cycle moved 8 steps, 523341 records; lag 3.04 hours change owner broken stopped intact dropped in whs broken w/o whs ----------------------------- --------------------------- ------ ------- ------- -------------- -------------- item.type -> item.typeCode catalog (external) 1 2 5 3 2 loan.branchId -> loan.branch loan service (in-house) 1 4 3 5 1 member.regDay removed membership system 1 1 6 2 2 edges: without warehouse 10 (report -> source); with warehouse 10 (4 pulls + 4 warehouse reads + 2 uncovered direct reads) owners the management unit talks to: without warehouse 4, with warehouse 3 ``` The step table shows where the cost comes from. In the three-hour cycle, only 146 records come in from the loan side, but the total moved is 523,341. The whole difference is dimension reading: the `enriched` step takes in 128,156 records, because enriching 144 loan records means reading the entire catalog. The `clean` step passes 144 of the 146 records through; the two records with an empty branch are dropped. The breakage table gives this course's question. All three changes touch a single field, and all three break exactly **one** step of the pipeline. What differs is where the break stops. When the catalog's field breaks, two more steps stop and three reports are left unanswered — without a warehouse, only two reports used that field. When the loan service's field breaks, four steps stop and **all five of the five reports** drop — yet only one report read that field directly. In the membership field, the two counts come out equal. The warehouse ties together reports that read the same source. The occupancy report never uses the catalog's type field; without a warehouse it would be unaffected by that change, but with a warehouse it is affected, because the `branch-summary` step sits downstream of the `enriched` step. The compensation is that fixing it is localized: the break is a single step and belongs to a single team's responsibility. Without a warehouse, the same fix has to be made separately, in each report's independently written query. The last lines count the edges. The edge count does not change: 10 without a warehouse, 10 with one. What changes is direction and ownership. The management unit talks to four owners without a warehouse and three with one; the two remaining owners are there because of the occupancy report the warehouse cannot cover. An integration point does not reduce the edge count — it changes who the coordination is with. #### Summary - Five reports impose 163 queries and 5,354,308 records read per day on the operational system; seventy-five percent of that comes from a single report, because it scans the catalog's 128,000 rows on every run (IN10, IN11). - As the period shortens, the reports answered from the warehouse rise from two to four, but because the cycle count grows, records read climb back from 27.2 percent to 79.2 percent; the cheapest period is not at either end, it is in the middle. - Incremental dimension loading brings the operational load down from 27.2 percent to 1.2 percent at the same period; what decides it is not the warehouse's existence but how it reads the sources (IN13). - The report with a one-hour freshness requirement never moves to the warehouse at any period, because the shortest period's lag is 1.04 hours. This is how the warehouse reports the path it cannot cover (IN12). - The eight-step pipeline moves 523,341 records in one cycle; a single field change in the source breaks a single step, but in the loan field four steps stop downstream of the break, so all five of the five reports drop — only one report read that field directly (IN14). #### Next Step This lesson's measurements leaned on an assumption: that the source systems' fields are known, and that a schema change shows up as an error message. Not every system on the network is like that. There is a record system that has held years of loan history, and no one can reach its source code. The next lesson takes this up: when an unchangeable system's call surface is wrapped, how many paths close, how many stay outside the wrapper, and how long the window lasts where two systems write together while data ownership is handed off. ### Working With Legacy Systems Source: https://academia.sh/en/courses/enterprise-context/working-with-legacy-systems Wrapping a record system whose source code is unreachable, and the phased handoff of its ownership: reading the system's surface from a 60,000-call trace rather than from code, a wrapper that fully closes the interface closing 71.8 percent of calls while leaving seven call pairs and four owners outside, the coordination steps in the ownership handoff dropping from 24 to 13, and the co-written window never getting shorter than the slowest owner's release cycle. The previous lesson's measurements all assumed the source systems could be read: their fields were known, and when their schemas changed it showed up as an error message. In the regional library network's fictional model there is a system that does not meet this assumption. A record system that has held loan history for years is still running, and the network's oldest records sit in it, but no one can reach its source code: no field can be added, no behavior can be changed, and no one can ask what is happening inside. This lesson does not measure how the migration is carried out path by path. Its question comes earlier: what can be built around an unchangeable system, what remains that thing cannot close off, and how long the two sides write together while the data the system holds moves to a different owner. The network is a **fictional model** whose owners and budgets are separate. #### The Unreachable System's Surface Is Read From Traffic A system's call surface is usually derived from its code. Here there is no code, so the measure has to come from somewhere else: the **calls arriving** at the system are traced, and the surface is counted from that trace. The surface's unit is not a single operation name — it is the **(caller, operation) pair**; two systems calling the same operation are two separate edges, because the two have separate owners. **IN15: the calling systems, their owners, paths, and call shares are fictional.** **IN16: the legacy system's surface cannot be read from code; the measure is a 60,000-call trace observed over ninety days.** **IN17: the wrapper can only close operations on the call interface; work that enters the database directly and transfers that arrive through the file directory cannot be wrapped.** ```js // legacy/model.mjs — fictional model of a record system whose source code is unreachable. Its surface // is measured from traffic, not code: the call trace is actually generated and the next blocks scan it. export const SEED = 4127, DAYS = 90, EVENTS = 60_000, START = 30; // Each owner has its own release cycle (days) and hands off exactly one writer per cycle (IN18). export const CYCLE = { wrap: 7, loan: 7, branch: 14, fee: 14, membership: 21, data: 30, management: 30 }; // Calling systems: their owner, the path they reach the legacy system by, and the operations they // use. The "interface" path can be wrapped; the "database" and "file" paths stay outside the wrapper. export const CALLERS = { "branch-front": { owner: "branch", path: "interface", share: 34, operations: ["open-loan", "accept-return", "read-history"] }, "loan-service": { owner: "loan", path: "interface", share: 30, operations: ["open-loan", "accept-return", "extend", "write-fine"] }, membership: { owner: "membership", path: "interface", share: 8, operations: ["match-member", "read-history"] }, "night-job": { owner: "data", path: "database", share: 12, operations: ["batch-close", "archive"] }, "fee-accounting": { owner: "fee", path: "database", share: 9, operations: ["write-fine", "read-history"] }, "reporting-tool": { owner: "management", path: "database", share: 5, operations: ["read-history"] }, "branch-transfer": { owner: "branch", path: "file", share: 2, operations: ["batch-close", "archive"] }, }; // Each operation touches one data entity and either reads or writes it. export const OPERATIONS = { "open-loan": ["loan", "write"], "accept-return": ["loan", "write"], extend: ["loan", "write"], "batch-close": ["loan", "write"], "write-fine": ["fine", "write"], "match-member": ["member-link", "write"], archive: ["history", "write"], "read-history": ["history", "read"], }; export const ENTITIES = ["loan", "fine", "member-link", "history"]; // Call trace: each event carries a caller, an operation, and a day. export function generateTrace(t = SEED) { let s = t; const r = () => (s = (s * 48271) % 2147483647) / 2147483647; const names = Object.keys(CALLERS); const total = names.reduce((a, k) => a + CALLERS[k].share, 0); const trace = []; for (let i = 0; i < EVENTS; i += 1) { let p = r() * total, caller = names[0]; for (const k of names) { p -= CALLERS[k].share; if (p <= 0) { caller = k; break; } } const operation = CALLERS[caller].operations[Math.floor(r() * CALLERS[caller].operations.length)]; const [entity, direction] = OPERATIONS[operation]; trace.push({ caller, operation, entity, direction, path: CALLERS[caller].path, day: Math.floor(r() * DAYS) }); } return trace; } // The wrapper can only close operations reached over the interface path. export const closable = (e, scope) => e.path === "interface" && scope.includes(e.operation); export const SCOPE = { none: [], narrow: ["open-loan", "accept-return"], "full-interface": ["open-loan", "accept-return", "extend", "write-fine", "match-member", "read-history"], }; ``` #### What the Wrapper Closes and What It Cannot The wrapper is a layer placed in front of the legacy system that routes every call through itself. What it can close depends on which path the caller arrives by, and that choice of path is a decision the caller's owner made years ago. ```js // legacy/surface.mjs — the call surface and the share the wrapper closes, counted from the trace import { generateTrace, CALLERS, SCOPE, closable, EVENTS } from "./model.mjs"; const trace = generateTrace(); const surface = (d) => new Set(d.map((e) => `${e.caller}|${e.operation}`)).size; console.log(`trace: ${trace.length} calls, ${Object.keys(CALLERS).length} caller systems, ` + `${surface(trace)} distinct (caller, operation) pairs\n`); console.log("path callers surface calls share owners"); console.log("------------- ------- ------- ------- ------- ------"); for (const path of ["interface", "database", "file"]) { const events = trace.filter((e) => e.path === path); const callers = Object.entries(CALLERS).filter(([, v]) => v.path === path); console.log(`${path.padEnd(13)} ${String(callers.length).padStart(7)} ${String(surface(events)).padStart(7)} ` + `${String(events.length).padStart(7)} ${`${(events.length / EVENTS * 100).toFixed(1)}%`.padStart(7)} ` + `${String(new Set(callers.map(([, v]) => v.owner)).size).padStart(6)}`); } console.log("\nwrapper scope closed surface closed calls share outside surface outside calls outside owners"); console.log("--------------- --------------- ------------- -------- ---------------- -------------- ---------------"); for (const [name, scope] of Object.entries(SCOPE)) { const closed = trace.filter((e) => closable(e, scope)), outside = trace.filter((e) => closable(e, scope) === false); const outsideOwners = new Set(outside.map((e) => CALLERS[e.caller].owner)).size; console.log(`${name.padEnd(15)} ${String(surface(closed)).padStart(15)} ${String(closed.length).padStart(13)} ` + `${`${(closed.length / EVENTS * 100).toFixed(1)}%`.padStart(8)} ${String(surface(outside)).padStart(16)} ` + `${String(outside.length).padStart(14)} ${String(outsideOwners).padStart(15)}`); } ``` ``` trace: 60000 calls, 7 caller systems, 16 distinct (caller, operation) pairs path callers surface calls share owners ------------- ------- ------- ------- ------- ------ interface 3 9 43073 71.8% 3 database 3 5 15697 26.2% 3 file 1 2 1230 2.1% 1 wrapper scope closed surface closed calls share outside surface outside calls outside owners --------------- --------------- ------------- -------- ---------------- -------------- --------------- none 0 0 0.0% 16 60000 6 narrow 4 22516 37.5% 12 37484 6 full-interface 9 43073 71.8% 7 16927 4 ``` The first table shows that a system whose source code is unreachable can still be measured. There are seven caller systems, sixteen distinct (caller, operation) pairs, and three separate paths. 71.8 percent of calls come through the call interface; the rest either enter the database directly or arrive through a file directory. The file path, small as a number — 2.1 percent — carries two separate surface pairs, and one of them **writes** data. The second table gives the wrapper's limit. Even when its scope covers every operation on the interface, the closed share stops at 71.8 percent: seven surface pairs and 16,927 calls stay outside, because those calls never touch the interface at all. A wrapper closes a system's **door**; it does not close the holes in its wall. Since there is no access to the source code, the only way to close those holes is to talk to whoever opened them — the caller's owner. The **owners** column left outside says a second thing. The narrow-scope wrapper closes 37.5 percent of calls but does not lower the outside owner count from six at all. The share of closed calls does not mean coordination has dropped; what lowers coordination is **all** of an owner's paths closing. The wrapper that fully closes the interface brings six owners down to four. #### Ownership Handoff and the Co-Written Window The wrapper is a preparation; the actual work is the handoff of ownership of the data entities the legacy system holds to a new system. An entity's ownership does not move at a single moment: every path that writes to it is converted one at a time, and until all of them are converted, two systems write to it at once. This **window** can be measured. **IN18: each owner has its own release cycle and hands off exactly one writer per cycle.** **IN19: every writer left outside needs a notification cycle and a verification cycle (two steps); every writer behind the wrapper is handed off with a single redirect (one step).** ```js // legacy/handoff.mjs — data ownership handoff: the window in which two systems write at once, // and the coordination steps across owners. Every number is read from the same call trace. import { generateTrace, CALLERS, SCOPE, closable, ENTITIES, START, CYCLE } from "./model.mjs"; const trace = generateTrace(); function handoff(entity, scope) { const isWriter = (e) => e.entity === entity && e.direction === "write"; const writers = [...new Set(trace.filter(isWriter).map((e) => `${e.caller}|${e.operation}`))]; const cutover = new Map(), counter = {}, outside = []; for (const writer of writers) { const [caller, operation] = writer.split("|"); if (closable({ path: CALLERS[caller].path, operation }, scope)) { cutover.set(writer, CYCLE.wrap); continue; } const owner = CALLERS[caller].owner; counter[owner] = (counter[owner] ?? 0) + 1; cutover.set(writer, counter[owner] * CYCLE[owner]); outside.push(owner); } const coWritten = trace.filter((e) => isWriter(e) && e.day >= START && e.day < START + cutover.get(`${e.caller}|${e.operation}`)).length; return { writerCount: writers.length, insideCount: writers.length - outside.length, outsideCount: outside.length, ownerCount: new Set(outside).size, steps: (writers.length > outside.length ? 1 : 0) + 2 * outside.length, window: Math.max(...cutover.values()), coWritten }; } console.log("entity scope writers in wrap outside owners coord steps window (days) co-written"); console.log("------------ --------------- -------- -------- -------- ------- ------------ -------------- ------------"); const totals = {}; for (const entity of ENTITIES) for (const name of ["none", "full-interface"]) { const d = handoff(entity, SCOPE[name]); totals[name] = { steps: (totals[name]?.steps ?? 0) + d.steps, coWritten: (totals[name]?.coWritten ?? 0) + d.coWritten, window: Math.max(totals[name]?.window ?? 0, d.window) }; console.log(`${entity.padEnd(12)} ${name.padEnd(15)} ${String(d.writerCount).padStart(8)} ${String(d.insideCount).padStart(8)} ` + `${String(d.outsideCount).padStart(8)} ${String(d.ownerCount).padStart(7)} ${String(d.steps).padStart(12)} ` + `${String(d.window).padStart(14)} ${String(d.coWritten).padStart(12)}`); } for (const [name, t] of Object.entries(totals)) console.log(`total (${name}): ${t.steps} coordination steps, longest window ${t.window} days, ` + `${t.coWritten} calls written to both systems`); ``` ``` entity scope writers in wrap outside owners coord steps window (days) co-written ------------ --------------- -------- -------- -------- ------- ------------ -------------- ------------ loan none 7 0 7 3 14 42 6862 loan full-interface 7 5 2 2 5 30 3386 fine none 2 0 2 2 4 14 781 fine full-interface 2 1 1 1 3 14 781 member-link none 1 0 1 1 2 21 556 member-link full-interface 1 1 0 0 1 7 173 history none 2 0 2 2 4 30 1307 history full-interface 2 0 2 2 4 30 1307 total (none): 24 coordination steps, longest window 42 days, 9506 calls written to both systems total (full-interface): 13 coordination steps, longest window 30 days, 5647 calls written to both systems ``` What the wrapper gains is not concentrated in a single column of the table. Coordination steps drop from 24 to 13, co-written calls from 9,506 to 5,647. The longest window falls from 42 days to 30 — but it does not go below thirty. The four rows give four separate results, and what makes the difference is who writes to the entity. Seven paths write to loan records; five of them go through the interface, so they are handed off behind the wrapper with a single redirect, and the remaining two wait for their own owners' cycles: coordination drops from 14 steps to 5, the window from 42 days to 30. The member link has a single writer, and it goes through the interface; with the wrapper the window drops from 21 days to 7 and co-written calls from 556 to 173. In the fine records, the wrapper gains one step but does not shorten the window at all, because the window is set by the writer outside. In the history records, the two rows are identical: both paths that write to that entity are outside the interface, so the wrapper changes nothing in this handoff. The limit that comes out of this is this lesson's real measure. The wrapper does not set the window's lower bound; **the slowest owner's release cycle** does. The owner that runs the night job has a thirty-day cycle, and that owner writes to the history records directly from the database. No matter how wide the wrapper is, it cannot close that path, so thirty days will be paid. A decision that wants to speed up the migration should look first not at the wrapper's scope, but at the owners of the paths left outside and their cycles. There is also a warning in the co-written calls column. Over the course of the window, 5,647 calls are written to both systems at once; these calls have to be correct on both sides, and each one means two write operations. The length of the window is not a calendar problem — it is the number of records double-written over that calendar. #### Summary - The call surface of a system whose source code is unreachable is measured from a trace, not from code: a 60,000-call trace shows 7 callers, 16 distinct (caller, operation) pairs, and three separate paths (IN15, IN16). - The wrapper that fully closes the interface closes 71.8 percent of calls; 7 surface pairs and 16,927 calls stay outside, because the paths entering through the database and the file directory cannot be wrapped (IN17). - The share of closed calls does not show that coordination has dropped: the narrow-scope wrapper closes 37.5 percent of calls yet does not lower the outside owner count from six. An owner drops off the list only once all of its paths are closed. - In the ownership handoff, the wrapper brings coordination steps down from 24 to 13, and co-written calls from 9,506 to 5,647; in the member link the window drops from 21 days to 7, while in the history records the two cases stay identical (IN18, IN19). - The window's lower bound is set not by the wrapper's scope but by the slowest owner's release cycle: because the owner with a thirty-day cycle writes to the database directly, the longest window can only fall from 42 days to 30. #### Next Step This lesson worked with what was already there: the system stands, it cannot be changed, a layer is built around it, and its ownership is handed off step by step. The decision itself had already been made — that system was once either bought or written in-house, and every cost paid today is a continuation of that decision. The course's last lesson tries to measure that decision. Two options are modeled for one capability: buying and building. Total cost of ownership is added up item by item — adaptation, integration edges, release upgrades, training, and exit — and the winner is shown to change with the time horizon. A second number is added next to it: the risk that a path whose owner is someone else gets cut, that is, dependency risk. ### The Buy-or-Build Decision Source: https://academia.sh/en/courses/enterprise-context/buy-or-build-decision Comparing a capability's two options by total cost of ownership: adding up the adaptation, integration edge, training, release upgrade, and exit line items separately, buying coming to 402 hours plus 114 a year against building's 606 hours plus 54 a year, the winner changing at a 3.40-year horizon, the breakeven horizon falling to 1.79 years once the cut risk of the eight points with an outside owner is added, and that same bound rising to 12.00 years as the product's built-in scope grows. The previous lesson worked with what was already there: an unchangeable system stood, a layer was built around it, and its ownership was handed off step by step. That system had once either been bought or written in-house; every wrapper, coordination, and window cost paid today is a continuation of that old decision. This lesson tries to measure the decision itself. In the regional library network's fictional model, the management unit wants a reporting surface. There are two options: buying an outside product or writing it in-house. The decision goes unanswered if it is asked as "which is cheaper," because cheapness is not defined without stating a time span. #### Total Cost of Ownership Is Line by Line A buy decision is often defended with a single number — the setup cost. But over a capability's lifetime there are five separate line items it pays: two only once, two every year, and one paid on the way out. **IN20: the capability count, user count, and hour rates are fictional.** **IN21: exit cost is not measured in this lesson; it was measured in another lesson by scanning a source tree, and it enters here as a ready-made line item — 96 hours for the bought path, 24 for the built path.** **IN22: total cost of ownership is the sum of the once-paid line items plus the horizon times the yearly line items; exit is paid at the end of the horizon.** ```js // decision.mjs — a capability's two options: buy or build. Every line item is computed from // model sizes; the exit item is taken as external input (IN21). const NEED = 18; // the report capability the management unit wants const USERS = 9; // people using the surface const SOURCE_EDGES = 5; // source systems to connect to // IN20: hour rates and sizes are fictional. const RATE = { setup: 40, adaptHours: 14, writeHours: 26, buyEdge: 22, buildEdge: 12, trainBuy: 4, trainBuild: 6, releaseCount: 2, releasePoint: 3, maintainCapability: 2.5, exitBuy: 96, exitBuild: 24 }; // One option's line items. "builtin": the number of capabilities the product already provides. function lineItems(builtin) { const adapted = NEED - builtin; const buyEdges = SOURCE_EDGES + 1; // the product wants its own copy of the data too const buy = { adapt: [RATE.setup + adapted * RATE.adaptHours, 0], "integration edge": [buyEdges * RATE.buyEdge, 0], train: [USERS * RATE.trainBuy, USERS * 2 * RATE.releaseCount], "release upgrade": [0, RATE.releaseCount * (adapted + buyEdges) * RATE.releasePoint], exit: [RATE.exitBuy, 0], }; const build = { adapt: [NEED * RATE.writeHours, 0], "integration edge": [SOURCE_EDGES * RATE.buildEdge, 0], train: [USERS * RATE.trainBuild, USERS], "release upgrade": [0, NEED * RATE.maintainCapability], exit: [RATE.exitBuild, 0], }; const sum = (o, i) => Object.values(o).reduce((t, v) => t + v[i], 0); return { buy, build, buyOnce: sum(buy, 0), buyYearly: sum(buy, 1), buildOnce: sum(build, 0), buildYearly: sum(build, 1), adapted, buyEdges }; } // IN22: total cost of ownership = paid once + horizon x yearly. const tco = (once, yearly, horizon) => once + horizon * yearly; // The horizon where two lines cross; negative means no crossing. const breakeven = (o1, y1, o2, y2) => (o1 - o2) / (y2 - y1); const m = lineItems(11); console.log(`need ${NEED} capabilities, the product provides ${NEED - m.adapted} of them built in, ` + `${m.adapted} will be adapted; edges: buy ${m.buyEdges}, build ${SOURCE_EDGES}\n`); console.log("item buy (once) buy (yearly) build (once) build (yearly)"); console.log("------------------ ------------ ------------ ------------------- ------------------"); for (const name of Object.keys(m.buy)) console.log(`${name.padEnd(18)} ${m.buy[name][0].toFixed(0).padStart(12)} ${m.buy[name][1].toFixed(0).padStart(12)} ` + `${m.build[name][0].toFixed(0).padStart(19)} ${m.build[name][1].toFixed(0).padStart(18)}`); console.log(`${"total".padEnd(18)} ${m.buyOnce.toFixed(0).padStart(12)} ${m.buyYearly.toFixed(0).padStart(12)} ` + `${m.buildOnce.toFixed(0).padStart(19)} ${m.buildYearly.toFixed(0).padStart(18)}`); // IN23: a point with an outside owner is cut with probability 0.12 a year, an inside owner with 0.20. // The outside path exists only in the "buy" option; the inside path (source systems) exists in both. const OUTSIDE = 0.12, INSIDE = 0.20; const outsideCost = RATE.exitBuy + (RATE.setup + m.adapted * RATE.adaptHours) + m.buyEdges * RATE.buyEdge; const insideCostBuy = SOURCE_EDGES * RATE.buyEdge, insideCostBuild = SOURCE_EDGES * RATE.buildEdge; const buyYearlyRisk = m.buyYearly + OUTSIDE * outsideCost + INSIDE * insideCostBuy; const buildYearlyRisk = m.buildYearly + INSIDE * insideCostBuild; console.log("\nhorizon (yrs) buy build winner diff buy (risky) build (risky) winner"); console.log("------------- ------- --------- -------- ------ ------------- -------------- --------"); for (const horizon of [1, 2, 3, 4, 5, 8]) { const buy = tco(m.buyOnce, m.buyYearly, horizon), build = tco(m.buildOnce, m.buildYearly, horizon); const buyRisk = tco(m.buyOnce, buyYearlyRisk, horizon), buildRisk = tco(m.buildOnce, buildYearlyRisk, horizon); console.log(`${String(horizon).padStart(13)} ${buy.toFixed(0).padStart(7)} ${build.toFixed(0).padStart(9)} ` + `${(buy < build ? "buy" : "build").padStart(8)} ${Math.abs(buy - build).toFixed(0).padStart(6)} ` + `${buyRisk.toFixed(0).padStart(13)} ${buildRisk.toFixed(0).padStart(14)} ${(buyRisk < buildRisk ? "buy" : "build").padStart(8)}`); } console.log(`\npoints with an outside owner: buy ${m.adapted + 1}, build 0; ` + `edges with an inside owner: ${SOURCE_EDGES} in both options`); console.log(`cut cost: outside ${outsideCost} hours (buy), inside ${insideCostBuy} / ${insideCostBuild} hours; ` + `yearly expected buy ${(OUTSIDE * outsideCost + INSIDE * insideCostBuy).toFixed(1)}, build ${(INSIDE * insideCostBuild).toFixed(1)} hours`); console.log(`breakeven horizon: risk-free ${breakeven(m.buyOnce, m.buyYearly, m.buildOnce, m.buildYearly).toFixed(2)} years, ` + `risky ${breakeven(m.buyOnce, buyYearlyRisk, m.buildOnce, buildYearlyRisk).toFixed(2)} years`); console.log("\nbuiltin adapted buy (once) buy (yearly) breakeven (yrs) winner at 5 yrs"); console.log("-------- --------- ------------ ------------ ------------------ -----------------"); for (const builtin of [6, 9, 11, 14, 17]) { const s = lineItems(builtin); const breakevenYears = breakeven(s.buyOnce, s.buyYearly, s.buildOnce, s.buildYearly); const winner = tco(s.buyOnce, s.buyYearly, 5) < tco(s.buildOnce, s.buildYearly, 5) ? "buy" : "build"; console.log(`${String(builtin).padStart(7)} ${String(s.adapted).padStart(9)} ` + `${s.buyOnce.toFixed(0).padStart(12)} ${s.buyYearly.toFixed(0).padStart(12)} ${breakevenYears.toFixed(2).padStart(18)} ` + `${winner.padStart(17)}`); } ``` ``` need 18 capabilities, the product provides 11 of them built in, 7 will be adapted; edges: buy 6, build 5 item buy (once) buy (yearly) build (once) build (yearly) ------------------ ------------ ------------ ------------------- ------------------ adapt 138 0 468 0 integration edge 132 0 60 0 train 36 36 54 9 release upgrade 0 78 0 45 exit 96 0 24 0 total 402 114 606 54 horizon (yrs) buy build winner diff buy (risky) build (risky) winner ------------- ------- --------- -------- ------ ------------- -------------- -------- 1 516 660 buy 144 582 672 buy 2 630 714 buy 84 762 738 build 3 744 768 buy 24 942 804 build 4 858 822 build 36 1122 870 build 5 972 876 build 96 1302 936 build 8 1314 1038 build 276 1841 1134 build points with an outside owner: buy 8, build 0; edges with an inside owner: 5 in both options cut cost: outside 366 hours (buy), inside 110 / 60 hours; yearly expected buy 65.9, build 12.0 hours breakeven horizon: risk-free 3.40 years, risky 1.79 years builtin adapted buy (once) buy (yearly) breakeven (yrs) winner at 5 yrs -------- --------- ------------ ------------ ------------------ ----------------- 6 12 472 144 1.49 build 9 9 430 126 2.44 build 11 7 402 114 3.40 build 14 4 360 96 5.86 buy 17 1 318 78 12.00 buy ``` #### The Winner Depends on the Time Horizon The first table splits the two options into five line items and shows the items pointing in different directions. Buying is about a third cheaper on the once-paid side (402 against 606 hours) but more than twice as expensive on the yearly side (114 against 54 hours). Two items produce this difference. The release-upgrade item is 78 hours for buying, 45 for building; the number for buying does not depend on the product's own release calendar but on **how many points that calendar touches** — seven adaptation points and six edges. If adaptation drops, upgrades get cheaper too; as adaptation grows, every release costs more. The edge item carries the second difference. The bought path connects not to five sources but to six, because the product wants its own copy of the data; and since every edge needs its own adapter, the cost per edge is 22 hours against 12 hours for the in-house build. The same five sources come to 132 hours against 60 across the two options. The second table follows these two lines across the horizon. At a one-year horizon, buying is ahead by 144 hours; at three years the difference falls to 24 hours; in the fourth year, building takes the lead. The crossing point is exactly **3.40 years**. The 24-hour difference in the three-year row is small enough that choosing one of the rates slightly differently would change the winner; a document defending the decision has to write down not just a single winner's name but the horizon and the difference too. #### Dependency Risk Is a Line Item Too The math so far assumes both paths run without a hitch. At enterprise scale, this assumption does not hold: if a path's owner is someone else, that path can be cut, and the cost of the cut can be measured. **IN23: a path with an outside owner is cut with probability 0.12 a year, a path with an inside owner with 0.20.** The bought path has eight points with an outside owner: seven adaptation points and the edge for the product's own data copy. The cost of cutting these is exit plus re-adaptation plus edges, that is, 366 hours. The built path has no outside owner; in both options, the five source edges are owned by other teams, and their cut cost is 110 hours for buying, 60 for building. The yearly expected cost comes to 65.9 hours for buying, 12.0 for building. When these two numbers are added on top of the yearly line item, the breakeven horizon falls from **3.40 years to 1.79 years**: the option that looked like the winner for three years without accounting for risk starts losing from the second year once risk is included. Dependency risk is not a warning sentence, it is a yearly line item; until it is turned into a line item, it cannot enter the decision. The last table shows what variable the decision actually depends on. When the capability the product provides built in rises from 6 to 17, the breakeven horizon stretches from 1.49 years to 12.00 years, and the winner at the five-year horizon changes. The difference between the two options is not in their names — it is in how much of the need the product already covers out of the box. As scope rises, the adaptation points fall; as adaptation falls, both the once-paid and the yearly release item drop; the same math, with the same rates, turns the other way. #### Summary - Total cost of ownership is five line items: buying 402 hours plus 114 a year, building 606 hours plus 54 a year; the option that is cheaper once-paid is more expensive yearly (IN20, IN22). - The release-upgrade item depends not on the product's calendar but on the number of adapted points: seven adaptation points and six edges come to 78 hours a year, while maintaining the in-house build costs 45. - The bought path connects not to five sources but to six, and pays 22 hours per edge; the same sources cost 60 hours in the in-house build. A product's own copy of the data is an edge line item. - The winner depends on the horizon: breakeven at 3.40 years, and the difference at the three-year horizon is only 24 hours. A document defending the decision must write down the horizon and the difference along with the winner's name. - Dependency risk is a yearly line item. The eight points with an outside owner and their 366-hour cut cost come to 65.9 hours a year; once this item is added, the breakeven horizon falls to 1.79 years (IN21, IN23). #### Course Wrap-Up Over the course, the same fictional network was modeled ten times, and every lesson read a different quantity off the same network. | Lesson | Modeled enterprise object | Measured quantity | Cost of the decision | |---|---|---|---| | The Enterprise Architecture Concept | 9 systems, 6 owners, 13 capabilities, 10 data entities | Alignment 11/13; 2 gaps, 6 overlaps, 1 orphan system; 20 of 23 directed edges cross an ownership boundary | 5 of the 6 overlaps spread across two separate owners and require 5 reconciliations | | Frameworks | Layer, view, governance cycle, maturity level | The detailed form answers 10 of 12 questions with 106 items, the light form answers 5 with 22 items | 16.8 items per additional answer; maintenance per change is 18 items against 2 | | Business Process Modeling | A 16-step process; actor, input set, output | 10 steps automated, 4 a human decision; 8 actors, 5 owners, the owner changes in 9 of 10 handoffs | Pushing the boundary gains 62 touches but produces 10 wrong decisions and 30 rollbacks; breakeven at 6.20 touches | | Capability Mapping | Capability, the system serving it, the data written, the chain of reading systems | 11 of 13 capabilities are traceable; the short chain shows 1.55 systems, the full chain 4.73 | Coordination rounds rise from 5 to 29; 67 percent of the systems to touch do not show up in the first ring | | Integration Patterns | A single edge, built with four different patterns | With remote calls, 100 source requests for a hundred queries, 0 for the others; freshness windows of 0 and 1 tick | When the source renames a single column, 3 of 5 edges silently return the wrong answer, and coordination is needed with 2 of 3 owners | | The Enterprise Service Bus | Nine flows, in a pairwise and a centralized layout | 9 against 8 units written; the hub processes 24 messages for 6 events; with the hub down, 0 of 18 deliveries go through | The seventh system drops from 5 units and 4 owners to 2 units and 2 owners; in exchange, 4 rules with an outside owner pile up in the hub | | Master Data Management | A member record held in three systems, and the golden record | 720 write events produce 131 conflicting (member, field) pairs in the distributed layout, 95 with the golden record | Remaining inconsistency is 48 in both layouts; write steps rise from 1.00 to 1.61, and with the source system down, accepted writes fall from 496/720 to 238/720 | | The Data Warehouse and Transformation Pipelines | Five reports, four sources, an eight-step transformation pipeline | 163 queries and 5,354,308 records read per day; scanning across periods, operational load at 90.2, 27.2, and 79.2 percent | A single field change breaks one step but drops all five of the five reports; the report with a one-hour freshness requirement never makes it to the warehouse at any period | | Working With Legacy Systems | A record system whose source code is unreachable, and its wrapper | 9 of 16 (caller, operation) pairs close, 71.8 percent of calls; 7 pairs and 16,927 calls stay outside | Coordination falls from 24 steps to 13, but the co-written window does not go below 30 days because of the slowest owner's cycle | | The Buy-or-Build Decision | A capability's two options and five cost line items | Buying 402 hours plus 114 a year, building 606 plus 54 a year; breakeven at 3.40 years | The 8 points with an outside owner bring an expected cost of 65.9 hours a year, and the breakeven horizon falls to 1.79 years | The ten lessons' shared rule is a single sentence: **at enterprise scale, the unit of measure is the edge.** Inside a single system, the unit of measure was a call, a record, or a second. At enterprise scale none of these carries a decision by itself; in every table, what decided things was the number and direction of edges, which ownership boundary an edge crossed, and how many owners a change called into coordination. The wrapper's scope, the choice of golden record, and the warehouse's period were all bounded in the same place: at the owner its own decision could not reach. The gap the course leaves is this. Edges were built and counted, ownership was separated, buying was told apart from building — but none of these decisions said **which quality attribute** they were protecting. What was a centralized layout chosen for: availability, changeability, consistency? And no mechanism was built to check whether decisions stayed on paper or kept holding as they were made. The next course, **M20/K04 Quality Attributes and Governance**, opens with these two questions.