Developer Portal and Service Catalog

The previous lesson generated five services as real files; each one’s definition has an owner line. That line was correct the day it was written. But the three questions the person answering the page asks at midnight when the billing calculator goes down are different: who is on call for this service, what does this service depend on, and who is affected if this service goes down. The answer to the third question is written in no service’s own file, because nobody keeps the reverse edge in their own repository.

A service catalog is a list that keeps the ownership and dependency records of running services in one place; a developer portal is the surface that makes that list queryable. What this lesson measures is not the portal’s appearance, but the questions the catalog answers and how correct the answer is.

The measurement runs on the regional measurement network: fictional software that collects water-meter readings, validates them, and opens field work orders. The catalog records and the ground truth are a model written in node.

PC13. The catalog holds twelve records. A record has five fields: name, owner team, oncall list, dependency list, days since the last manual update. PC14. The ground truth is derived from the last ninety days of run history — observed call edges and the team that ran the pipeline — and stands as a separate structure in the model. PC15. No person’s name appears; what is reached is an oncall list. PC16. Five incidents and every step’s minutes are written in advance, with no randomness. PC17. The owner field can be read from the service definition the scaffold generates; the oncall list and the dependency list are written by hand. PC18. An unowned service is a record whose owner field is empty or names a team that does not exist. An orphan dependency is an edge pointing to a service that has no record in the catalog.

From Records to a Graph

The catalog is a flat list; what is queryable is not the list itself but the graph built from it. Each record gives a node, each dependency gives an edge; the ownership field ties the node to a team. The graph’s value comes from two places: the edge’s reverse direction and the gap between record and reality. Neither can be seen by looking at individual records one at a time.

// catalog.mjs — service catalog records and the ground truth derived from the run history. Model, hand-written.
// record: [name, owner team, oncall list, dependencies, days since last manual update]
export const RECORD = [
  ["reading-collector", "meter-reading", "oncall-meter", ["measurement-store", "identity-edge"], 9],
  ["reading-resolver", "meter-reading", "oncall-meter", ["reading-collector"], 21],
  ["validation-flow", "validation", "oncall-validation", ["reading-collector", "rule-store"], 14],
  ["rule-store", "validation", "oncall-validation", [], 27],
  ["billing-calculator", "billing", "oncall-billing", ["validation-flow", "tariff-service"], 24],
  ["tariff-service", "billing", "oncall-billing", [], 210],
  ["invoice-writer", "billing", "oncall-billing", ["billing-calculator"], 33],
  ["field-order", "field", "oncall-field", ["validation-flow", "map-edge"], 150],
  ["map-edge", "", "", [], 260],
  ["report-compiler", "reporting", "oncall-reporting",
    ["measurement-store", "archive-store", "invoice-writer"], 120],
  ["dashboard-edge", "reporting", "oncall-reporting", ["report-compiler"], 19],
  ["measurement-store", "distributed-team", "oncall-legacy", [], 180]];

export const ACTUAL_DEPS = {                         // call edges observed over the last ninety days
  "reading-collector": ["measurement-store", "identity-edge"], "reading-resolver": ["reading-collector"],
  "validation-flow": ["reading-collector", "rule-store"], "rule-store": [],
  "billing-calculator": ["validation-flow", "tariff-service", "identity-edge"], "tariff-service": [],
  "invoice-writer": ["billing-calculator"], "field-order": ["validation-flow"],
  "map-edge": ["identity-edge"], "report-compiler": ["measurement-store", "invoice-writer"],
  "dashboard-edge": ["report-compiler"], "measurement-store": [], "identity-edge": [] };

export const ACTUAL_OWNER = {                        // team that ran the pipeline over the last ninety days
  "reading-collector": "meter-reading", "reading-resolver": "meter-reading",
  "validation-flow": "validation", "rule-store": "validation", "billing-calculator": "billing",
  "tariff-service": "reporting", "invoice-writer": "billing", "field-order": "field",
  "map-edge": "field", "report-compiler": "reporting", "dashboard-edge": "reporting",
  "measurement-store": "meter-reading", "identity-edge": "validation" };

export const MULTI_TEAM = ["measurement-store", "map-edge"];   // service touched by more than one team
export const TEAM = ["meter-reading", "validation", "billing", "field", "reporting"];

// graph.mjs — an ownership and dependency graph is built from catalog records and compared to reality.
import { RECORD, ACTUAL_DEPS, ACTUAL_OWNER, TEAM } from "./catalog.mjs";

const registered = new Set(RECORD.map((k) => k[0]));
const reverse = {};                                  // who depends on me: the catalog builds the reverse edge
for (const [name, , , deps] of RECORD) for (const b of deps) (reverse[b] ??= []).push(name);

const unregistered = Object.keys(ACTUAL_DEPS).filter((s) => !registered.has(s));
const unowned = RECORD.filter(([, s]) => !s || !TEAM.includes(s));
const orphaned = RECORD.flatMap(([name, , , deps]) => deps.filter((b) => !registered.has(b)).map((b) => [name, b]));
console.log(`catalog ${RECORD.length} records, ${RECORD.reduce((a, k) => a + k[3].length, 0)} dependency` +
  ` edges, reverse edges derivable for ${Object.keys(reverse).length} services`);
console.log(`services actually running but never registered ${unregistered.length}: ${unregistered.join(", ")}`);
console.log(`unowned services ${unowned.length}: ` +
  unowned.map(([a, s]) => `${a} (${s || "empty field"})`).join(", "));
console.log(`orphan dependencies ${orphaned.length}: ` + orphaned.map(([a, b]) =>
  `${a} -> ${b} [${b in ACTUAL_DEPS ? "service exists, no record" : "no service, stale record"}]`).join("; "));
const affectedByUnregistered = Object.entries(ACTUAL_DEPS)
  .filter(([, b]) => b.some((x) => unregistered.includes(x))).length;
console.log(`${affectedByUnregistered} services actually connect to an unregistered service;` +
  ` the portal's dependency answer is incomplete for those ${affectedByUnregistered} services`);

let mismatchedFields = 0, recordsMismatched = 0, depMismatch = 0, ownerMismatch = 0;   // record aging: record versus reality
const age = { young: [0, 0], old: [0, 0] };
for (const [name, owner, , deps, day] of RECORD) {
  const g = ACTUAL_DEPS[name] ?? [];
  const missing = g.filter((b) => !deps.includes(b)).length;
  const extra = deps.filter((b) => !g.includes(b)).length;
  const ownerDiff = ACTUAL_OWNER[name] !== owner ? 1 : 0;
  const diff = missing + extra + ownerDiff;
  mismatchedFields += diff; depMismatch += missing + extra; ownerMismatch += ownerDiff;
  recordsMismatched += diff > 0 ? 1 : 0;
  const bucket = day <= 30 ? "young" : "old";
  age[bucket][0]++; age[bucket][1] += diff > 0 ? 1 : 0;
}
const manual = RECORD.length * 2, generated = RECORD.length * 2;     // oncall and dependency are written by hand
console.log(`\nrecord fields ${manual + generated}: ${generated} are read from the service` +
  ` definition the scaffold generates, ${manual} (oncall list, dependencies) are written by hand`);
console.log(`records disagreeing with reality ${recordsMismatched}/${RECORD.length}, mismatched fields ${mismatchedFields}`);
console.log(`record age <= 30 days: ${age.young[1]}/${age.young[0]} mismatched` +
  `   > 30 days: ${age.old[1]}/${age.old[0]} mismatched`);
console.log(`source of the mismatch: dependencies ${depMismatch}, ownership ${ownerMismatch};` +
  ` both are derivable from the run history, the oncall list is not`);

const affected = (s) => {                            // walk over the reverse edge
  const stack = [s], result = new Set();
  while (stack.length) for (const u of reverse[stack.pop()] ?? [])
    if (!result.has(u)) { result.add(u); stack.push(u); }
  return [...result];
};
console.log(`\n"who is affected if billing-calculator goes down" — with the catalog ${affected("billing-calculator")
  .length} services (${affected("billing-calculator").join(", ")}), 1 graph walk`);
console.log(`the same question without the catalog: ${RECORD.length} services' dependency lists are scanned by hand,` +
  ` the reverse edge is not written in any file`);

catalog 12 records, 14 dependency edges, reverse edges derivable for 11 services
services actually running but never registered 1: identity-edge
unowned services 2: map-edge (empty field), measurement-store (distributed-team)
orphan dependencies 2: reading-collector -> identity-edge [service exists, no record]; report-compiler -> archive-store [no service, stale record]
3 services actually connect to an unregistered service; the portal's dependency answer is incomplete for those 3 services

record fields 48: 24 are read from the service definition the scaffold generates, 24 (oncall list, dependencies) are written by hand
records disagreeing with reality 6/12, mismatched fields 7
record age <= 30 days: 1/6 mismatched   > 30 days: 5/6 mismatched
source of the mismatch: dependencies 4, ownership 3; both are derivable from the run history, the oncall list is not

"who is affected if billing-calculator goes down" — with the catalog 3 services (invoice-writer, report-compiler, dashboard-edge), 1 graph walk
the same question without the catalog: 12 services' dependency lists are scanned by hand, the reverse edge is not written in any file

Building the graph shows three gaps at once. There are two unowned services, and they are unowned in different ways: map-edge’s owner field is empty, while measurement-store’s owner is a team that no longer exists. The second is more dangerous, because the record looks filled in and no empty-field scan catches it. There are also two orphan dependencies, and their causes are opposite: identity-edge is a service that genuinely runs but has no record at all in the catalog, while archive-store is a service that sits in a record but does not exist in reality. One is the record’s absence; the other is the record’s staleness.

The reverse edge is the answer only the catalog can give on its own. When the billing calculator goes down, the three affected services come out with a single graph walk; answering the same question without the catalog requires scanning all twelve services’ dependency lists one by one, because no service knows who depends on it.

The Correct Oncall List in an Incident

// incident.mjs — time to reach the correct oncall list across five incidents: with and without the catalog.
import { RECORD, ACTUAL_OWNER, MULTI_TEAM } from "./catalog.mjs";

const STEP = { repo: 6, firstMessage: 8, teamChannel: 15, findOncall: 12, readCatalog: 2, call: 3,
  wrongList: 10, review: 4 };                       // minutes, written in advance; no randomness
const INCIDENT = ["billing-calculator", "map-edge", "tariff-service", "field-order", "measurement-store"];

const withoutCatalog = (s) => STEP.repo + STEP.firstMessage + STEP.findOncall +
  (MULTI_TEAM.includes(s) ? STEP.teamChannel : 0);  // repo history, message, team channel if needed
const withCatalog = (s) => {
  const rec = RECORD.find(([a]) => a === s);
  if (!rec || !rec[2]) return [STEP.readCatalog + withoutCatalog(s), rec ? "oncall field empty" : "no record"];
  if (ACTUAL_OWNER[s] !== rec[1])
    return [STEP.readCatalog + STEP.wrongList + withoutCatalog(s), "record pointed to the wrong list"];
  return [STEP.readCatalog + STEP.call, "record up to date"];
};

console.log("incident             without cat  with cat  diff  status");
let a = 0, b = 0, faster = 0;
for (const s of INCIDENT) {
  const x = withoutCatalog(s), [y, note] = withCatalog(s);
  a += x; b += y; faster += y < x ? 1 : 0;
  console.log(`${s.padEnd(20)}${String(x).padStart(9)} min${String(y).padStart(7)} min` +
    `${String(y - x).padStart(6)}  ${note}`);
}
const upToDate = INCIDENT.length * (STEP.readCatalog + STEP.call);
console.log(`total ${a} min -> ${b} min   average ${(a / INCIDENT.length).toFixed(1)} min -> ` +
  `${(b / INCIDENT.length).toFixed(1)} min   incidents shortened ${faster}/${INCIDENT.length}`);
console.log(`if every record were up to date: ${upToDate} min, average ` +
  `${(upToDate / INCIDENT.length).toFixed(1)} min; current gain ${a - b} min, possible gain ${a - upToDate} min`);

const maintenance = RECORD.length * STEP.review;    // monthly review: four minutes per record
const perIncident = (a - upToDate) / INCIDENT.length;
console.log(`\nmonthly review ${maintenance} min (${RECORD.length} records x ${STEP.review} min);` +
  ` an up-to-date record's gain per incident ${perIncident} min`);
console.log(`threshold: review pays for itself after ${Math.ceil(maintenance / perIncident)} incidents a month;` +
  ` it does not pay off at ${Math.ceil(maintenance / perIncident) - 1} incident`);

incident             without cat  with cat  diff  status
billing-calculator         26 min      5 min   -21  record up to date
map-edge                   41 min     43 min     2  oncall field empty
tariff-service             26 min     38 min    12  record pointed to the wrong list
field-order                26 min      5 min   -21  record up to date
measurement-store          41 min     53 min    12  record pointed to the wrong list
total 160 min -> 144 min   average 32.0 min -> 28.8 min   incidents shortened 2/5
if every record were up to date: 25 min, average 5.0 min; current gain 16 min, possible gain 135 min

monthly review 48 min (12 records x 4 min); an up-to-date record's gain per incident 27 min
threshold: review pays for itself after 2 incidents a month; it does not pay off at 1 incident

Where the catalog pays off is clear: for the two services whose records are up to date, the time to reach the right list drops from twenty-six minutes to five. Where it does not pay off is more instructive. In three of the five incidents, the catalog lengthened the time. The record with an empty oncall field adds two minutes of reading cost and says nothing; the two records with the wrong owner add twelve minutes, because the person paging first writes to the wrong oncall list, waits for a reply, and only then falls back to the no-catalog path. A wrong record is more expensive than no record at all — the absence of a record misleads nobody.

Overall, the average drops from 32 minutes to 28.8 minutes; the measured gain is sixteen minutes. If every record agreed with reality, the same five incidents would finish in twenty-five minutes total and the gain would be a hundred thirty-five minutes. Only twelve percent of the value the catalog offers is actually captured, and what is missing is not the portal — it is the record’s accuracy.

The Record’s Own Cost

The record’s cost comes down to two numbers. Half of the catalog’s forty-eight fields are read from the service definition the scaffold generates and cost nothing; the other half — the oncall list and the dependency list — are written by hand. The hand-written fields age: six of twelve records disagree with reality, and seven fields are wrong. The aging tracks with age itself; one of the six records younger than thirty days disagrees with reality, against five of the six records older than thirty days.

The math on maintenance follows from this. Monthly review is forty-eight minutes at four minutes per record; an up-to-date record’s gain per incident is twenty-seven minutes. The threshold is two incidents a month: in a network with only one incident a month, review does not pay for itself; it pays off after two. In a place with as few services as the measurement network, this threshold sits close by, and that is exactly why catalog maintenance looks good on paper and does not happen in practice.

The layer the catalog hides is this: every team no longer has to read other teams’ repositories, channel history, and deployment records — it reads a single record instead. The leak sits at exactly this point too — the record behaves like a single source of truth but is not one, and when it leaks it gives no sign of doing so. Those left outside are countable: three services actually connect to the one unregistered service, meaning the portal’s dependency answer is incomplete for those three services. The teams behind those three services look at the catalog and see a false sense of completeness; a team working without the catalog at least knows it is looking at an incomplete picture.

There is a third path, and it changes the number at its root. Four of the seven mismatches come from dependencies, three from ownership; dependency edges can be derived from run records, and ownership from pipeline runs. If both are derived, the mismatch drops to zero and the hand-written fields drop from twenty-four to twelve. The one field left is the oncall list, and it cannot be derived from any run history, because oncall is not an observation — it is a decision.

Summary

• The graph built from twelve records showed three gaps: 2 unowned services, 2 orphan dependencies, and 1 service that actually runs but has no record at all in the catalog.
• The two cases of unownedness and the two orphan dependencies were different in kind: one an empty field, the other a team that no longer exists; one the record’s absence, the other the record’s staleness.
• The reverse edge was the answer only the catalog could give: when the billing calculator went down, the 3 affected services came out with a single walk, while the same question without the catalog would require scanning 12 dependency lists.
• Across five incidents, the average time to reach the right list dropped from 32 minutes to 28.8 minutes, but in 3 of the 5 incidents the catalog lengthened the time; a wrong record added 12 minutes, an empty field added 2 minutes.
• If the records had been up to date, the same five incidents would have finished in 25 minutes: the measured gain was 16 minutes, the possible gain was 135 minutes.
• 24 of 48 record fields are written by hand and 6 of 12 records disagree with reality; all 7 mismatches can be closed by deriving them from run history — the one field that cannot be closed is the oncall list.

Next Step

The catalog says who is responsible, but it sets no rule anyone is required to follow. For the measurement network’s three services carrying personal data, retention period, log masking, and signed packages are mandatory; today these rules are written in a document, and every team is expected to read and apply them. The next lesson compares the same rule set in two arrangements: publishing it as a document and embedding it in the pipeline. What is measured is the compliance rate, the number of violations, and the time teams spend — and the real number is how many teams comply without ever seeing the rule.