Keeping Documentation Current

The previous lesson made quality requirements measurable and produced a document; that document was in step with the code on the day it was written. This lesson measures what comes after: the code keeps changing, the document stays put, and the gap between them grows. This gap’s name is documentation drift, and it is found not by guessing but by counting.

The measurement method is borrowed. The Service Architectures course’s Boundary-Drawing Criteria lesson extracted import edges from source files and counted boundary violations. The same extraction is used here to compare the graph read from code against the document’s claim; the measured quantity is different.

Code, Document, and Audit

The block below defines three things: the regional library network system’s module map, a fourteen-claim document, and the audit that tests the claims against the code. The module map is a model; each module is written to disk as a source file, and the graph is extracted from those files — the measurement happens on real files.

Splitting the claims into three is the lesson’s axis: a rule claim states architectural intent, and if the code does not comply, there is drift; a snapshot claim is a count of the code’s current state on that day, and it goes stale when the code changes; an unverifiable claim cannot be checked against the code at all.

// environment.mjs — defines the codebase, the document, and the audit; when run directly,
// reports the starting state. The next two blocks import this file.
import { mkdirSync, rmSync, writeFileSync, readdirSync, readFileSync } from "node:fs";

// VM19: the six domain modules of the regional library network system. Each module's source
// carries which modules it imports, which tables it writes to, and which table it owns.
export const START = {
  branch:       { imports: [],                    writes: ["branch"],      owns: ["branch"] },
  catalog:      { imports: [],                    writes: ["book", "copy"], owns: ["book", "copy"] },
  membership:   { imports: [],                    writes: ["member"],      owns: ["member"] },
  loan:         { imports: ["catalog", "membership"], writes: ["loan"],    owns: ["loan"] },
  notification: { imports: ["membership", "loan"], writes: ["notification"], owns: ["notification"] },
  fee:          { imports: ["loan"],               writes: ["fee"],        owns: ["fee"] },
};

export const write = (m) => {          // the code/ directory is rebuilt from scratch on every write
  rmSync("code", { recursive: true, force: true });
  mkdirSync("code", { recursive: true });
  for (const [name, x] of Object.entries(m)) {
    writeFileSync(`code/${name}.mjs`,
      `// owns: ${x.owns.join(", ")}\n` +
      x.imports.map((i) => `import { ${i}Op } from "./${i}.mjs";\n`).join("") +
      `export const write = (t, v) => ({ t, v });\n` +
      `export function ${name}Op(v) {\n` +
      x.writes.map((t) => `  write("${t}", v);\n`).join("") + "}\n");
  }
};

// The import graph is extracted from source files. The method is borrowed from the Service Architectures course's
// Boundary-Drawing Criteria lesson, where it measured boundary violations; here it measures documentation drift.
export const graph = () => Object.fromEntries(readdirSync("code").map((d) => {
  const t = readFileSync(`code/${d}`, "utf8");
  return [d.replace(".mjs", ""), {
    imports: [...t.matchAll(/from "\.\/(\w+)\.mjs"/g)].map((x) => x[1]),
    writes: [...t.matchAll(/write\("(\w+)"/g)].map((x) => x[1]),
    owns: t.match(/^\/\/ owns: (.+)$/m)[1].split(", "),
  }];
}));

const importerOf = (g, a) => Object.entries(g).filter(([, m]) => m.imports.includes(a)).map(([x]) => x);
const writerOf = (g, t) => Object.entries(g).filter(([, m]) => m.writes.includes(t)).map(([x]) => x);

// The document carries three kinds of claim. RULE: architectural intent — a violation in the code is drift. SNAPSHOT: a count
// of the code's current state — it goes stale when the code changes. UNVERIFIABLE: cannot be checked against the code at all.
export const RULE = [
  ["K1", "membership", "Membership does not import any domain module.", (g) => g.membership.imports.length === 0],
  ["K2", "catalog", "Only loan may import catalog.",
    (g) => importerOf(g, "catalog").every((a) => a === "loan")],
  ["K3", "notification", "No module may depend on notification.", (g) => importerOf(g, "notification").length === 0],
  ["K4", "fee", "Fee may only import the loan module.",
    (g) => g.fee.imports.every((a) => a === "loan")],
  ["K5", "branch", "Branch does not import any domain module.", (g) => g.branch.imports.length === 0],
  ["K6", "loan", "Only loan may write to the loan table.",
    (g) => writerOf(g, "loan").every((a) => a === "loan")],
];

export const SNAPSHOT = [
  ["A1", "system", "The system consists of six domain modules.", (g) => Object.keys(g).length],
  ["A2", "loan", "Loan imports the catalog and membership modules.", (g) => g.loan.imports],
  ["A3", "system", "The graph has five import edges.",
    (g) => Object.values(g).reduce((s, m) => s + m.imports.length, 0)],
  ["A4", "notification", "Notification imports the membership and loan modules.", (g) => g.notification.imports],
  ["A5", "fee", "Fee writes only to the fee table.", (g) => g.fee.writes],
  ["A6", "catalog", "Catalog owns the book and copy tables.", (g) => g.catalog.owns],
];

export const UNVERIFIABLE = [
  ["D1", "catalog", "The catalog system is sourced externally."],
  ["D2", "system", "The nightly maintenance window is 02:00-04:00."],
];

// VM20: eight changes applied over the next two quarters. The intent label says whether the change
// complies with an architectural rule; the label is not asserted, it is compared against the audit result.
export const CHANGE = [
  ["reservation module is added", "compliant", "reservation",
    (m) => { m.reservation = { imports: ["membership"], writes: ["reservation"], owns: ["reservation"] }; }],
  ["loan imports notification", "non-compliant", "loan", (m) => m.loan.imports.push("notification")],
  ["fee imports membership", "non-compliant", "fee", (m) => m.fee.imports.push("membership")],
  ["notification drops its membership import", "compliant", "notification",
    (m) => { m.notification.imports = m.notification.imports.filter((a) => a !== "membership"); }],
  ["reservation writes to the loan table", "non-compliant", "reservation", (m) => m.reservation.writes.push("loan")],
  ["catalog takes over the tag table", "compliant", "catalog",
    (m) => { m.catalog.owns.push("tag"); m.catalog.writes.push("tag"); }],
  ["fee writes to the penalty table", "compliant", "fee", (m) => m.fee.writes.push("penalty")],
  ["branch imports membership", "non-compliant", "branch", (m) => m.branch.imports.push("membership")],
];

if (process.argv[1]?.endsWith("environment.mjs")) {
  write(structuredClone(START));
  const g = graph();
  const row = [
    ...RULE.map(([ad, ozne, metin, sina]) => [ad, "rule", ozne, sina(g) ? "true" : "false", metin]),
    ...SNAPSHOT.map(([ad, ozne, metin, oku]) => [ad, "snapshot", ozne, String(oku(g)), metin]),
    ...UNVERIFIABLE.map(([ad, ozne, metin]) => [ad, "unverifiable", ozne, "-", metin]),
  ];
  console.log(`${"claim".padEnd(6)}${"type".padEnd(13)}${"subject".padEnd(13)}${"t=0".padEnd(21)}text`);
  for (const [a, b, c, d, e] of row) {
    console.log(`${a.padEnd(6)}${b.padEnd(13)}${c.padEnd(13)}${d.padEnd(21)}${e}`);
  }
  console.log(`\n${Object.keys(g).length} modules, ` +
    `${Object.values(g).reduce((s, x) => s + x.imports.length, 0)} import edges; the document carries ${row.length} ` +
    `claims (${RULE.length} rule, ${SNAPSHOT.length} snapshot, ${UNVERIFIABLE.length} unverifiable)`);
  console.log(`t=0 drift: 0 false rule claims, 0 stale snapshot claims ` +
    `(the document was written the same day as the code)`);
}

claim type         subject      t=0                  text
K1    rule         membership   true                 Membership does not import any domain module.
K2    rule         catalog      true                 Only loan may import catalog.
K3    rule         notification true                 No module may depend on notification.
K4    rule         fee          true                 Fee may only import the loan module.
K5    rule         branch       true                 Branch does not import any domain module.
K6    rule         loan         true                 Only loan may write to the loan table.
A1    snapshot     system       6                    The system consists of six domain modules.
A2    snapshot     loan         catalog,membership   Loan imports the catalog and membership modules.
A3    snapshot     system       5                    The graph has five import edges.
A4    snapshot     notification membership,loan      Notification imports the membership and loan modules.
A5    snapshot     fee          fee                  Fee writes only to the fee table.
A6    snapshot     catalog      book,copy            Catalog owns the book and copy tables.
D1    unverifiable catalog      -                    The catalog system is sourced externally.
D2    unverifiable system       -                    The nightly maintenance window is 02:00-04:00.

6 modules, 5 import edges; the document carries 14 claims (6 rule, 6 snapshot, 2 unverifiable)
t=0 drift: 0 false rule claims, 0 stale snapshot claims (the document was written the same day as the code)

How Drift Grows

The document starts with zero drift. The next block applies the eight changes in order, retests all fourteen claims after each step, and never touches the document.

// drift.mjs — imports environment.mjs; measures the drift of an unupdated document, step by step
import { START, CHANGE, RULE, SNAPSHOT, UNVERIFIABLE, write, graph } from "./environment.mjs";

const m = structuredClone(START);
write(m);
const initial = graph();
const expected = Object.fromEntries(SNAPSHOT.map(([ad, , , oku]) => [ad, JSON.stringify(oku(initial))]));

const driftMeasure = (g) => ({
  rule: RULE.filter(([, , , test]) => !test(g)).map(([ad]) => ad),
  snapshot: SNAPSHOT.filter(([ad, , , oku]) => JSON.stringify(oku(g)) !== expected[ad]).map(([ad]) => ad),
});

console.log(`${"t".padStart(2)}  ${"change".padEnd(42)}${"intent".padEnd(15)}` +
  `${"claim dropped".padEnd(16)}${"false rules".padStart(12)}${"stale snapshots".padStart(16)}` +
  `${"drift".padStart(7)}`);
let previous = [];
CHANGE.forEach(([ad, intent, , apply], t) => {
  apply(m);
  write(m);
  const s = driftMeasure(graph());
  const all = [...s.rule, ...s.snapshot];
  const fresh = all.filter((a) => !previous.includes(a));
  previous = all;
  console.log(`${String(t + 1).padStart(2)}  ${ad.padEnd(42)}${intent.padEnd(15)}` +
    `${(fresh.join(",") || "-").padEnd(16)}${String(s.rule.length).padStart(12)}` +
    `${String(s.snapshot.length).padStart(16)}${String(all.length).padStart(7)}`);
});

const final = driftMeasure(graph());
const total = RULE.length + SNAPSHOT.length + UNVERIFIABLE.length;
console.log(`\nafter eight changes, ${final.rule.length + final.snapshot.length} of ${total} claims are ` +
  `wrong, ${UNVERIFIABLE.length} are unverifiable, ` +
  `${total - final.rule.length - final.snapshot.length - UNVERIFIABLE.length} are correct ` +
  `(correct ones remaining: ${RULE.filter(([a]) => !final.rule.includes(a)).map(([a]) => a).join(", ")})`);
console.log(`${CHANGE.filter(([, n]) => n === "compliant").length} compliant changes ` +
  `dropped no rule claim; ` +
  `${CHANGE.filter(([, n]) => n === "non-compliant").length} non-compliant changes ` +
  `dropped ${final.rule.length} rule claims`);

 t  change                                    intent         claim dropped    false rules stale snapshots  drift
 1  reservation module is added               compliant      A1,A3                      0               2      2
 2  loan imports notification                 non-compliant  K3,A2                      1               3      4
 3  fee imports membership                    non-compliant  K4                         2               3      5
 4  notification drops its membership import  compliant      A4                         2               4      6
 5  reservation writes to the loan table      non-compliant  K6                         3               4      7
 6  catalog takes over the tag table          compliant      A6                         3               5      8
 7  fee writes to the penalty table           compliant      A5                         3               6      9
 8  branch imports membership                 non-compliant  K5                         4               6     10

after eight changes, 10 of 14 claims are wrong, 2 are unverifiable, 2 are correct (correct ones remaining: K1, K2)
4 compliant changes dropped no rule claim; 4 non-compliant changes dropped 4 rule claims

Drift starts at the first change and grows in one direction: 2, 4, 5, 6, 7, 8, 9, 10. By the end, ten of the fourteen claims are wrong and two are unverifiable; the two that stay correct are rule claims. The document keeps talking, but most of it no longer matches the code.

The two claim types behave differently. None of the four compliant changes dropped a rule claim; all of them only staled snapshot claims. All four non-compliant changes dropped a rule claim. A wrong rule claim is a code defect; a stale snapshot claim is a document defect. Both show up in the same table, but one is fixed in the code, the other in the document.

Three Keeping-Current Regimes

The same eight changes now run under three regimes. Manual update: whoever makes the change reviews only the claims of the module they touched. Generated audit: after every change, every claim is tested against the code. Rule-only document: the document never carries a snapshot claim.

// regime.mjs — imports environment.mjs; compares three keeping-current regimes on the same eight changes
import { START, CHANGE, RULE, SNAPSHOT, UNVERIFIABLE, write, graph } from "./environment.mjs";

// manual:    whoever makes the change reviews the claims of the module they touched.
// audit:     after every change, every claim is automatically tested against the code.
// rule-only: the document carries only rule claims; no snapshot claim is ever written.
const run = (regime) => {
  const m = structuredClone(START);
  write(m);
  const doc = regime === "rule-only" ? [] :
    SNAPSHOT.map(([ad, ozne, , oku]) => ({ ad, ozne, oku, value: JSON.stringify(oku(graph())) }));
  const caught = new Set();
  let updates = 0, warnings = 0, falseAlarms = 0;

  for (const [, intent, touched, apply] of CHANGE) {
    apply(m);
    write(m);
    const g = graph();
    const violated = RULE.filter(([, , , test]) => !test(g));
    const stale = doc.filter((i) => JSON.stringify(i.oku(g)) !== i.value);
    if (regime === "manual") {
      for (const [ad, ozne] of violated) if (ozne === touched) caught.add(ad);
      for (const i of stale) if (i.ozne === touched) { i.value = JSON.stringify(i.oku(g)); updates += 1; }
    } else {
      for (const [ad] of violated) caught.add(ad);
      warnings += stale.length;
      if (intent === "compliant") falseAlarms += stale.length;
      for (const i of stale) { i.value = JSON.stringify(i.oku(g)); updates += 1; }
    }
  }
  const g = graph();
  const actual = RULE.filter(([, , , test]) => !test(g)).map(([ad]) => ad);
  return { regime, doc, actual, caught, updates, warnings, falseAlarms,
    stale: doc.filter((i) => JSON.stringify(i.oku(g)) !== i.value).map((i) => i.ad) };
};

const RESULT = ["manual", "audit", "rule-only"].map(run);
console.log(`${"regime".padEnd(10)}${"caught".padStart(9)}${"missed".padStart(18)}` +
  `${"updates".padStart(11)}${"snapshot warn.".padStart(16)}${"false alarms".padStart(14)}` +
  `${"still stale".padStart(13)}`);
for (const s of RESULT) {
  const missed = s.actual.filter((a) => !s.caught.has(a));
  console.log(`${s.regime.padEnd(10)}${`${s.caught.size}/${s.actual.length}`.padStart(9)}` +
    `${(missed.join(",") || "-").padStart(18)}${String(s.updates).padStart(11)}` +
    `${String(s.warnings).padStart(16)}${String(s.falseAlarms).padStart(14)}` +
    `${(s.stale.join(",") || "-").padStart(13)}`);
}

// Seven questions someone who comes later asks the document; each question lands on one claim.
const QUESTION = [
  ["How many domain modules does the system consist of?", "A1"],
  ["Which modules does the loan module use?", "A2"],
  ["How many import edges are in the graph?", "A3"],
  ["Who may depend on the notification module?", "K3"],
  ["Who may write to the loan table?", "K6"],
  ["Which tables does catalog own?", "A6"],
  ["When is the nightly maintenance window?", "D2"],
];
const g = graph();
const answer = (s, claim) => {
  if (RULE.some(([a]) => a === claim)) return "correct";          // a rule states intent
  if (UNVERIFIABLE.some(([a]) => a === claim)) return s.regime === "rule-only" ? "unanswered" : "unverifiable";
  const i = s.doc.find((x) => x.ad === claim);
  if (!i) return "unanswered";
  return JSON.stringify(i.oku(g)) === i.value ? "correct" : "wrong";
};

console.log(`\n${"question".padEnd(54)}${"claim".padEnd(7)}${"manual".padEnd(14)}${"audit".padEnd(14)}rule-only`);
for (const [question, claim] of QUESTION) {
  console.log(`${question.padEnd(54)}${claim.padEnd(7)}` +
    RESULT.map((s) => answer(s, claim).padEnd(14)).join("").trimEnd());
}
for (const s of RESULT) {
  const v = QUESTION.map(([, i]) => answer(s, i));
  console.log(`${s.regime.padEnd(10)}${v.filter((x) => x === "correct").length} correct, ` +
    `${v.filter((x) => x === "wrong").length} wrong, ` +
    `${v.filter((x) => x === "unanswered").length} unanswered, ` +
    `${v.filter((x) => x === "unverifiable").length} unverifiable`);
}

regime       caught            missed    updates  snapshot warn.  false alarms  still stale
manual          3/4                K6          4               0             0        A1,A3
audit           4/4                 -         10              10             6            -
rule-only       4/4                 -          0               0             0            -

question                                              claim  manual        audit         rule-only
How many domain modules does the system consist of?   A1     wrong         correct       unanswered
Which modules does the loan module use?               A2     correct       correct       unanswered
How many import edges are in the graph?               A3     wrong         correct       unanswered
Who may depend on the notification module?            K3     correct       correct       correct
Who may write to the loan table?                      K6     correct       correct       correct
Which tables does catalog own?                        A6     correct       correct       unanswered
When is the nightly maintenance window?               D2     unverifiable  unverifiable  unanswered
manual    4 correct, 2 wrong, 0 unanswered, 1 unverifiable
audit     6 correct, 0 wrong, 0 unanswered, 1 unverifiable
rule-only 2 correct, 0 wrong, 5 unanswered, 0 unverifiable

Manual update catches three of the four real drifts and misses the fourth. The missed drift is K6: the new module writing to the loan table is reservation, but the claim’s subject is loan, and touching reservation does not trigger a look at loan’s claim. This is manual update’s blind spot — drift is invisible when the touched module and the claim’s subject differ. Its maintenance cost is the lowest: four claim updates.

The generated audit catches all four of the drifts and leaves no stale claim behind. Its price is ten claim updates and ten warnings. Six of the warnings come from compliant changes, meaning they arrived while the code was still correct: a false alarm. The source is clear — snapshot claims go stale on every legitimate change, and the audit does not tell them apart from real drift.

The rule-only document zeroes out both line items: four drifts caught, zero updates, zero false alarms. Its price is in the question table — five of the seven questions go unanswered. This regime is cheap because it says little.

The question table separates the three regimes by this course’s own measure. The manually updated document answers four of seven questions correctly and two incorrectly: it confidently states the module count and the import-edge count, and both are wrong. An unanswered question sends the reader to the code; a wrongly answered question sends them to the wrong place. Rule claims stay correct in all three regimes: a rule claim that does not hold against the code is wrong about the code, not the document.

Summary

• Documentation drift is a measurable quantity: the import graph extracted from code is compared against the document’s claim; the method is borrowed from the Service Architectures course’s boundary-violation measurement.
• Claims are three types and behave differently: the four compliant changes dropped no rule claim, only staling snapshot claims; the four non-compliant ones dropped four rule claims. In the unupdated document, drift grows from 2 to 10, and only 2 of the 14 claims stay correct.
• Manual update catches 3 of 4 drifts and costs 4 updates; its blind spot is when the touched module and the claim’s subject differ. The generated audit catches all 4, at a cost of 10 updates and 10 warnings; 6 of the warnings are false alarms, and their source is the snapshot claims.
• The seven questions’ answers split by regime: 4 correct and 2 wrong under manual update, 6 correct under the generated audit, 2 correct and 5 unanswered under the rule-only document.

Next Step

Everything measured up to this point looked at the past or the present: a decision made, a boundary drawn, code running. There is one more thing an architect commits to in writing, and it looks at the future — how long this work will take. The next lesson measures estimation: the same set of work is estimated as a single number and as an uncertainty band, then compared against what actually happened. How the deviation is distributed, how much of it the band actually covers, and whether splitting the estimate into pieces shrinks the deviation.