Technology Evaluation

The previous lesson measured how a decision would be narrated and held the decision’s content fixed: loan records would move to a continuous stream. Which tool would carry that stream was not chosen. In the regional library network’s model, this choice is made once and lives a long time; the loan services are written in-house, but the tool carrying the stream comes from outside.

This lesson does not name candidates by product. Writing a product name would date this lesson within five years; instead, the candidates are modeled with descriptive features — A, B, and C, each a maturity profile, an ecosystem profile, and a built-in capability set. Three things get measured: maturity, ecosystem, and exit cost. The third stands apart from the others, because it alone can be read directly from real code.

Exit Cost Is Read from the Source Tree

Maturity and ecosystem are properties belonging to the candidate itself and are given from outside. Exit cost is not: it is not a property of the candidate, but of how the candidate enters our own code. That is why it can be measured.

The module set below is actually written to disk and then scanned. Numbers are read from the files. The loan service’s model has twelve modules, and each one needs a subset of seven capabilities. If a candidate provides a needed capability natively, the module links to that candidate; if it does not, the module hand-writes that capability and does not link to the candidate.

AP13: exit cost = link lines × 0.5 hours + linked modules × 2 hours + records to migrate / 50,000. The three terms count separate jobs: rewriting the call, retesting the module, and moving the data. Records to migrate is 1,200,000 in the model, and it is the same no matter which candidate is chosen.

AP14: the candidates’ age, breaking-release frequency, open-defect closure time, adopter count, documented-function ratio, and independent-implementation count are model values; none is taken from a real tool, and they exist to show how the metrics get weighted.

// evaluation/code.mjs — model measuring candidate selection on real code. The three candidates
// are not a real product, they are a descriptive feature set. The module set is actually written
// to disk, and exit cost is measured by scanning the written files; numbers are read from files, never typed by hand.
import { mkdirSync, writeFileSync, readFileSync, rmSync } from "node:fs";

export const CAPABILITIES = ["publish", "subscribe", "batch", "transaction", "delay", "replay", "deadletter"];

// The twelve modules in the loan service's model and the capabilities each one needs.
export const MODULES = {
  "api": ["publish", "subscribe"],
  "renewal": ["publish"],
  "overdue": ["publish", "delay"],
  "reservation": ["publish", "subscribe", "transaction"],
  "notification": ["subscribe", "delay", "deadletter"],
  "report": ["subscribe", "batch"],
  "branch-transfer": ["publish", "transaction"],
  "penalty": ["publish", "deadletter"],
  "membership": ["subscribe"],
  "catalog-bridge": ["publish", "subscribe", "replay"],
  "log": ["batch", "replay"],
  "metrics": ["subscribe", "batch"],
};

// AP14: candidates are known only by their properties. "support" is what a candidate provides
// natively; every capability it does not provide is hand-written in the module.
export const CANDIDATES = {
  A: { support: ["publish", "subscribe"], age: 11, breakingReleaseFreq: 0.4, defectCloseDays: 21,
       adopters: 9, documentedRatio: 0.95, independentImpls: 3 },
  B: { support: ["publish", "subscribe", "batch", "deadletter"], age: 6, breakingReleaseFreq: 1.1, defectCloseDays: 9,
       adopters: 38, documentedRatio: 0.92, independentImpls: 1 },
  C: { support: CAPABILITIES, age: 2, breakingReleaseFreq: 3.0, defectCloseDays: 4,
       adopters: 41, documentedRatio: 0.55, independentImpls: 1 },
};

// Produces one module as a real source file.
function moduleText(name, needed, support, clientPath, wrapperPath = null) {
  const native = needed.filter((c) => support.includes(c));
  const manual = needed.filter((c) => !support.includes(c));
  const s = [];
  if (native.length > 0) s.push(`import { client } from "${wrapperPath ?? clientPath}";`);
  for (const c of manual) {
    s.push(`function manual_${c}(records) {`);
    s.push(`  const queue = [];`);
    s.push(`  for (const r of records) queue.push({ name: "${c}", r });`);
    s.push(`  return queue;`);
    s.push(`}`);
  }
  s.push(`export async function ${name.replace(/-/g, "_")}(records) {`);
  for (const c of native) s.push(`  await client.${c}(records);`);
  for (const c of manual) s.push(`  manual_${c}(records);`);
  s.push(`  return records.length;`);
  s.push(`}`);
  return s.join("\n") + "\n";
}

// Wrapper: exposes only the capability set it is given. Writes call forwarding and error
// translation for each capability.
function wrapperText(scope) {
  const s = [`import { client as raw } from "../candidate/client.mjs";`, `export const client = {`];
  for (const c of scope) {
    s.push(`  ${c}: async (records) => {`);
    s.push(`    try { return await raw.${c}(records); }`);
    s.push(`    catch (e) { throw new Error("message broker error: " + e.message); }`);
    s.push(`  },`);
  }
  s.push(`};`);
  return s.join("\n") + "\n";
}

// The produced files are scanned: the modules linked to the candidate's client and the linking lines are counted.
function scan(root, moduleNames, extraFiles = []) {
  let linkedModules = 0, linkLines = 0, totalLines = 0, manualFuncs = 0;
  const files = [...moduleNames.map((a) => `${root}/${a}.mjs`), ...extraFiles];
  for (const path of files) {
    const lines = readFileSync(path, "utf8").trimEnd().split("\n");
    totalLines += lines.length;
    const raw = lines.some((s) => /from "\.\.\/candidate\/client\.mjs"/.test(s));
    if (raw) linkedModules += 1;
    if (raw) linkLines += lines.filter((s) => /(client|raw)\.\w+\(/.test(s)).length;
    manualFuncs += lines.filter((s) => /^function manual_/.test(s)).length;
  }
  return { linkedModules, linkLines, totalLines, manualFuncs };
}

// Produces and measures a candidate's direct use.
export function direct(root, candidate) {
  rmSync(root, { recursive: true, force: true });
  mkdirSync(`${root}/candidate`, { recursive: true });
  mkdirSync(`${root}/loan`, { recursive: true });
  writeFileSync(`${root}/candidate/client.mjs`, `export const client = {};\n`);
  for (const [name, needed] of Object.entries(MODULES))
    writeFileSync(`${root}/loan/${name}.mjs`, moduleText(name, needed, candidate.support, "../candidate/client.mjs"));
  return scan(`${root}/loan`, Object.keys(MODULES));
}

// The same candidate used behind a wrapper. A module that needs a capability the wrapper does not
// cover has to link straight to the client: that is the leak.
export function wrapped(root, candidate, scope) {
  rmSync(root, { recursive: true, force: true });
  mkdirSync(`${root}/candidate`, { recursive: true });
  mkdirSync(`${root}/shared`, { recursive: true });
  mkdirSync(`${root}/loan`, { recursive: true });
  writeFileSync(`${root}/candidate/client.mjs`, `export const client = {};\n`);
  writeFileSync(`${root}/shared/wrapper.mjs`, wrapperText(scope));
  let leaked = 0;
  for (const [name, needed] of Object.entries(MODULES)) {
    const native = needed.filter((c) => candidate.support.includes(c));
    const outside = native.filter((c) => !scope.includes(c));
    if (outside.length > 0) leaked += 1;
    const path = outside.length > 0 ? "../candidate/client.mjs" : "../shared/wrapper.mjs";
    writeFileSync(`${root}/loan/${name}.mjs`,
      moduleText(name, needed, candidate.support, "../candidate/client.mjs", path));
  }
  const wrapperLines = readFileSync(`${root}/shared/wrapper.mjs`, "utf8").trimEnd().split("\n").length;
  return { ...scan(`${root}/loan`, Object.keys(MODULES), [`${root}/shared/wrapper.mjs`]),
    leaked, wrapperLines };
}

// AP13: exit cost = linking call line x 0.5 hours + linked module x 2 hours testing
// + records to migrate / 50,000 hours. The data share is independent of the candidate.
export const RECORDS_TO_MIGRATE = 1_200_000;
export const exitCost = (o) =>
  o.linkLines * 0.5 + o.linkedModules * 2 + RECORDS_TO_MIGRATE / 50_000;

Weight Determines the Winner

The metrics are in different units: years, days, counts, ratios, hours. For them to be summable, all are squeezed between zero and one, and a high value always means good. Everything after that is a weighting decision, and a weight is not a technical number — it is a declaration of preference.

// evaluation/selection.mjs — maturity, ecosystem, and exit cost are weighted
import { CANDIDATES, MODULES, CAPABILITIES, direct, wrapped, exitCost, RECORDS_TO_MIGRATE }
  from "./code.mjs";

const ROOT = "./generated";
const measured = {};
for (const [name, candidate] of Object.entries(CANDIDATES)) measured[name] = direct(`${ROOT}/${name}`, candidate);

console.log(`modules ${Object.keys(MODULES).length}, capabilities ${CAPABILITIES.length}, records to migrate ${RECORDS_TO_MIGRATE}\n`);
console.log("candidate  support  linked modules  link lines  handwritten funcs  total lines  exit (hours)");
console.log("--------- -------- --------------- ----------- ------------------ ------------ -------------");
for (const [name, candidate] of Object.entries(CANDIDATES)) {
  const o = measured[name];
  console.log(`${name.padStart(9)} ${`${candidate.support.length}/7`.padStart(8)} ${String(o.linkedModules).padStart(15)} ` +
    `${String(o.linkLines).padStart(11)} ${String(o.manualFuncs).padStart(18)} ` +
    `${String(o.totalLines).padStart(12)} ${exitCost(o).toFixed(1).padStart(13)}`);
}

// Every metric is squeezed into the 0-1 range; high always means good.
const metric = {
  age: (a) => Math.min(1, a.age / 10),
  breakingReleaseFreq: (a) => 1 / (1 + a.breakingReleaseFreq),
  defectCloseDays: (a) => 1 / (1 + a.defectCloseDays / 10),
  independentImpls: (a) => Math.min(1, a.independentImpls / 3),
  adopters: (a) => Math.min(1, a.adopters / 40),
  documentedRatio: (a) => a.documentedRatio,
  coverage: (a) => a.support.length / CAPABILITIES.length,
  exit: (a, name) => 1 / (1 + exitCost(measured[name]) / 50),
  fewLines: (a, name) => 1 / (1 + measured[name].totalLines / 100),
};

const unit = Object.fromEntries(Object.keys(metric).map((k) => [k, 1]));
const WEIGHTS = {
  "maturity-weighted": { ...unit, age: 3, breakingReleaseFreq: 3, defectCloseDays: 3, independentImpls: 3 },
  "ecosystem-weighted": { ...unit, adopters: 3, documentedRatio: 3, coverage: 3 },
  "exit-weighted": { ...unit, exit: 6 },
  "development-weighted": { ...unit, coverage: 3, fewLines: 4 },
  "equal": { ...unit },
};

const score = (name, w) => {
  let t = 0, sum = 0;
  for (const [k, f] of Object.entries(metric)) { t += w[k] * f(CANDIDATES[name], name); sum += w[k]; }
  return t / sum;
};

console.log(`\nweight set               A       B       C   winner   margin`);
console.log(`--------------------- ------- ------- ------- ------- ------`);
for (const [name, w] of Object.entries(WEIGHTS)) {
  const p = Object.fromEntries(Object.keys(CANDIDATES).map((k) => [k, score(k, w)]));
  const s = Object.entries(p).sort((x, y) => y[1] - x[1]);
  console.log(`${name.padEnd(21)} ${p.A.toFixed(4).padStart(7)} ${p.B.toFixed(4).padStart(7)} ` +
    `${p.C.toFixed(4).padStart(7)} ${s[0][0].padStart(7)} ${(s[0][1] - s[1][1]).toFixed(4).padStart(6)}`);
}

// The wrapper can cover the candidates' common intersection; every capability outside the intersection is a leak.
const intersection = CAPABILITIES.filter((c) => Object.values(CANDIDATES).every((a) => a.support.includes(c)));
console.log(`\ncandidates' shared capability intersection: ${intersection.join(", ")} (${intersection.length}/7)`);
console.log(`\nwrapper scope if candidate B is chosen:`);
console.log("scope                                 wrapper lines  leaked modules  linked modules  link lines  exit (hours)  decrease");
console.log("--------------------------------------- -------------- --------------- --------------- ----------- ------------- --------");
const raw = exitCost(measured.B);
for (const scope of [intersection, ["publish", "subscribe", "batch"], CANDIDATES.B.support]) {
  const o = wrapped(`${ROOT}/wrapper`, CANDIDATES.B, scope);
  const m = exitCost(o);
  console.log(`${scope.join(",").padEnd(39)} ${String(o.wrapperLines).padStart(14)} ` +
    `${String(o.leaked).padStart(15)} ${String(o.linkedModules).padStart(15)} ${String(o.linkLines).padStart(11)} ` +
    `${m.toFixed(1).padStart(13)} ${`${((1 - m / raw) * 100).toFixed(1)}%`.padStart(8)}`);
}
const full = wrapped(`${ROOT}/wrapper`, CANDIDATES.B, CANDIDATES.B.support);
const data = RECORDS_TO_MIGRATE / 50_000;
const gain = raw - exitCost(full);
console.log(`\nexit for B: ${raw.toFixed(1)} hours unwrapped, ${exitCost(full).toFixed(1)} hours with a full-scope wrapper`);
console.log(`code share ${(raw - data).toFixed(1)} -> ${(exitCost(full) - data).toFixed(1)} hours; data share ${data.toFixed(1)} hours the same in both cases`);
console.log(`AP15: the wrapper's ${full.wrapperLines} lines require ${(full.wrapperLines * 0.1).toFixed(1)} hours of maintenance per year`);
console.log(`break-even switch probability = annual maintenance / gain = ${(full.wrapperLines * 0.1 / gain).toFixed(3)}`);
const missing = CAPABILITIES.filter((c) => !CANDIDATES.A.support.includes(c));
console.log(`had A been chosen, ${measured.A.manualFuncs} hand-written functions would have been needed; the same functions would be written ${missing.length} times behind a wrapper`);

modules 12, capabilities 7, records to migrate 1200000

candidate  support  linked modules  link lines  handwritten funcs  total lines  exit (hours)
--------- -------- --------------- ----------- ------------------ ------------ -------------
        A      2/7              11          14                 11          127          53.0
        B      4/7              12          19                  6          103          57.5
        C      7/7              12          25                  0           73          60.5

weight set               A       B       C   winner   margin
--------------------- ------- ------- ------- ------- ------
maturity-weighted      0.6763  0.5416  0.4749       A 0.1347
ecosystem-weighted     0.5563  0.6812  0.6785       B 0.0026
exit-weighted          0.5608  0.5472  0.5243       A 0.0136
development-weighted   0.5226  0.5683  0.6294       C 0.0612
equal                  0.6026  0.5928  0.5642       A 0.0098

candidates' shared capability intersection: publish, subscribe (2/7)

wrapper scope if candidate B is chosen:
scope                                 wrapper lines  leaked modules  linked modules  link lines  exit (hours)  decrease
--------------------------------------- -------------- --------------- --------------- ----------- ------------- --------
publish,subscribe                                   11               5               6          11          41.5    27.8%
publish,subscribe,batch                             15               2               3           7          33.5    41.7%
publish,subscribe,batch,deadletter                  19               0               1           4          28.0    51.3%

exit for B: 57.5 hours unwrapped, 28.0 hours with a full-scope wrapper
code share 33.5 -> 4.0 hours; data share 24.0 hours the same in both cases
AP15: the wrapper's 19 lines require 1.9 hours of maintenance per year
break-even switch probability = annual maintenance / gain = 0.064
had A been chosen, 11 hand-written functions would have been needed; the same functions would be written 5 times behind a wrapper

Every number in the first table is read from files written to disk; the second table is a computation that takes those numbers as input.

Built-in Capability Is Both a Saving and a Bond

The first table shows a single trend running in two directions at once. As built-in capability rises from 2/7 to 7/7, total lines drop from 127 to 73 (a 42 percent decrease) and hand-written functions drop from 11 to 0. In the same direction, exit cost rises from 53.0 hours to 60.5 hours and link lines climb from 14 to 25.

Two columns are counting the same event. Every capability a candidate provides is both a function not written and a bond established. With the richest candidate, no code gets written at all; on the way out, twenty-five lines remain to be rewritten. With the narrowest candidate, eleven functions get hand-written; those functions travel with you when you leave, because they are not linked to anything. A rich tool does not leave an expensive exit because it speeds up development — it leaves one in exact proportion to how much it does.

The Winner Depends on the Weights

The second table has five weight sets and three different winners. When maturity is weighted, A wins; when ecosystem is weighted, B wins; when development load is weighted, C wins. The candidates did not change; only the metrics’ multipliers changed.

The margin column carries a second warning. In the ecosystem-weighted row, B beats C by a margin of 0.0026; in the maturity-weighted row, the margin is 0.1347 — more than fifty times as much. In the first row, the winner is a result; in the second, it is a rounding remainder. An evaluation table that produces only a single winner’s name is incomplete; it has to write down the margin too, because a small margin means defining one metric slightly differently would change the winner.

In the exit-weighted row, the winner does not change. The reason shows up in the last section: most of exit cost is independent of the candidate.

What the Wrapper Costs

The third table measures the wrapper layer put in place to cheapen the exit if the choice turns out wrong. A wrapper exposes only the capability it is given; a module that needs a capability outside that scope has to link straight to the tool. This is a leak, and it is countable.

The three candidates’ shared capability intersection is 2/7. If the wrapper is meant to be genuinely portable, its scope can be at most this intersection — and at that scope, 5 modules leak, and exit cost drops from 57.5 hours to only 41.5 hours (27.8 percent). When the scope is widened to the chosen candidate’s own capability set, the leak drops to zero and exit falls to 28.0 hours (51.3 percent) — but now, because the wrapper exposes the chosen candidate’s capabilities, switching to a different candidate requires hand-writing those capabilities behind the wrapper. The last row measures this: using a narrow candidate directly needs 11 hand-written functions; the same functions get written 5 times behind a wrapper.

The real limit is in the separation between the code share and the data share. The wrapper cuts the code share from 33.5 hours to 4.0 hours; the data share, 24.0 hours, stays the same in both cases, because 1,200,000 records will need to move no matter which tool is switched to. The most the wrapper can reduce is about half of the total exit cost.

The cost itself is also a number. AP15: every line of the wrapper needs 0.1 hours of maintenance per year. Nineteen lines comes to 1.9 hours a year; the gain is 29.5 hours. The wrapper pays for itself if the annual probability of switching tools is greater than 0.064. This threshold ties whether the wrapper gets written not to a guess, but to a comparison.

Summary

• A candidate is described by maturity and ecosystem metrics, not by product name; exit cost is a property of how the candidate enters the code, not of the candidate itself, and is measured by writing a twelve-module source tree to disk and scanning it (AP13, AP14).
• As built-in capability rises from 2/7 to 7/7, total lines drop from 127 to 73 and hand-written functions from 11 to 0; at the same time, exit cost rises from 53.0 hours to 60.5 hours. Every capability provided is both a function not written and a bond established.
• Five weight sets produced three different winners. The winner is a result of the weights, not the candidates; the margin is 0.0026 in the ecosystem-weighted row and 0.1347 in the maturity-weighted row — the first margin is not a result, and the table must write down the margin alongside the winner.
• The wrapper’s scope is limited by the candidates’ shared intersection (2/7); at that scope, 5 modules leak and exit drops by only 27.8 percent. Full scope zeroes out the leak but binds the wrapper to the chosen candidate.
• The wrapper cuts exit’s code share from 33.5 hours to 4.0 hours and never reduces the 24.0-hour data share at all; measured against a nineteen-line maintenance load, it pays for itself if the annual switch probability is greater than 0.064 (AP15).

Next Step

Every number in this lesson depends on a moment in time. A candidate’s age, breaking-release frequency, adopter count, and documented-function ratio are true on the day they are measured; the weights, too, reflect that day’s priorities. The decision gets made with this table, and the table stays put — but the numbers it rests on do not. The next lesson, the course’s last, turns this into a model: a decision’s basis has a validity period, a decision resting on a stale basis has a rate of turning out wrong, and the share allocated to learning lowers that rate. Its question is: if knowledge freshness is not a virtue but a budget allocation, how does that budget get allocated?