Platform Metrics

The previous lesson moved five teams in waves and counted who was left behind, by name. Throughout this topic, every lesson counted what the platform did; none of them asked whether any of it actually made teams’ work faster.

The four delivery metrics were defined in M22/K01 and are not repeated here; they are the input. What is measured is not the metrics themselves, but where the gap in the metric comes from. The through-line is again the fictional regional measurement network and its five fictional teams.

Tying Adoption to a Single Number

PC37: each of the eight capabilities has its setup work, annual maintenance, number of teams using it, and usage count written down. PC38: the five teams’ four delivery metrics are read before and after adoption. PC39: three teams have adopted; the order comes from the migration wave, meaning the teams with no obstacle migrated first. PC40: the platform is a dependency; over twelve months, outages, their durations, the number of teams on the platform at that moment, and the source are written down. PC41: if adoption is forced, every team-capability pair brings one unit of compliance work, and only a portion of the forced ones produce usage. PC42: annual support requests per team and the platform team’s capacity are written down.

Capabilities, metrics, and outages are a model built within the lesson; there is no randomness.

// metric/data.mjs — platform metrics model: there is no real metrics pipeline; capabilities,
// team metrics, and outages are a data structure, with no randomness. Metric *definitions* were
// established in M22/K01 and are an input here. Work-days are fictional.

export const TEAM = ["meter-reading", "validation", "billing", "field", "reporting"];
export const ADOPTED = [0, 1, 4];      // comes from the migration wave: teams with no obstacle went first
export const NOT_ADOPTED = [2, 3];     // critical period and incompatible requirement

// capability: name, setup work, annual maintenance (work-days), teams using it, usage in the period
export const CAPABILITY = [
  ["environment on demand", 34, 7, 4, 60], ["scaffold generation", 22, 5, 5, 5],
  ["service catalog", 18, 6, 3, 5], ["policy gate", 27, 8, 5, 72],
  ["showback", 15, 4, 3, 30], ["migration tool", 31, 3, 4, 4],
  ["secret rotation", 19, 5, 0, 0], ["shared dashboard library", 24, 6, 1, 3],
];

// metric: name, unit, direction (+1 higher is better, -1 lower is better), before adoption, after (team order)
export const METRIC = [
  ["deployment frequency", "week", +1, [4.0, 3.5, 1.6, 1.1, 2.8], [8.4, 4.6, 2.4, 1.5, 5.9]],
  ["lead time", "hours", -1, [52, 61, 96, 120, 47], [18, 52, 84, 112, 12]],
  ["failure rate", "%", -1, [14, 11, 18, 21, 9], [6, 9.5, 15.5, 19, 4]],
  ["time to restore", "min", -1, [190, 145, 240, 310, 160], [95, 130, 215, 300, 70]],
];

// outage: month, duration (min), number of teams on the platform at that time, source
export const OUTAGE = [[2, 45, 1, "platform"], [4, 120, 2, "lower layer"],
  [6, 25, 3, "platform"], [8, 210, 3, "platform"], [10, 60, 3, "lower layer"],
  [12, 90, 3, "platform"]];

export const REQUEST_PER_TEAM = 34, CAPACITY = 90;  // annual requests per team / platform capacity
export const COMPLY_WORK = 4, FORCED_USERS = 4, FORCED_USAGE = 3;   // the cost of forcing adoption

export const sum = (v) => v.reduce((a, b) => a + b, 0);
const avg = (v) => sum(v) / v.length;
const group = (g, d, y) => avg(g.map((i) => d[i])) * y;

// for one metric: raw gap, selection bias, difference-in-differences, and the adopting group's own internal spread
export function decompose([, , direction, before, after]) {
  const d = (i) => (after[i] - before[i]) * direction;      // improvement, sign-corrected
  const adoptedDiff = ADOPTED.map(d);
  return { afterGap: group(ADOPTED, after, direction) - group(NOT_ADOPTED, after, direction),
    beforeGap: group(ADOPTED, before, direction) - group(NOT_ADOPTED, before, direction),
    diffInDiff: avg(adoptedDiff) - avg(NOT_ADOPTED.map(d)),
    spread: Math.max(...adoptedDiff) - Math.min(...adoptedDiff) };
}

// metric/measure.mjs — adoption breakdown, decomposition of the four delivery metrics, the
// platform's own metric, and what forcing adoption does to the metric.
import { TEAM, ADOPTED, CAPABILITY, METRIC, OUTAGE, REQUEST_PER_TEAM, CAPACITY, COMPLY_WORK,
  FORCED_USERS, FORCED_USAGE, sum, decompose } from "./data.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const t0 = process.hrtime.bigint();
const WORK = sum(CAPABILITY.map((y) => y[1] + y[2])), USAGE = sum(CAPABILITY.map((y) => y[4]));
const pairs = sum(CAPABILITY.map((y) => y[3])), totalPairs = CAPABILITY.length * TEAM.length;

console.log(`${TEAM.length} teams, ${CAPABILITY.length} capabilities, ${ADOPTED.length} adopters`);

console.log("\n1. adoption: teams and usage per capability");
const A = [-26, 8, 8, 9, 11, 14];
write(A, "capability", "setup", "maint.", "teams", "usage", "usage/work");
for (const [name, k, b, e, u] of CAPABILITY)
  write(A, name, k, b, `${e}/${TEAM.length}`, u, u ? ((k + b) / u).toFixed(2) : "-");
write(A, "total", sum(CAPABILITY.map((y) => y[1])), sum(CAPABILITY.map((y) => y[2])),
  `${pairs}/${totalPairs}`, USAGE, (WORK / USAGE).toFixed(2));
const idle = CAPABILITY.filter((y) => y[3] <= 1), idleWork = sum(idle.map((y) => y[1] + y[2]));
console.log(`  adoption ${(100 * pairs / totalPairs).toFixed(1)}%, ` +
  `${(pairs / CAPABILITY.length).toFixed(2)} teams/capability; least-used ` +
  `${idle.length} capabilities ${idleWork} work-days (${(100 * idleWork / WORK).toFixed(0)}%)`);

console.log("\n2. four delivery metrics: does the gap come from the platform");
const B = [-29, 11, 17, 14, 10, 10];
write(B, "metric", "raw gap", "selection bias", "diff-in-diff", "spread", "separable");
for (const o of METRIC) {
  const r = decompose(o);
  write(B, `${o[0]} (${o[1]})`, r.afterGap.toFixed(2), r.beforeGap.toFixed(2),
    r.diffInDiff.toFixed(2), r.spread.toFixed(2),
    Math.abs(r.diffInDiff) > r.spread ? "yes" : "no");
}
const p = METRIC.map((o) => { const r = decompose(o); return 100 * r.beforeGap / r.afterGap; });
console.log(`  selection bias explains ${Math.min(...p).toFixed(0)}%-` +
  `${Math.max(...p).toFixed(0)}% of the raw gap`);
console.log("  diff-in-diff is smaller than the spread in all four: the remaining share cannot be separated");

console.log("\n3. the platform is a dependency: how many teams does the same outage stop");
const duration = sum(OUTAGE.map((k) => k[1]));
const actual = sum(OUTAGE.map((k) => k[1] * k[2]));
const platformOutages = OUTAGE.filter((k) => k[3] === "platform");
console.log(`  ${OUTAGE.length} outages ${duration} min; platform-caused ${platformOutages.length} outages ` +
  `(${sum(platformOutages.map((k) => k[1]))} min), lower layer ${OUTAGE.length - platformOutages.length} outages`);
const C = [-11, 14, 18, 15, 16];
write(C, "adopters", "distributed", "central team-min", "support req.", "over capacity");
for (const n of [1, 3, 5])
  write(C, n, `${duration} team-min`, n * duration, n * REQUEST_PER_TEAM,
    n * REQUEST_PER_TEAM > CAPACITY ? `${(100 * (n * REQUEST_PER_TEAM / CAPACITY - 1)).toFixed(0)}%` : "-");
console.log(`  actual ${actual} team-min (adoption grew within the period), ` +
  `${(actual / duration).toFixed(1)}x the distributed figure`);

console.log("\n4. what forcing adoption does to the metric");
const forced = totalPairs - pairs, extraWork = forced * COMPLY_WORK;
const extraUsage = FORCED_USERS * FORCED_USAGE;
const D = [-22, 14, 13, 20];
write(D, "arrangement", "adoption", "usage", "work per usage");
write(D, "voluntary", `${(100 * pairs / totalPairs).toFixed(1)}%`, USAGE, (WORK / USAGE).toFixed(2));
write(D, "forced", "100.0%", USAGE + extraUsage, ((WORK + extraWork) / (USAGE + extraUsage)).toFixed(2));
const better = (100 * (totalPairs / pairs - 1)).toFixed(0);
const worse = (100 * ((WORK + extraWork) / (USAGE + extraUsage) / (WORK / USAGE) - 1)).toFixed(0);
const noUsage = forced - FORCED_USERS;
console.log(`  of the ${forced} forced pairs, ${noUsage} produce no usage`);
console.log(`  adoption improves ${better}% while work per usage worsens ${worse}%`);

const ms = Number(process.hrtime.bigint() - t0) / 1e6;   // raw measurement is not printed, the band is audited
console.log(`\nfour tables in the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`);

5 teams, 8 capabilities, 3 adopters

1. adoption: teams and usage per capability
capability                   setup  maint.    teams      usage    usage/work
environment on demand           34       7      4/5         60          0.68
scaffold generation             22       5      5/5          5          5.40
service catalog                 18       6      3/5          5          4.80
policy gate                     27       8      5/5         72          0.49
showback                        15       4      3/5         30          0.63
migration tool                  31       3      4/5          4          8.50
secret rotation                 19       5      0/5          0             -
shared dashboard library        24       6      1/5          3         10.00
total                          190      44    25/40        179          1.31
  adoption 62.5%, 3.13 teams/capability; least-used 2 capabilities 54 work-days (23%)

2. four delivery metrics: does the gap come from the platform
metric                           raw gap   selection bias  diff-in-diff    spread separable
deployment frequency (week)         4.35             2.08          2.27      3.30        no
lead time (hours)                  70.67            54.67         16.00     26.00        no
failure rate (%)                   10.75             8.17          2.58      6.50        no
time to restore (min)             159.17           110.00         49.17     80.00        no
  selection bias explains 48%-77% of the raw gap
  diff-in-diff is smaller than the spread in all four: the remaining share cannot be separated

3. the platform is a dependency: how many teams does the same outage stop
  6 outages 550 min; platform-caused 4 outages (370 min), lower layer 2 outages
adopters      distributed  central team-min   support req.   over capacity
1            550 team-min               550             34               -
3            550 team-min              1650            102             13%
5            550 team-min              2750            170             89%
  actual 1440 team-min (adoption grew within the period), 2.6x the distributed figure

4. what forcing adoption does to the metric
arrangement                 adoption        usage      work per usage
voluntary                      62.5%          179                1.31
forced                        100.0%          191                1.54
  of the 15 forced pairs, 11 produce no usage
  adoption improves 60% while work per usage worsens 18%

four tables in the 2000 ms band: yes

The numbers are of the measurement class; their inputs are the assumptions above.

Adoption Is Three Numbers, Not One

Twenty-five of forty team-capability pairs are filled (62.5%), coming to 3.13 teams per capability. The third column changes the table: work per usage is 0.49 work-days for the policy gate capability and 10.00 work-days for the shared dashboard library capability — a twentyfold difference.

Two capabilities go unrewarded. secret rotation is never used at all: 19 work-days of setup, 5 work-days of maintenance, zero usage. shared dashboard library serves a single team. Together they account for 54 work-days, 23% of total investment — the case where the platform does not pay off is readable here in a single column: the work was done, and nothing came back for it.

Does the Gap Come From the Platform

The raw gap is the difference between the two groups after adoption, and it is large across all four metrics: 4.35 deployments a week, 70.67 hours of lead time, 10.75 points of failure rate, 159.17 minutes of time to restore.

The second column writes down how much of that gap already existed before adoption: of lead time’s 70.67-hour gap, 54.67 of it; of failure rate’s 10.75 points, 8.17. The confounder’s name is selection bias, and its cause was measured in the previous lesson — the teams that migrated to the platform first were the ones with no obstacle, and they were already faster. It explains 48% to 77% of the raw gap.

The third column takes the difference between the two. The difference-in-differences also subtracts the non-adopting group’s own improvement over the same period — clearing out the second confounder, the common trend. What is left is 2.27 for deployment frequency and 16.00 hours for lead time.

The fourth column knocks down even what is left. The spread among the three adopting teams themselves is 3.30 for deployment frequency and 26.00 for lead time; it is bigger than the difference-in-differences on every metric, and the last column says no on all four: the gap is real, but where it comes from cannot be separated out.

The Platform Is a Dependency

The third table counts the platform’s own metric: six outages and 550 minutes over twelve months, four of them platform-caused (370 minutes). The real number is not the duration — it is team-minutes. Under a distributed arrangement, one failure stops one team: 550 team-minutes. Under central adoption, that same 550 minutes gets multiplied by however many teams have adopted — 2,750 team-minutes at five teams. What actually happened is 1,440, 2.6 times the distributed figure.

This is where the abstraction leaks: because a single layer is hidden, a single failure stops five teams at once. The more successful the platform is, the more expensive one of its outages becomes. Support requests also exceed capacity by 89% at five teams.

Where the Metric Points the Wrong Way

The fourth table is the result of targeting the adoption rate. Once forced, the rate climbs from 62.5% to 100%; the fifteen forced pairs bring 60 work-days of compliance work, and eleven of them produce no usage at all. The metric improves 60% while work per usage worsens from 1.31 to 1.54, an 18% decline — the metric moves, the thing it was supposed to measure does not.

The second wrong-way case is in the second table. Migrating the easy teams first is the right call for the migration program, but that same call produces 48% to 77% of the gap in the delivery metrics: an ordering that is efficiency in one place is bias in another.

Summary

• 25 of forty team-capability pairs are filled (62.5%); work per usage ranges from 0.49 to 10.00 work-days, and the two least-used capabilities hold 23% of the investment (54 work-days).
• Across the four delivery metrics, 48%-77% of the raw gap comes from selection bias; the difference-in-differences also clears out the common trend, but stays smaller than the within-group spread on all four — where the gap comes from cannot be separated out.
• The same 550-minute outage is 550 team-minutes under a distributed arrangement and 2,750 team-minutes at five teams; what actually happened is 1,440, 2.6 times the distributed figure, and support requests exceed capacity by 89%.
• Forcing adoption pushes the rate to 100%, and 11 of the 15 forced pairs produce no usage at all; work per usage worsens by 18%.

Course Wrap-Up

Lesson	Hidden layer	How many teams, how much gain	Leak or stepping outside
What Is Platform Engineering	18 items, 666 work-days	threshold 7 teams; at 5, −121 days	leftover 30.6%; exemption in 9/18
Platform as a Product	request→need, 20→14 capabilities	5 teams, 132 work-days	128 days to product teams (97%)
Golden Paths	10 support items, three axes	64 work-days/year; 1/5 fits	field loses all ten items
Standardization and Autonomy Balance	mode choice, 80 work-days	hybrid arrangement +10 work-days	the wall blocks 10 legitimate requirements
Self-Service	request queue, 1,551 requests	6/12 types, 79 work-days	region transfer −32 work-days
On-Demand Environment Provisioning	environment setup steps	336 person-hours; threshold 5 teams	pool full 16/90 days; field outside
Template and Scaffold Generation	19 decisions collapse into 3 answers	5 services, 95→15 answers	4/5 services behind at version 2
Developer Portal and Service Catalog	reverse edge and ownership	32→28.8 min; threshold 2 incidents	6/12 records disagree
Policy and Compliance Automation	the text of 6 rules	5 teams, 830→152 min	4 exemptions, lifespan 3.2x
Cost Visibility	the bill and the accounting pipeline	85 u/day; threshold 3 teams	408 u/day outside audit
Migration Strategies	10 of migration’s 14 steps	4 teams at wave=1, 40 steps	1–3 teams behind, legacy arrangement indefinite
Platform Metrics	the measurement pipeline itself	3 teams, 25/40 pairs (62.5%)	1,440 team-min of outage

In none of the twelve rows is the third column ever without the fourth. In every lesson, the gain was tied to a number, and a cost stood right next to it: the abstraction does not remove the layer it hides — it multiplies its gain by the number of teams, and makes someone pay its cost through leaking or stepping outside. A threshold showed up explicitly in three lessons (7, 5, and 3 teams), and in all three, staying under it made the platform harmful.

Course	Measurement axis	Unit of measure
K01 DevOps Culture and Fundamentals	where the gap is hidden	gap item
K02 Containers	isolation is a budget	byte, layer, resource
K03 Container Orchestration	the cost of delegating a decision	information item
K04 Continuous Integration and Delivery	what each gate catches, what it misses	marginal gate value
K05 Infrastructure as Code	the gap between record and reality	record item
K06 Observability and Operations	not data, a question	cost per question
K07 Servers, Networking and Hardening	the default itself	open-surface item
K08 Platform Engineering	the cost of abstraction	hidden layer

Eight courses told eight different stories, but measured in a single form. The gap item, the open-surface item, the hidden layer — all of them were asking the same three things: a decision, a number, and the cost paid in exchange for that number. In K02, the cost of isolation was measured in bytes; in K04, a gate’s value was measured by the violations it caught; in K08, the value of abstraction was measured by the number of teams it saved. In none of them did “it is good” carry a decision on its own.

Measuring itself was not free either: K06 counted the measuring side as its own line item on the bill; K08 showed that 23% of the measurement investment went unrewarded. Not writing down the unmeasurable as if it were measured is itself a measurement decision — and that is this curriculum’s last line.