Delivery Metrics

The previous lesson read the four metrics as a result of the map: the map changed, and the metrics followed. But the metrics are not independent results; they form a set, and the same decision moves all of them at once. This lesson reads that set from a single run.

The four metrics are these. Deployment frequency: the number of releases reaching production per unit of time. Lead time: the time from a change entering the flow to taking permanent hold in production. Change failure rate: the share of changes that produce an error in production, that is, that fail. Time to restore: the time from an error surfacing in production to the service recovering. In the Process, Team and Delivery course, delivery time was the time a work item spent from start to finish in a team’s flow; the lead time here waits for the change to take permanent hold in production — for a rolled-back change, the clock does not stop.

Building the Set

The example is again the fictional regional measurement network; the network and the release arrangement are both fictional. CF19: sixty changes enter the flow four steps apart. CF20: two unconditional dice are rolled for every change — whether it is defective (rate 0.20) and whether the defect would be caught at the gate (rate 0.60); the seed is 20260804 and the roll order does not change from run to run. CF21: changes are collected into a batch; prep starts once the batch is full or the first item has waited 30 steps. One release is prepared at a time, and prep takes 4 plus the batch size steps. CF22: if there is a gate, it adds 5 steps to prep and runs before release; a defect it catches is pulled from the batch, fixed in 6 steps, and re-enters the queue. CF23: if a release ships an uncaught defect, it is an error in production; without automatic rollback, sound changes stay in production, and the defective one returns to the queue after 12 steps of detection and 6 of fixing, with time to restore counted until the fix ships. CF24: with automatic rollback, the whole release is rolled back with 3 steps of detection and 4 of rollback; time to restore is 7 steps, but every change in the batch returns to the queue. CF25: a fixed change never becomes defective again.

The release arrangement and the decisions are a process-internal model; there is no real release record.

// metric/set.mjs — the model of the four delivery metrics as a linked set (model): there is no real
// release record, steps are a data structure. Randomness is written with a generator, the seed is visible.

export const N = 60, ARRIVAL = 4, LIMIT = 4000, SEED = 20260804;
export const DEFECT = 0.20, GATE_CATCH = 0.60;     // defect rate and the gate's catch rate
export const PREP0 = 4, PREP1 = 1, GATE = 5;       // prep = PREP0 + PREP1 * batch (+ gate)
export const DETECT = 12, FIX = 6;                 // manual detection and fix
export const AUTO_DETECT = 3, ROLLBACK = 4;        // automatic detection and rollback
export const MAX_WAIT = 30;                        // a half-batch waiting this long still ships

export function rng(seed) {                        // linear congruential generator
  let x = seed >>> 0;
  return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32);
}

export function run({ batch = 8, gate = false, automatic = false } = {}) {
  const roll = rng(SEED);
  const D = Array.from({ length: N }, (_, i) => ({ i, arrival: i * ARRIVAL, ready: i * ARRIVAL,
    status: "in-transit", defective: roll() < DEFECT, catchable: roll() < GATE_CATCH,
    finishAt: null, errorAt: null, recovery: null, rolledBackCount: 0, queuedAt: 0 }));
  const S = { releases: 0, brokenReleases: 0, caughtAtGate: 0, rollbacks: 0, prodErrors: 0,
    batchTotal: 0, rolledBackDefective: 0, rolledBackInnocent: 0 };
  let active = null;

  const releaseFinish = (rel, t) => {
    let group = rel.group;
    if (gate) {                                    // a gate catches part of the defects before release
      const caught = group.filter((d) => d.defective && d.catchable);
      S.caughtAtGate += caught.length;
      for (const d of caught) { d.status = "in-transit"; d.ready = t + FIX; d.defective = false; }
      group = group.filter((d) => !caught.includes(d));
    }
    S.releases++; S.batchTotal += group.length;
    const broken = group.filter((d) => d.defective);
    if (!broken.length) { for (const d of group) { d.status = "done"; d.finishAt = t; } return; }
    S.brokenReleases++; S.prodErrors += broken.length;
    if (!automatic) {                              // an innocent change stays in production
      for (const d of group) {
        if (!d.defective) { d.status = "done"; d.finishAt = t; continue; }
        d.errorAt = t; d.defective = false; d.status = "in-transit"; d.ready = t + DETECT + FIX;
      }
      return;
    }
    S.rollbacks++;                                 // the whole release is rolled back, innocents too
    for (const d of group) {
      const wasDefective = d.defective;
      if (wasDefective) { d.errorAt = t; d.defective = false; d.recovery = AUTO_DETECT + ROLLBACK; S.rolledBackDefective++; }
      else S.rolledBackInnocent++;
      d.rolledBackCount++; d.status = "in-transit";
      d.ready = t + AUTO_DETECT + ROLLBACK + (wasDefective ? FIX : 0);
    }
  };

  for (let t = 0; t <= LIMIT; t++) {
    for (const d of D) if (d.status === "in-transit" && d.ready <= t) { d.status = "queued"; d.queuedAt = t; }
    if (active && active.finishAt === t) { releaseFinish(active, t); active = null; }
    if (active) continue;                          // one release prepared at a time
    const K = D.filter((d) => d.status === "queued").sort((a, b) => a.queuedAt - b.queuedAt || a.i - b.i);
    if (!K.length) continue;
    if (K.length < batch && t - K[0].queuedAt < MAX_WAIT) continue;
    const group = K.slice(0, batch);
    for (const d of group) d.status = "releasing";
    active = { group, finishAt: t + PREP0 + PREP1 * group.length + (gate ? GATE : 0) };
  }
  for (const d of D) if (d.errorAt !== null && d.recovery === null && d.finishAt !== null)
    d.recovery = d.finishAt - d.errorAt;            // manual recovery: until the fix ships
  return { D, S };
}

export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-");

export function measure(config) {
  const { D, S } = run(config);
  const windowEnd = Math.max(...D.map((d) => d.finishAt ?? 0));
  const recoveries = D.filter((d) => d.recovery !== null).map((d) => d.recovery);
  return { D, S, windowEnd,
    deployFrequency: (100 * S.releases / windowEnd).toFixed(1), releases: S.releases,
    avgBatch: (S.batchTotal / S.releases).toFixed(1),
    leadTime: avg(D.map((d) => d.finishAt - d.arrival)),
    releaseFailureRate: (100 * S.brokenReleases / S.releases).toFixed(0) + "%",
    changeFailureRate: (100 * S.prodErrors / N).toFixed(0) + "%",
    recovery: avg(recoveries), prodErrors: S.prodErrors,
    caughtAtGate: S.caughtAtGate, rolledBack: D.reduce((s, d) => s + d.rolledBackCount, 0),
    rolledBackDefective: S.rolledBackDefective, rolledBackInnocent: S.rolledBackInnocent, rollbacks: S.rollbacks,
    unfinished: D.filter((d) => d.finishAt === null).length };
}

// metric/measure.mjs — the change a decision measures across four metrics; the decision that improves one metric while breaking another.
import { N, SEED, DEFECT, measure, avg } from "./set.mjs";

const print = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const K = {
  "baseline (batch 8)": {}, "batch 8 -> 2": { batch: 2 }, "batch 8 -> 1": { batch: 1 },
  "gate added": { gate: true }, "automatic rollback": { automatic: true },
  "batch 2 + automatic": { batch: 2, automatic: true },
  "batch 2 + gate + automatic": { batch: 2, gate: true, automatic: true },
};
const R = Object.fromEntries(Object.entries(K).map(([ad, a]) => [ad, measure(a)]));
const T = R["baseline (batch 8)"];

console.log(`${N} changes, defect rate ${DEFECT}, seed ${SEED}; ` +
  `defective ${T.D.filter((d) => d.errorAt !== null).length}, ` +
  `unfinished ${Object.values(R).reduce((s, r) => s + r.unfinished, 0)}`);

const A = [-27, 11, 11, 16, 16, 11];
console.log("\n1. four metrics as a linked set");
print(A, "run", "deploy", "lead time", "change failure", "change failure", "recovery");
print(A, "", "(100 steps)", "(steps)", "(release)", "(change)", "(steps)");
for (const [ad, r] of Object.entries(R))
  print(A, ad, r.deployFrequency, r.leadTime, r.releaseFailureRate, r.changeFailureRate, r.recovery);

const B = [-27, 9, 12, 13, 16, 14];
console.log("\n2. the decision's mechanics, in the same runs");
print(B, "run", "releases", "avg. batch", "prod errors", "caught at gate", "rolled back");
for (const [ad, r] of Object.entries(R))
  print(B, ad, r.releases, r.avgBatch, r.prodErrors, r.caughtAtGate, r.rolledBack);

const C = [-27, 11, 12, 24, 12];
console.log("\n3. raw difference from baseline (direction reads separately per metric)");
print(C, "decision", "deploy", "lead time", "change failure/release", "recovery");
const dev = (r, k) => {
  const d = +r[k].replace("%", "") - +T[k].replace("%", "");
  return (d > 0 ? "+" : "") + d.toFixed(1);
};
for (const ad of Object.keys(R).slice(1))
  print(C, ad, dev(R[ad], "deployFrequency"), dev(R[ad], "leadTime"),
    dev(R[ad], "releaseFailureRate") + " points", dev(R[ad], "recovery"));

const E = [-27, 9, 18, 12, 10, 26];
console.log("\n4. innocent changes in a rolled-back release");
print(E, "run", "rollbacks", "change-instances", "defective", "innocent", "innocent items' lead time");
for (const ad of ["automatic rollback", "batch 2 + automatic", "batch 2 + gate + automatic"]) {
  const r = R[ad], g = r.D.filter((d) => d.rolledBackCount > 0 && d.errorAt === null);
  print(E, ad, r.rollbacks, r.rolledBack, r.rolledBackDefective, r.rolledBackInnocent,
    `${avg(g.map((d) => d.finishAt - d.arrival))} (baseline ${T.leadTime})`);
}

60 changes, defect rate 0.2, seed 20260804; defective 12, unfinished 0

1. four metrics as a linked set
run                             deploy  lead time  change failure  change failure   recovery
                           (100 steps)    (steps)       (release)        (change)    (steps)
baseline (batch 8)                 3.2       34.6             80%             20%       45.4
batch 8 -> 2                      13.5       14.9             31%             20%       26.3
batch 8 -> 1                      20.0       69.7             17%             20%       65.8
gate added                         3.1       36.8             40%              7%       54.8
automatic rollback                 5.1       50.6             50%             20%        7.0
batch 2 + automatic               15.4       24.1             27%             20%        7.0
batch 2 + gate + automatic         9.0      109.3             11%              7%        7.0

2. the decision's mechanics, in the same runs
run                         releases  avg. batch  prod errors  caught at gate   rolled back
baseline (batch 8)                10         7.2           12               0             0
batch 8 -> 2                      36         2.0           12               0             0
batch 8 -> 1                      72         1.0           12               0             0
gate added                        10         6.4            4               8             0
automatic rollback                16         7.5           12               0            60
batch 2 + automatic               41         2.0           12               0            22
batch 2 + gate + automatic        38         1.8            4               8             8

3. raw difference from baseline (direction reads separately per metric)
decision                        deploy   lead time  change failure/release    recovery
batch 8 -> 2                     +10.3       -19.7            -49.0 points       -19.1
batch 8 -> 1                     +16.8       +35.1            -63.0 points       +20.4
gate added                        -0.1        +2.2            -40.0 points        +9.4
automatic rollback                +1.9       +16.0            -30.0 points       -38.4
batch 2 + automatic              +12.2       -10.5            -53.0 points       -38.4
batch 2 + gate + automatic        +5.8       +74.7            -69.0 points       -38.4

4. innocent changes in a rolled-back release
run                        rollbacks  change-instances   defective  innocent innocent items' lead time
automatic rollback                 8                60          12        48      64.5 (baseline 34.6)
batch 2 + automatic               11                22          12        10      35.8 (baseline 34.6)
batch 2 + gate + automatic         4                 8           4         4     163.3 (baseline 34.6)

The numbers are of the measurement kind; their inputs are the assumptions above.

Shrinking Batch Size: The Metric That Moves, the Ground That Does not

At baseline the batch is eight: ten releases, an average of 7.2 changes, deployment frequency 3.2, lead time 34.6 steps, recovery 45.4 steps. The release failure rate is 80% — eight of the ten releases carry a defect.

Once the batch shrinks to two, all four metrics improve relative to baseline: deployment frequency goes to 13.5, lead time to 14.9 steps, recovery to 26.3 steps, the release failure rate to 31%. The second table underlines this: the number of errors surfacing in production stays at 12. The change failure rate is also 20% — the same as baseline. The release failure rate falling from 80% to 31% is not a quality gain, it is the same twelve errors spread across more releases. A metric improves by forty-nine points while nothing on the ground has changed.

Shrinking the batch to one shows this limit. Deployment frequency climbs to 20.0, the number of releases to 72, and the release failure rate falls to 17% — the best value on three metrics. But lead time climbs from 34.6 to 69.7 steps and recovery from 45.4 to 65.8 steps. The reason is the fixed share of prep: a fixed cost of 4 steps per release becomes 5 steps per change at batch one, and the arrival interval is 4 steps. The flow saturates. The gain from shrinking batch size ends where the fixed release cost runs out.

Adding a Gate

The gate is the only decision that genuinely reduces errors in production: eight of the twelve defects are caught before release, four get through, and the change failure rate falls from 20% to 7%. This is the one improvement that does not come from the metric’s own definition.

The bill is in the others. Since 5 steps are added to every release, lead time goes from 34.6 to 36.8 steps and deployment frequency from 3.2 to 3.1. Time to restore also gets worse: from 45.4 to 54.8 steps. The four defects that get past the gate have their fix pass through the same gate too. Every gate placed on a release is also a gate on the fix.

Automatic Rollback and the Innocent Change

Automatic rollback brings time to restore from 45.4 down to 7.0 steps — the single largest gain across the four metrics (−38.4). Its cost is in the fourth table: across eight rollback events, sixty change-instances are rolled back, and forty-eight of them are innocent. The innocent changes’ lead time is 64.5 steps; baseline is 34.6. Overall lead time climbs from 34.6 to 50.6 steps.

The cost depends on batch size. Once the batch is shrunk to two and automatic rollback is turned on, the forty-eight rolled-back innocent changes fall to ten, the innocents’ lead time pulls in to 35.8 steps, and the set balances out: deployment 15.4, lead time 24.1, recovery 7.0. The size of the rollback unit is the cost of the rollback.

Applying all three decisions at once breaks this. Batch two, the gate, and automatic rollback together bring the release failure rate to 11% and recovery to 7.0 steps, but push lead time to 109.3 steps: for two changes, 4 plus 2 plus 5, that is 11 steps of prep, comes to 5.5 steps per change, and the arrival interval is 4 steps. The sum of three decisions that each improve a separate metric produces the worst lead time in the set.

Where the Difference Hides

In this lesson, the difference hides in the metric’s own definition, and its number is this: across the same seven runs, the release-based failure rate swings between 11% and 80%, the change-based rate takes only two values (20% and 7%), and the number of errors surfacing in production is 12 in five of the seven runs. The declared part is the rate itself; the undeclared part is the denominator. Seven runs and four metrics give twenty-eight measurements: in four of the six decisions at least one metric gets worse, in two (batch 2, and batch 2 plus automatic) all four improve — and in both of those two, the number of errors on the ground stays fixed at 12.

How many steps it takes the signal to reach whom also changes with these decisions: with automatic rollback, a defect is detected in 3 steps and closes in 7; detected manually, it takes 12 steps and averages 45.4 steps until the fix ships. The Testing Process and Automation Infrastructure course’s quality indicators measured a change’s content; the four metrics here measure the delivery itself, and only two of them connect at all, and only through the defect rate.

Summary

• The four metrics are read from a single run and are linked: in none of the seven runs does a decision stand alone.
• Shrinking batch size from 8 to 2 takes deployment frequency 3.2 → 13.5, lead time 34.6 → 14.9, recovery 45.4 → 26.3 steps, and the release failure rate 80% → 31%; errors surfacing in production stay at 12 and the change-based rate stays fixed at 20%.
• Shrinking the batch to 1 pushes deployment frequency to 20.0, but the fixed release cost saturates the flow: lead time 69.7, recovery 65.8 steps.
• The gate is the only decision that brings production errors from 12 to 4 (change failure rate 20% → 7%), and in exchange pushes lead time to 36.8 and time to restore to 54.8 steps — the fix passes through the same gate.
• Automatic rollback brings recovery to 7.0 steps; at batch 8 it rolls back 48 innocent changes and carries their lead time to 64.5 steps, at batch 2 that same count falls to 10 and 35.8 steps.
• The sum of the three decisions (batch 2, gate, automatic rollback) brings the release failure rate to 11% while pushing lead time to 109.3 steps.

Next Step

Every one of these decisions was made against a threshold, but the threshold itself was never written down: how much error is acceptable, and what stops when it becomes unacceptable. The next lesson ties that threshold to a number — given a target availability, the error budget minutes left in a period, what each incident takes from the budget, and the constraint placed on deployment frequency once the budget is exhausted are all measured; the budget rule’s effect, measured across these four metrics, is counted.