Quality Metrics

The previous lesson made a single run’s result readable and resolved that run’s root into a decision. One question a report cannot answer remained: is this team testing better over time. Schema mismatch showing up one night is an event; showing up every week is a process defect, and a single run’s report makes the two look the same. This lesson moves the measure to the period level.

What is measured here is not the system, it is the team. The running system’s response time, error rate, and saturation are the Performance Anti-Patterns and Monitoring course’s subject; the metrics here measure the testing itself. The resource distributed is again attention, and its quantity is the metric count: every metric wants to be collected and read once per period, and in exchange is expected to produce a decision. If it does not, it is noise.

Three Metrics, One Data Set

Defect leakage rate is the ratio of defects that reach production without being caught at any pipeline stage to total defects; the escaped defects in its numerator are the same concept as M21/K03’s escaped real defects. Rework is work redone because of a defect, and it is measured in hours; this is a different concept from the failback the Resilience and Reliability course describes — the return to the primary copy after a takeover. Cycle time is the time from a change being submitted to it entering the main branch; the time up to release is lead time, and this lesson does not measure it.

What the three share is this: all of them come out of a single event set — which defect was caught at which stage. Stage order, feedback minutes, and the defect set are taken from the Testing in Continuous Integration lesson’s three-frequency arrangement.

// period.mjs -- five metrics for two periods over the same defect set
// TP9 (automation-infrastructure/01): 100 changes, 20 workdays. Class names and per-hundred-
// change counts are from the same lesson's TP7 defect set; stage is the place of the team
// catching that class in the three-frequency arrangement. Total 80 defects, 20 clean changes.
export const CLASSES = [
  ['boundary comparison', 13, 'fast'], ['breaking change', 6, 'fast'],
  ['request shape drift', 5, 'fast'], ['field lost from contract', 4, 'fast'],
  ['schema mismatch', 8, 'mid'], ['card status not updating', 7, 'mid'],
  ['status code mapping', 6, 'mid'], ['missing warning on rejected request', 4, 'mid'],
  ['data loss in migration', 3, 'mid'], ['check-then-act race', 2, 'mid'],
  ['distribution-dependent report defect', 2, 'mid'], ['latency regression', 4, 'nightly'],
  ['concatenated query', 3, 'nightly'], ['recovery gap', 1, 'nightly'],
  ['capacity limit', 1, 'nightly'], ['accessibility criterion', 5, 'release'],
  ['percentile read from few samples', 1, 'release'], ['semantic mismatch', 3, 'escapes'],
  ['process-lifetime state', 2, 'escapes'],
];

// Feedback minutes read from automation-infrastructure/01's three-frequency arrangement.
// TP20: rework hours by stage -- a late-caught defect grows from lost context, retesting, and
// incident handling in production. TP21: an escaped defect is reported in production after one
// release cycle (2400 min); a change's baseline cycle time is 480 min.
export const FEEDBACK = { fast: 8, mid: 49, nightly: 425, release: 2221, escapes: 2400 };
export const REWORK = { fast: 0.25, mid: 1, nightly: 3, release: 6, escapes: 16 };
export const BASELINE = 480, DAY = 480, CHANGES = 100;
export const CAPACITY = 800;   // TP22: 5 people x 20 workdays x 8 hours

// Period 2's only difference is automation-infrastructure/01's "manual off pipeline"
// arrangement: the manual security review leaves the release gate, and accessibility criterion is no longer caught.
export const period = (second) => CLASSES.map(([name, n, a]) =>
  [name, n, second && name === 'accessibility criterion' ? 'escapes' : a]);

export function measure(k) {
  const defectCount = k.reduce((a, [, n]) => a + n, 0);
  const missed = k.filter(([, , a]) => a === 'escapes').reduce((a, [, n]) => a + n, 0);
  const rework = k.reduce((a, [, n, x]) => a + n * REWORK[x], 0);
  const feedback = k.reduce((a, [, n, x]) => a + n * FEEDBACK[x], 0) / defectCount;
  // Cycle time: clean and escaped changes converge at the baseline; a caught defect waits on
  // its stage's feedback and its rework.
  const d = [];
  for (let i = 0; i < CHANGES - defectCount; i += 1) d.push(BASELINE);
  for (const [, n, x] of k)
    for (let i = 0; i < n; i += 1)
      d.push(x === 'escapes' ? BASELINE : BASELINE + FEEDBACK[x] + REWORK[x] * 60);
  d.sort((a, b) => a - b);
  const percentile = (q) => d[Math.ceil((q / 100) * d.length) - 1] / DAY;
  return { defectCount, missed, leak: missed / defectCount, rework, reworkShare: rework / CAPACITY, feedback,
    median: percentile(50), p90: percentile(90), caught: defectCount - missed };
}

Cycle time’s calculation turns on one detail that decides the whole outcome: an escaped defect does not lengthen cycle time. No one sees it, so the change merges on time; its cost is paid in production, in a different column. This line is why the table further down shows cycle time moving in the wrong direction.

Two Periods, One Decision

There is exactly one difference between the two periods compared. The first period uses the three-frequency arrangement. In the second period, the team decides to cheapen the pipeline and removes the manual security review from the release gate — the Testing in Continuous Integration lesson measured that this arrangement drops from 98.2 to 86.2 minutes per change. The same lesson also measured that this decision raises escaped defects from five to ten; the question here is whether the metrics can see that.

// decision.mjs -- five metrics' value in two periods, and which one changed a decision
import { CLASSES, REWORK, CAPACITY, CHANGES, period, measure } from './period.mjs';

const PERIODS = [measure(period(false)), measure(period(true))];
const p = (x, n) => String(x).padStart(n);
const b = (x, n) => x.toFixed(n);

// Threshold and its source. A metric with no threshold has no decision tied to it either.
const METRICS = [
  { name: 'leakage rate', get: (x) => x.leak * 100, format: (v) => `${b(v, 2)}%`, threshold: 8,
    decision: 'add a test to inventory' },
  { name: 'rework share', get: (x) => x.reworkShare * 100, format: (v) => `${b(v, 2)}%`, threshold: 25,
    decision: 'move the class to an earlier stage' },
  { name: 'cycle time p90', get: (x) => x.p90, format: (v) => `${b(v, 2)} days`, threshold: 3.5,
    decision: 'change stage placement' },
  { name: 'cycle time median', get: (x) => x.median, format: (v) => `${b(v, 2)} days`, threshold: null,
    decision: 'none' },
  { name: 'avg feedback', get: (x) => x.feedback, format: (v) => `${b(v, 1)} min`, threshold: 480,
    decision: 'change gate frequency' },
  { name: 'caught defects', get: (x) => x.caught, format: (v) => String(v), threshold: null,
    decision: 'none' },
];

console.log(`${CHANGES} changes, ${CLASSES.length} defect classes, ${PERIODS[0].defectCount} defects, `
  + `${CHANGES - PERIODS[0].defectCount} clean changes; team capacity ${CAPACITY} hours`);
console.log(`\n${'metric'.padEnd(20)}${p('period 1', 11)}${p('period 2', 11)}${p('threshold', 11)}`
  + `${p('P1 over', 9)}${p('P2 over', 9)}${p('changed', 9)}${'  linked decision'}`);
let changing = 0;
for (const g of METRICS) {
  const [v1, v2] = [g.get(PERIODS[0]), g.get(PERIODS[1])];
  const [c1, c2] = [g.threshold !== null && v1 >= g.threshold, g.threshold !== null && v2 >= g.threshold];
  const changed = g.threshold !== null && c1 !== c2;
  if (changed) changing += 1;
  console.log(`${g.name.padEnd(20)}${p(g.format(v1), 11)}${p(g.format(v2), 11)}`
    + `${p(g.threshold === null ? '-' : g.format(g.threshold), 11)}${p(g.threshold === null ? '-' : (c1 ? 'yes' : 'no'), 9)}`
    + `${p(g.threshold === null ? '-' : (c2 ? 'yes' : 'no'), 9)}${p(changed ? 'YES' : 'no', 9)}`
    + `  ${g.decision}`);
}
// TP23: a metric's per-period collection and review cost is 30 minutes.
console.log(`\n${METRICS.length} metrics x 30 min = ${METRICS.length * 30} min/period review; `
  + `changing a decision ${changing}, noise ${(METRICS.length - changing) * 30} min`);

// The decision also asks "which class": rework hours broken down by class.
console.log(`\ntop three classes eating rework hours `
  + `(total ${b(PERIODS[0].rework, 0)} -> ${b(PERIODS[1].rework, 0)} hours)`);
console.log(`${' '.repeat(3)}${p('period 1', 32)}${p('period 2', 32)}`);
const rank = (second) => period(second).map(([name, n, a]) => [name, n * REWORK[a]])
  .sort((x, z) => z[1] - x[1]).slice(0, 3);
const [r1, r2] = [rank(false), rank(true)];
for (let i = 0; i < 3; i += 1)
  console.log(`${`${i + 1}.`.padEnd(3)}${p(`${r1[i][0]} ${b(r1[i][1], 0)} hrs`, 32)}`
    + `${p(`${r2[i][0]} ${b(r2[i][1], 0)} hrs`, 32)}`);

100 changes, 19 defect classes, 80 defects, 20 clean changes; team capacity 800 hours

metric                 period 1   period 2  threshold  P1 over  P2 over  changed  linked decision
leakage rate              6.25%     12.50%      8.00%       no      yes      YES  add a test to inventory
rework share             22.75%     29.00%     25.00%       no      yes      YES  move the class to an earlier stage
cycle time p90        2.26 days  1.23 days  3.50 days       no       no       no  change stage placement
cycle time median     1.05 days  1.05 days          -        -        -       no  none
avg feedback          386.8 min  398.0 min  480.0 min       no       no       no  change gate frequency
caught defects               75         70          -        -        -       no  none

6 metrics x 30 min = 180 min/period review; changing a decision 2, noise 120 min

top three classes eating rework hours (total 182 -> 232 hours)
                           period 1                        period 2
1.         semantic mismatch 48 hrs  accessibility criterion 80 hrs
2.    process-lifetime state 32 hrs        semantic mismatch 48 hrs
3.   accessibility criterion 30 hrs   process-lifetime state 32 hrs

Thresholds and Their Sources

A metric’s value alone does not carry a decision; what turns it into one is a threshold, and a threshold has to have a source.

• Leakage rate, 8 percent. Its source is the inventory’s blind spot: the Testing in Continuous Integration lesson counted two defect classes that no test team sees, five of eighty defects. That is leakage rate’s floor — 6.25 percent. The threshold cannot be set below that floor, because an unreachable threshold produces no decision in any period, it only leaves a permanent alarm.
• Rework share, 25 percent. Its source is team capacity: a quarter of 800 hours is 200 hours, and going above that means a quarter of a period going to fixes instead of new work.
• Cycle time p90, 3.5 workdays. Its source is the release gate itself: that gate’s average wait is 1200 minutes, that is, 2.5 workdays. If the threshold were set below that, every change caught at the release gate would cross it, and the metric would report the gate arrangement’s own outcome as if it were a defect.
• Average feedback, 480 minutes. One workday: the criterion that a defect reaches its owner within the same day.
• Cycle time median and caught defect count have no threshold, because they have no decision tied to them.

The Metric That Changed a Decision

Two of the six metrics changed a decision. Leakage rate crossed its threshold, rising from 6.25 percent to 12.50 percent; rework share crossed its threshold too, rising from 22.75 percent to 29.00 percent. The remaining four either never crossed their threshold in either period or had none.

The two metrics moved in the same direction but do not show the same thing. Leakage rate counts defects, rework share weighs them in hours, and their rankings differ. In the first period, the class eating the most rework hours is semantic mismatch: 48 hours, but only three defects. Three defects out of eighty never rise to the top of any count-based list; weighed in hours, it ranks first. In the second period, accessibility criterion jumps from 30 hours to 80 and takes first place — all five of its defects are now found in production, and fixing them takes sixteen hours instead of six. The decision, for this reason, comes out not as “do something” but as “do this.”

The remaining four metrics are this lesson’s real finding. Cycle time p90 dropped from 2.26 days to 1.23 — that is, this metric improved while quality got worse. The reason is written in a single line of code: an escaped defect does not lengthen cycle time. The five changes that used to wait at the release gate no longer wait there, because there is no longer a test there to hold them. The median, meanwhile, did not move at all, staying at 1.05 days. Deleting a test always improves cycle time.

Average feedback time only worsened by about three percent, from 386.8 minutes to 398.0 — while leakage doubled. The reason is how close the two numbers are: the release gate used to report a defect in 2221 minutes, an escaped defect in production is reported in 2400. The average cannot tell catching apart from missing. Caught defect count also dropped from 75 to 70; read on its own, this row looks like “fewer defects,” but what it says is fewer catches.

The attention arithmetic gives the price of this: six metrics ask for 180 minutes of collection and review per period, and 120 minutes of that changes no decision. A metric that changes no decision is noise — and its cost is not just that 120 minutes. In the same period, cycle time and caught defect count look like they improved together; a team looking at just those two could read the decision that doubled leakage as a success.

The Return on the Decision

Leakage rate’s decision changes the inventory: a new test is written for the blind-spot class, and that test’s run cost is added to the next period’s pipeline time. This worsens cycle time. Metrics are not independent of each other; while one is improved, another pays for it, and this is why no single metric is targeted alone.

Rework share’s decision moves a defect class to an earlier stage, and its owner is whoever built the gate arrangement. Cycle time’s decision changes stage placement. A metric with no decision tied to it has no owner; it gets read out as a number in the period meeting, no one acts on it, and its cost is paid again every period.

Summary

• Leakage rate, rework share, and cycle time were computed from the same defect set; rework here is a separate concept from the failback in the Resilience and Reliability course — the return to the primary copy after a takeover.
• A single pipeline-cheapening decision raised leakage rate from 6.25 percent to 12.50 percent and rework share from 22.75 percent to 29.00 percent; both crossed their threshold and produced a decision.
• The same decision dropped cycle time p90 from 2.26 days to 1.23, because an escaped defect does not lengthen cycle time; the median did not change at all, and caught defects dropped from 75 to 70.
• Average feedback time only worsened by three percent: because the time to notice in production (2400 min) is close to the release gate’s wait (2221 min), the average conceals the miss.
• Two of the six metrics changed a decision; the remaining four ate 120 minutes of attention per period, and two pointed the wrong way. A metric that changes no decision is noise.

Next Step

Metrics say one thing and do not say another. What they say: schema mismatch came up eight times this period, boundary comparison thirteen times, and most rework hours pile up in a few classes. What they do not say: why. All eight schema mismatches were fixed one by one, each one paying its own rework hours again; if there is a single shared cause under those eight, no metric shows it, because metrics count, they do not explain. A class recurring is not a defect, it is a symptom, and going from symptom to cause needs a method. The next lesson builds that method.