Reporting

The previous lessons put the pipeline, the environment, and secrets in order. The pipeline now produces a result on every change and every night. This lesson’s question is not the result itself, but what it tells whom: a run result counts as not having run at all until it turns into a decision, because no one acts on it otherwise.

This is not a writing task, it is a distribution task. The resource distributed is attention: how many lines get read before a result turns into a decision, how many steps it takes to find the cause, how many items produce no action at all. The same run result is presented in four arrangements, and these three numbers are measured.

A Run’s Raw Result

The result to be measured comes from a nightly run: the day’s five changes are taken together, and seven teams run, all but the two teams at the release gate. The red items inside the result come from four sources, and the report’s real job is telling them apart — root (the first test a defect drops), derived (the other tests the same root drops), environment (a failure that belongs to the mechanism, not the code under test), and flaky (a test that drops without the code changing).

// run.mjs -- a nightly run's raw result: test states, red items, roots
let s = 20260731;                                  // seed
const random = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);

// TP16: seven teams' test counts; flaky tests are spread at step intervals and drop at rate,
// history is the probability of dropping from a REAL defect. The release gate is not in this run.
export const teams = [
  { name: 'unit', tests: 240, step: 0, rate: 0.00, history: 0.010 },
  { name: 'contract', tests: 42, step: 21, rate: 0.03, history: 0.008 },
  { name: 'integration', tests: 96, step: 12, rate: 0.06, history: 0.015 },
  { name: 'end-to-end', tests: 34, step: 3, rate: 0.15, history: 0.012 },
  { name: 'perf-run', tests: 12, step: 6, rate: 0.10, history: 0.010 },
  { name: 'security-auto', tests: 58, step: 19, rate: 0.04, history: 0.006 },
  { name: 'resilience', tests: 14, step: 7, rate: 0.05, history: 0.010 },
];

// M21/K03: the decision looks at the MEASURED rate over 200 runs; the measured rate is the sum of two sources.
export const tests = [];
for (const t of teams) {
  for (let i = 0; i < t.tests; i += 1) {
    const flaky = t.step > 0 && i % t.step === 0 ? t.rate : 0;
    let dF = 0, dR = 0;
    for (let k = 0; k < 200; k += 1) {
      if (random() < flaky) dF += 1;
      if (random() < t.history) dR += 1;
    }
    tests.push({ id: `${t.name}/${i + 1}`, team: t.name, flaky, dF, dR,
      measured: (dF + dR) / 200, red: false, root: null });
  }
}

// TP17: this run's red roots. Class name and catching team are from automation-infrastructure/01's
// defect set; affected test count comes from the shared surface the defect touches.
export const roots = [
  { name: 'schema mismatch', team: 'integration', affected: 23, real: true },
  { name: 'boundary comparison', team: 'unit', affected: 3, real: true },
  { name: 'status code mapping', team: 'end-to-end', affected: 5, real: true },
  { name: 'dependent service did not start', team: 'resilience', affected: 14, real: false },
];
for (const t of tests) if (t.flaky > 0 && random() < t.flaky) { t.red = true; t.root = 'flaky'; }
for (const k of roots) {
  for (const t of tests.filter((x) => x.team === k.team).slice(0, k.affected)) {
    t.red = true; t.root = k.name;
  }
}
export const reds = tests.filter((t) => t.red);
export const realRoots = roots.filter((k) => k.real);

This run has three real defects and one environment failure; the number of items that drop is far larger, because the single defect touching a shared schema drops all twenty-three tests that use that schema at once. The report does not know this; all it sees are red items.

Four Reporting Arrangements

The harness below models reading as a cost (TP18). A line is three seconds, a step is three minutes; a step is any work that does not end with reading the line — opening a detail, rerunning, finding the owner. In an ungrouped report, the reader only merges the same signature by the third item.

TP19 — the attention one person gives to a run result is 30 minutes, that is, one sixteenth of a workday; its sensitivity is scanned in the second half of the table. A root that is not resolved carries over to the next nightly run and adds 240 minutes per TP8; a root that is never resolved escapes.

// report.mjs -- time-to-decision for the same run result across four reporting arrangements
import { teams, tests, reds, roots, realRoots } from './run.mjs';

// TP18: a line is 3 sec, a step is 3 min, the raw log has 12 detail lines per red item, the
// same root merges by the third item. TP19: attention given to one run result is 30 min.
const SEC = 3, STEP = 3, DETAIL = 12, RECOGNIZE = 3, BUDGET = 30;
const WAIT = 240;                        // automation-infrastructure/01 TP8: nightly wait

// An arrangement's reading flow: a sequence of {lines, step, root resolved} events.
function flow(arrangement, remaining) {
  const live = (t) => t.red && (t.root === 'flaky' || remaining.includes(t.root));
  const events = [], seen = {};
  const raw = (detailed) => {
    for (const tm of teams) {
      if (detailed) events.push({ lines: 2, step: 0 });
      for (const t of tests.filter((x) => x.team === tm.name)) {
        if (live(t) === false) { if (detailed) events.push({ lines: 1, step: 0 }); continue; }
        const n = (seen[t.root] = (seen[t.root] ?? 0) + 1);
        const group = tests.filter((x) => live(x) && x.root === t.root).length;
        events.push({ lines: detailed ? 1 + DETAIL : 1, step: n <= RECOGNIZE ? 1 : 0,
          done: n === Math.min(RECOGNIZE, group) ? t.root : null });
      }
    }
  };
  if (arrangement === 'raw log') { events.push({ lines: 1, step: 1 }); raw(true); }
  if (arrangement === 'flat list') { events.push({ lines: 1, step: 0 }); raw(false); events.push({ lines: 1, step: 0 }); }
  if (arrangement === 'grouped') {
    for (const k of roots.filter((x) => x.real && remaining.includes(x.name)))
      events.push({ lines: 3, step: 1, done: k.name });          // header + root test + "n more items"
    events.push({ lines: 1, step: 1 }, { lines: 1, step: 1 }, { lines: 2, step: 0 }); // environment, flaky, summary
  }
  if (arrangement === 'summary only') { events.push({ lines: 3, step: 2 }); raw(true); }
  return events;
}

const read = (arrangement, remaining, budget) => {
  let minutes = 0, lines = 0, step = 0; const done = [];
  for (const e of flow(arrangement, remaining)) {
    const cost = minutes + (e.lines * SEC) / 60 + e.step * STEP;
    if (cost > budget) break;
    minutes = cost; lines += e.lines; step += e.step;
    if (e.done && remaining.includes(e.done)) done.push(e.done);
  }
  return { minutes, lines, step, done };
};

// An unresolved root carries over to the next run (+240 min); a root never resolved escapes.
function measure(arrangement, budget = BUDGET) {
  let remaining = realRoots.map((k) => k.name), cycle = 0, first = null; const feedback = [];
  while (remaining.length > 0 && cycle < 20) {
    cycle += 1;
    const r = read(arrangement, remaining, budget);
    first ??= r;
    if (r.done.length === 0) break;
    for (const _ of r.done) feedback.push(WAIT * cycle + r.minutes);
    remaining = remaining.filter((a) => r.done.includes(a) === false);
  }
  const avg = feedback.length ? feedback.reduce((a, c) => a + c, 0) / feedback.length : NaN;
  return { ...first, firstCycle: first.done.length, cycle, missed: remaining.length, feedback: avg };
}

const ARRANGEMENTS = ['raw log', 'flat list', 'grouped', 'summary only'];
const noise = { 'raw log': tests.length - 3, 'flat list': reds.length - 3,
  grouped: 2, 'summary only': tests.length - 3 };
const p = (x, n) => String(x).padStart(n);
const total = flow('raw log', realRoots.map((k) => k.name)).reduce((a, e) => a + e.lines, 0);
console.log(`${tests.length} tests, ${reds.length} red items, ${roots.length} roots `
  + `(${realRoots.length} real defects + 1 environment), ${reds.filter((t) => t.root === 'flaky').length} independent flaky drops`);
console.log(`raw log ${total} lines = ${((total * SEC) / 60).toFixed(1)} min, `
  + `${((total * SEC) / 60 / BUDGET).toFixed(1)}x the budget\n`);
console.log(`${'arrangement'.padEnd(14)}${p('lines', 7)}${p('steps', 7)}${p('minutes', 9)}${p('noise', 9)}`
  + `${p('1st cycle', 11)}${p('cycles', 8)}${p('missed root', 13)}${p('avg feedback', 14)}`);
for (const d of ARRANGEMENTS) {
  const r = measure(d);
  console.log(`${d.padEnd(14)}${p(r.lines, 7)}${p(r.step, 7)}${p(r.minutes.toFixed(1), 9)}`
    + `${p(noise[d], 9)}${p(`${r.firstCycle}/3`, 11)}${p(r.cycle, 8)}${p(r.missed, 13)}`
    + `${p(Number.isNaN(r.feedback) ? '-' : r.feedback.toFixed(1), 14)}`);
}
console.log(`\nTP19 sensitivity (roots resolved in cycle 1 / roots missed)`);
console.log(`${'budget'.padEnd(8)}${ARRANGEMENTS.map((d) => p(d, 14)).join('')}`);
for (const b of [15, 30, 60])
  console.log(`${`${b} min`.padEnd(8)}${ARRANGEMENTS.map((d) => { const r = measure(d, b); return p(`${r.firstCycle} / ${r.missed}`, 14); }).join('')}`);

496 tests, 46 red items, 4 roots (3 real defects + 1 environment), 1 independent flaky drops
raw log 895 lines = 44.8 min, 1.5x the budget

arrangement     lines  steps  minutes    noise  1st cycle  cycles  missed root  avg feedback
raw log           325      4     28.3      493        1/3       3            1         389.0
flat list          32      9     28.6       43        3/3       1            0         268.6
grouped            13      5     15.7        2        3/3       1            0         255.7
summary only      299      5     30.0      493        1/3       2            2         270.0

TP19 sensitivity (roots resolved in cycle 1 / roots missed)
budget         raw log     flat list       grouped  summary only
15 min           1 / 2         1 / 0         3 / 0         0 / 3
30 min           1 / 1         3 / 0         3 / 0         1 / 2
60 min           2 / 0         3 / 0         3 / 0         2 / 0

The raw log is 895 lines; reading all of it is one and a half times the budget. The reader exhausts the budget at 325 lines and resolves only one root that night. Even read three nights running, the third root still does not show up: the end-to-end team sits at the end of the log, behind the detail lines of forty-six red items. A defect being present in the report is not the same as it having a place in the report.

The flat red list and the grouped report give the same decision: all three roots close on the first night, nothing escapes. The difference is in cost. The flat list has 32 lines to read, the grouped report 13; but the flat list opens 9 steps, the grouped report 5. This is where the numbers pull apart: 32 lines is 1.6 minutes, 9 steps is 27 minutes. A report’s cost is not the reading cost, it is the opening cost — and opening cost only drops with grouping, not with shortening. The flat list consumes ninety-five percent of the budget; the grouped report, half.

This share’s weight shows up in the sensitivity table. At a fifteen-minute budget, the grouped report still surfaces all three roots, the flat list only one. There is no symmetry in the other direction either: raising the budget to sixty minutes gives the raw log two roots in the first cycle, while the grouped report does three in fifteen. Raising attention does not do what grouping does, because most of the attention spent goes not to the decision itself but to recognizing the same root again and again.

The noise column gives the source of the difference: 493 items in the raw log resolve into no decision, 43 in the flat list, 2 in the grouped report — the environment and flaky sections. In the summary-only arrangement, the report is three lines and says nothing; the reader goes back to the raw log, and two roots escape. A short report is not a readable report.

The Flaky Section’s Threshold

Part of the grouped report’s gain comes from putting flaky items into a separate section. This is a threshold decision: a test whose measured drop rate crosses the threshold counts as flaky and is pulled out of the main section. The threshold’s source is the Flakiness Management lesson; there the same number was scanned as a quarantine threshold and dropped the test from the pipeline. The decision here is lighter — the test keeps running — but its cost is measured the same way.

// threshold.mjs -- the flaky section's culling threshold: noise cut against real red hidden
import { tests, reds, realRoots } from './run.mjs';

const p = (x, n) => String(x).padStart(n);
const total = (f) => tests.reduce((a, t) => a + f(t), 0);
console.log(`${'threshold'.padStart(9)}${p('separated', 11)}${p('to section', 12)}`
  + `${p('hidden items', 14)}${p('hidden roots', 14)}`
  + `${p('noise cut', 11)}${p('real hidden', 13)}`);
for (const e of [0.005, 0.01, 0.015, 0.025, 0.05, 0.1]) {
  const separated = new Set(tests.filter((t) => t.measured >= e).map((t) => t.id));
  const dropped = reds.filter((t) => separated.has(t.id));
  const items = dropped.filter((t) => realRoots.some((k) => k.name === t.root)).length;
  const hiddenRoots = realRoots.filter((k) =>
    reds.filter((t) => t.root === k.name).every((t) => separated.has(t.id))).length;
  console.log(`${`${(e * 100).toFixed(1)}%`.padStart(9)}${p(separated.size, 11)}${p(dropped.length, 12)}`
    + `${p(items, 14)}${p(hiddenRoots, 14)}`
    + `${p(total((t) => (separated.has(t.id) ? t.dF : 0)), 11)}`
    + `${p(total((t) => (separated.has(t.id) ? t.dR : 0)), 13)}`);
}
console.log(`\ncolumns: the first four are over THIS run's ${reds.length} red items, `
  + `the last two are over the 200-run history`);
console.log(`of the ${total((t) => t.dF + t.dR)} historical drops, `
  + `${total((t) => t.dF)} were from flakiness, ${total((t) => t.dR)} from a real defect; `
  + `the report cannot tell them apart, it only sees the combined rate`);

threshold  separated  to section  hidden items  hidden roots  noise cut  real hidden
     0.5%        431          45            31             3        541         1013
     1.0%        296          35            23             0        541          878
     1.5%        187          26            18             0        541          660
     2.5%         61          14             9             0        541          241
     5.0%         25           7             4             0        508           59
    10.0%         13           3             2             0        364           26

columns: the first four are over THIS run's 46 red items, the last two are over the 200-run history
of the 1554 historical drops, 541 were from flakiness, 1013 from a real defect; the report cannot tell them apart, it only sees the combined rate

The first row shows how far the threshold can go. At 0.5 percent, 431 of 496 tests count as flaky, 45 of 46 red items drop to the second section, and all three real roots are hidden at once. The report looks spotless and says nothing: a report that zeroes out the noise may have zeroed out the signal too.

The reason is written in the last two columns. Of the 1554 historical drops, 541 come from flakiness, 1013 from a real defect; the report cannot tell the two apart, it only sees the combined rate. This is why a low threshold hides the tests that have surfaced the most defects.

The 2.5 percent threshold is the lowest point where it can still tell the two apart: 61 tests separate out, 14 items drop to the second section, nine of them from a real defect, but no root is hidden entirely — every root keeps at least one item in the main section. In exchange, all 541 historical flaky drops are cut. Raising the threshold beyond this only loses: the noise cut falls to 508 at 5 percent, to 364 at 10 percent — the noise flows back into the main section.

The Return on the Decision

Every section of a report corresponds to an action; a section with no corresponding action is cut. The root section opens a record, and its owner is the owner of the change that touched that defect; since a nightly run has no single owner, this section has to show the root, counting items is not enough. The environment section is the pipeline operator’s business and says nothing about the code under test. The flaky section stops no one; it is a maintenance list and feeds the quarantine decision.

Not showing derived items is also a decision, not a concealment: twenty-three tests dropped from a single defect, so a single record opens. Shown separately, the record count would not change, but the attention spent would multiply by twenty-three.

Summary

• The same run result was presented in four arrangements: 496 tests, 46 red items, 3 real roots, 1 environment failure, 1 independent flaky drop.
• The grouped report resolved all three roots in 13 lines, 5 steps, and 15.7 minutes; the flat list gave the same decision in 32 lines, 9 steps, and 28.6 minutes — the difference comes from the steps opened (9 steps is 27 minutes, 32 lines is 1.6 minutes).
• In the raw log, one root was never resolved on any night and escaped; in the summary-only arrangement, two roots escaped. Average feedback time is 255.7 minutes in the grouped report, 389.0 in the raw log.
• Raising the attention budget did not stand in for grouping: at 60 minutes, the raw log surfaced two roots; the grouped report surfaced three in 15.
• When the flaky section’s threshold was lowered to 0.5 percent, all three roots were hidden; at 2.5 percent, all flaky drops were cut while no root was hidden.

Next Step

Every measure here belongs to a single run and gives a single night’s decision. No matter how well a report shows the root in its own run, it cannot answer this question: is this team testing better over time, or are the same defects arriving under different names. Schema mismatch showing up tonight is an event; showing up every week is a process defect, and a single run’s report makes the two look the same. The next lesson moves the measure to the period level: escaped defects, rework, and cycle time are computed from the same data, and each is asked what decision it changes.