--- title: 'The Testing Process and Automation Infrastructure' source: 'https://academia.sh/en/courses/testing-process' language: en updated: '2026-08-23T16:12:24+00:00' license: 'CC BY-SA 4.0' lessons: 11 outcomes: - 'Prepares a risk-based test plan' - 'Places test automation in the continuous integration pipeline' - 'Improves the process by interpreting quality metrics' --- # The Testing Process and Automation Infrastructure Dividing limited time among tests and turning a red result into a decision: writing scope, criteria and responsibilities into the plan, letting risk drive prioritization, structured exploratory testing finding what a scripted test cannot see, writing the definition of done, managing user acceptance testing, placing staged test gates in the continuous integration pipeline, the cost of environment variety, secret and data safety in the test environment, making results readable for the team, interpreting quality metrics, and tracing recurring defects to their source. ## Planning Deciding where limited time goes: writing down scope, entry and exit criteria, and responsibility, prioritizing by the order that risk's product of likelihood and impact produces, structured free-form testing finding defects the scripted test cannot see, turning the definition of done into acceptance criteria, and user acceptance testing's scope and feedback management. ### Test Plan Source: https://academia.sh/en/courses/testing-process/test-plan Dividing a limited release budget among the test suite written across four courses: listing the inventory together with its cost quantities, tying entry and exit criteria to a number, letting responsibility determine how many tests actually run, and comparing four plans on the same defect set by escaped defect and feedback time. The Non-Functional Testing course closed four courses' accumulation with a single question: which test runs when. Some of these tests are cheap enough to run on every change, some are expensive enough to run once per release; how limited time is divided among them was addressed in no lesson. Every lesson in this course follows the same shape: a limited resource is distributed, the **escaped defect** it produces and the **feedback time** it costs are measured, and what a red result turns into is written down. This escaped defect is the same concept measured in the Integration, Contract and End-to-End Testing course; this course keeps the short name. #### The Plan's Three Columns A **test plan** writes down three things: **scope** says which defect classes are tested, **criteria** say when the plan starts and when it closes, **responsibility** says who owns each item. Until all three are tied to a number, a plan is a statement of intent. Criteria have two faces. **Entry criteria** are the condition a stage must meet before it can start running; a test that runs without it satisfied produces no information, only spends time. **Exit criteria** say the plan has closed, and this lesson ties it to the share of defect classes covered. #### Inventory: Four Courses' Tests in One List Across four courses, twelve test forms were written for the library loan system. Each one's cost is recorded in its source lesson's closing table as a quantity **independent of the run**: how many processes, how many requests or queries, how many manual review steps. Two assumptions convert these into minutes. **TP1 (assumption) — run minutes: 0.5 min per process, 0.0002 min per call.** The process share comes from the six steps of the readiness poll in the Integration, Contract and End-to-End Testing course; the call share is 12 milliseconds per request. **TP2 (assumption) — 6 minutes of human time per manual review step.** ```js // library/inventory.mjs — the test suite written across four courses and the course's shared // defect set. Cost quantities are taken from the M21/K03 and M21/K04 closing tables; the // conversion to minutes uses the TP1 (run) and TP2 (human) assumptions. const RAW = [ // code, name, level, processes, calls, manual steps, defect class caught ["T01", "unit test suite", "unit", 0, 266, 0, "rule-boundary"], ["T02", "integration test", "integration", 1, 772, 0, "schema-mismatch"], ["T03", "database migration test", "integration", 1, 21, 0, "migration-data-loss"], ["T04", "contract test", "contract", 2, 100, 0, "contract-field"], ["T05", "end-to-end browser test", "end-to-end", 2, 16477, 0, "interface-state"], ["T06", "visual verification", "end-to-end", 1, 784, 1, "visual-deviation"], ["T07", "load test", "performance", 9, 55000, 0, "data-growth"], ["T08", "static security scan", "security", 0, 75, 7, "sql-concatenation"], ["T09", "dependency scan", "security", 0, 10, 5, "known-vulnerability"], ["T10", "accessibility audit", "security", 0, 14, 6, "accessibility"], ["T11", "fault injection", "resilience", 1, 13, 5, "degraded-response"], ["T12", "recovery verification", "resilience", 2, 7, 10, "recovery-gap"], ]; export const TESTS = RAW.map(([code, name, level, processes, calls, manual, cls]) => ({ code, name, level, processes, calls, manual, cls, run: 0.5 * processes + 0.0002 * calls, // TP1 human: 6 * manual, // TP2 min: 0.5 * processes + 0.0002 * calls + 6 * manual, })); // The two classes no member of the automated set catches. export const UNCOVERED = ["off-scenario", "semantic-drift"]; const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` The second structure is the **defect set**, and it is the shared input of every lesson in this course: fourteen classes, with likelihood, impact, and fix minutes drawn from a seed. Class names are taken from the defects each source lesson caught. Two of them — `off-scenario` and `semantic-drift` — are seen by no test in the inventory; the second is the semantic mismatch the API Testing lesson of the Integration, Contract and End-to-End Testing course recorded in its "missed" column. #### Four Plans, One Defect Set A plan is an ordered test list. A defect is caught in the minute the first test that sees its class finishes; if no test sees it, it escapes. **Defect leakage rate** is the escaped share of the set. **TP3 (assumption) — release test budget: 120 min.** **TP4 (assumption) — exit criterion: covered defect class share at least 70%.** **TP5 (assumption) — an escaped defect's fix cost is 6 times a caught one's.** **TP6 (assumption) — a horizon of 20 release candidates; the defect set appears across this horizon.** **TP7 (assumption) — a quarter of the candidates are red on the smoke test.** ```js // library/plan.mjs — builds the test plan as a data structure and compares four plans over // the same defect set. Input: library/inventory.mjs import { TESTS, UNCOVERED, defects } from "./inventory.mjs"; const BUDGET = 120, CANDIDATES = 20, MULTIPLIER = 6, SEED = 20260731, N = 60; // TP3, TP6, TP5 const OWNER = { T01: "development", T02: "development", T03: "development", T04: "development", T05: "testing", T06: "testing", T07: "testing", T08: "security", T09: "security", T10: "product", T11: "operations", T12: "operations" }; const DEFECTS = defects(SEED, N); const CLASSES = [...new Set([...TESTS.map((t) => t.cls), ...UNCOVERED])]; const v = (x, n = 2) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); // A plan is an ordered test list. Escaped defect = a defect no test in the plan sees. // Feedback time = the average of the cumulative minute at which a caught defect is caught. function measure(order) { const moment = new Map(); let t = 0; for (const test of order) { t += test.min; moment.set(test.cls, t); } const caught = DEFECTS.filter((k) => moment.has(k.cls)); const escaped = DEFECTS.filter((k) => moment.has(k.cls) === false); return { tests: order.length, cls: moment.size, time: t, escaped: escaped.length, rate: (100 * escaped.length) / DEFECTS.length, feedback: caught.reduce((a, k) => a + moment.get(k.cls), 0) / (caught.length || 1), fix: escaped.reduce((a, k) => a + k.fix, 0) }; } const cheapest = [...TESTS].sort((a, b) => a.min - b.min); const fitsBudget = []; for (const t of cheapest) if (fitsBudget.reduce((a, x) => a + x.min, 0) + t.min <= BUDGET) fitsBudget.push(t); const criterionSet = cheapest.slice(0, Math.ceil(0.7 * CLASSES.length)); // exit criterion: 70% of classes const owned = TESTS.filter((t) => OWNER[t.code] !== "product" && OWNER[t.code] !== "security"); const PLANS = [["P1 full coverage", TESTS], ["P2 fits budget (no criterion)", fitsBudget], ["P3 exit criterion >= 70%", criterionSet], ["P4 full coverage, 3 items unowned", owned]]; console.log("test inventory — cost quantities from the M21/K03 and M21/K04 closing tables"); console.log("code test level run human total class owner"); for (const t of TESTS) console.log(t.code + " " + t.name.padEnd(26) + t.level.padEnd(12) + g(v(t.run), 6) + g(v(t.human, 1), 8) + g(v(t.min), 8) + " " + t.cls.padEnd(21) + OWNER[t.code]); console.log("total".padEnd(44) + g(v(TESTS.reduce((a, t) => a + t.run, 0)), 6) + g(v(TESTS.reduce((a, t) => a + t.human, 0), 1), 8) + g(v(TESTS.reduce((a, t) => a + t.min, 0)), 8) + " " + TESTS.length + "/" + CLASSES.length + " classes"); console.log(`\ndefect set: seed ${SEED}, ${N} defects, ${CLASSES.length} classes`); console.log("class no test sees: " + UNCOVERED.map((s) => `${s} (${DEFECTS.filter((k) => k.cls === s).length})`).join(", ")); console.log(`\nplan comparison — budget TP3 = ${BUDGET} min/release, horizon TP6 = ${CANDIDATES} releases`); console.log("plan test class time budget escaped leak% feedback"); for (const [name, order] of PLANS) { const o = measure(order); console.log(name.padEnd(35) + g(o.tests, 4) + g(o.cls, 7) + g(v(o.time, 1), 7) + g(o.time <= BUDGET ? "fits" : "over", 8) + g(o.escaped, 9) + g(v(o.rate, 1), 9) + g(v(o.feedback, 1), 10)); } console.log("\nexit criterion scan: how many minutes a covered class share costs"); console.log("criterion classes needed cheapest time budget escaped"); for (const p of [50, 60, 70, 80, 90]) { const needed = Math.ceil((p / 100) * CLASSES.length); if (needed > TESTS.length) { console.log(g(p + "%", 9) + g(needed, 16) + g("unreachable", 15) + g("-", 8) + g("-", 9)); continue; } const o = measure(cheapest.slice(0, needed)); console.log(g(p + "%", 9) + g(needed, 16) + g(v(o.time, 1), 15) + g(o.time <= BUDGET ? "fits" : "over", 8) + g(o.escaped, 9)); } const smoke = TESTS[0].min + TESTS[1].min, ps = measure(fitsBudget).time; let y = 4242; // TP7: candidate seed const red = Array.from({ length: CANDIDATES }, () => (y = (1103515245 * y + 12345) % 2147483648) / 2147483648 < 0.25).filter(Boolean).length; console.log(`\nentry criterion: smoke test (T01+T02, ${v(smoke)} min) must be green or P2 does not run`); console.log(`${CANDIDATES} release candidates, seed 4242: ${red} candidates are red on the smoke test`); console.log(`without criterion P2 runs ${CANDIDATES} times, ${v(CANDIDATES * ps, 1)} min; ${red} runs produce no information -> ${v(red * ps, 1)} min wasted`); console.log(`with criterion P2 runs ${CANDIDATES - red} times, ${v((CANDIDATES - red) * ps, 1)} min + smoke ${v(CANDIDATES * smoke, 1)} min = ${v((CANDIDATES - red) * ps + CANDIDATES * smoke, 1)} min`); console.log(`\nescaped-defect multiplier TP5's sensitivity: total cost over ${CANDIDATES} releases (min)`); console.log("mult P1 full coverage P2 fits budget P3 criterion cheapest plan"); for (const mult of [2, 6, 20]) { const b = PLANS.slice(0, 3).map(([, s]) => { const o = measure(s); return CANDIDATES * o.time + mult * o.fix; }); const names = ["P1", "P2", "P3"]; console.log(g(mult, 4) + g(v(b[0], 0), 19) + g(v(b[1], 0), 17) + g(v(b[2], 0), 14) + g(names[b.indexOf(Math.min(...b))], 15)); } ``` ``` test inventory — cost quantities from the M21/K03 and M21/K04 closing tables code test level run human total class owner T01 unit test suite unit 0.05 0.0 0.05 rule-boundary development T02 integration test integration 0.65 0.0 0.65 schema-mismatch development T03 database migration test integration 0.50 0.0 0.50 migration-data-loss development T04 contract test contract 1.02 0.0 1.02 contract-field development T05 end-to-end browser test end-to-end 4.30 0.0 4.30 interface-state testing T06 visual verification end-to-end 0.66 6.0 6.66 visual-deviation testing T07 load test performance 15.50 0.0 15.50 data-growth testing T08 static security scan security 0.02 42.0 42.02 sql-concatenation security T09 dependency scan security 0.00 30.0 30.00 known-vulnerability security T10 accessibility audit security 0.00 36.0 36.00 accessibility product T11 fault injection resilience 0.50 30.0 30.50 degraded-response operations T12 recovery verification resilience 1.00 60.0 61.00 recovery-gap operations total 24.21 204.0 228.21 12/14 classes defect set: seed 20260731, 60 defects, 14 classes class no test sees: off-scenario (4), semantic-drift (6) plan comparison — budget TP3 = 120 min/release, horizon TP6 = 20 releases plan test class time budget escaped leak% feedback P1 full coverage 12 12 228.2 over 10 16.7 40.8 P2 fits budget (no criterion) 9 9 89.2 fits 18 30.0 16.5 P3 exit criterion >= 70% 10 10 125.2 over 14 23.3 26.0 P4 full coverage, 3 items unowned 9 9 120.2 over 20 33.3 13.7 exit criterion scan: how many minutes a covered class share costs criterion classes needed cheapest time budget escaped 50% 7 28.7 fits 24 60% 9 89.2 fits 18 70% 10 125.2 over 14 80% 12 228.2 over 10 90% 13 unreachable - - entry criterion: smoke test (T01+T02, 0.71 min) must be green or P2 does not run 20 release candidates, seed 4242: 3 candidates are red on the smoke test without criterion P2 runs 20 times, 1783.8 min; 3 runs produce no information -> 267.6 min wasted with criterion P2 runs 17 times, 1516.2 min + smoke 14.2 min = 1530.4 min escaped-defect multiplier TP5's sensitivity: total cost over 20 releases (min) mult P1 full coverage P2 fits budget P3 criterion cheapest plan 2 5974 4424 4544 P2 6 8794 9704 8624 P3 20 18664 28184 22904 P1 ``` Every number is deterministic: the same seed gives the same defect set. The inventory's first reading already shows an imbalance — 204 of the plan's 228.21 minutes are human time; the three most expensive items (T12, T08, T10) are almost entirely manual review steps. #### Two Distributions, Two Outcomes P1 covers every defect class and asks for 228.2 minutes per release — 108.2 minutes over budget. P2 takes the cheapest tests in order to fit the same budget: nine tests, 89.2 minutes. The two distributions' outcomes on the same defect set differ: P1 lets 10 defects escape (16.7%), P2 lets 18 escape (30.0%). Against that, P2's feedback time is 16.5 minutes, P1's is 40.8 — because P1's expensive tail reports what it catches late. **A distribution that lowers the escaped defect count lengthens the feedback time.** This is the trade-off that repeats in every lesson of this course. The escaped defect's cost depends on TP5, and this assumption is unmeasured; the sensitivity scan makes that visible. At multiplier 2 the cheapest plan is P2, at multiplier 6 it is P3, at multiplier 20 it is P1. So the plan's justification depends on the value of a single assumption — yet only one of the three plans, P2, fits the budget. The budget is not a cost item; it is a limit. #### The Exit Criterion's Ceiling The exit criterion scan prices the criterion itself. A 50% criterion costs 28.7 minutes, a 60% one costs 89.2, and both fit the budget. TP4's chosen 70%, though, asks for 125.2 minutes — 5.2 minutes over budget. The criterion and the budget conflict, and the plan has to choose between them. The 90% row says something different: **unreachable.** There is a test that sees twelve of the fourteen classes, none that sees the remaining two; no budget lifts this ceiling, because the ceiling comes from coverage, not time. The plan must either write these two classes explicitly out of scope, or add a test form that can see them. The escaped defect the out-of-scope choice produces in this set is 10 defects. #### Entry Criterion and Responsibility The entry criterion does not govern the run itself, but when it is allowed to start. Running P2 across twenty candidates while the smoke test (T01 and T02, 0.71 minutes) is not green spends 267.6 minutes on runs that produce no information. Once the criterion is in place, the smoke test running twenty times costs 14.2 minutes and the total drops to 1,530.4 minutes — a 253.4-minute gain, a cheap poll protecting an expensive stage. The responsibility column is the plan's quietest column. P4 writes the same scope as P1, but three items are owned outside the team and those items do not run. On paper 12 classes are covered, in practice 9; escaped defects rise from 10 to 20 and the leakage rate becomes 33.3% — a worse outcome than the budget-fitting, criterion-free plan, and this while the plan claims full coverage. **An item with no owner written down is not in scope.** What a red result does is also written into the plan: when T01–T05 turn red the release candidate does not advance, T06 through T10 open a record without stopping the release, and T12's red goes to a decision by operations. A test that changes no decision should not draw from the budget. #### Summary - A test plan is three columns: scope, criteria, and responsibility; until all three are tied to a number, a plan carries no decision. - Four courses' twelve tests total 228.21 minutes, and 204 of those are human time; since the budget (TP3) is 120 minutes, full coverage is a wish, not a plan. - On the same defect set, full coverage lets 10 defects escape and reports in 40.8 minutes; the budget-fitting plan lets 18 escape and reports in 16.5 minutes. - The exit criterion comes with a price: the 70% criterion asks for 125.2 minutes, over budget, and 90% is unreachable by any budget because two defect classes are seen by no test. - The entry criterion puts a cheap poll ahead of an expensive stage: 267.6 minutes of wasted runs are prevented by a 14.2-minute smoke test. - Three unowned items cut the full-coverage plan to nine tests and raise the leakage rate from 16.7% to 33.3%. #### Next Step The budget-fitting plan chose tests **cheapest first**; that ordering's only justification was fitting as many items as possible into the budget. Yet every defect in the set has a likelihood and an impact, and cheapness has nothing to do with either. If the same 120 minutes were distributed in a different order, how many defects would be caught? The next lesson turns ordering into a decision: a subset chosen by the product of likelihood and impact is compared, at the same budget, against a random order, a cheapest-first order, and an order that touches the most code first. ### Risk-Based Prioritization Source: https://academia.sh/en/courses/testing-process/risk-based-prioritization Distributing the same budget across five different orders: class risk derived from the product of likelihood and impact, the density order that emerges from dividing risk by minutes, comparison against random, cheapest-first, and most-components-first orders, the estimate error's sensitivity, and the order's twenty-release return. The previous lesson built the budget-fitting plan by taking the cheapest tests in order; that ordering's only justification was fitting as many items as possible into 120 minutes. Yet every defect has a likelihood and an impact, and a test's cheapness has nothing to do with either. What would change if the same budget were distributed in a different order? This lesson turns ordering into a decision. **Risk-based prioritization** distributes the limited budget by defect classes' risk; risk here is the product of two quantities — a defect's **likelihood** of appearing and the **impact** it has when it does. The distributed resource is again 120 minutes per release (TP3); what is measured is again escaped defect and feedback time. #### The Risk Score and Its Estimate A defect's **risk score** is the product of likelihood and impact; a **class risk** is the sum of the risk scores of the defects in that class. The likelihood and impact values in the defect set come from the first lesson's seed, not invented. **TP8 (assumption) — risk score is likelihood × impact, the two quantities weighted equally.** **TP9 (assumption) — the plan does not know the actual class risk, it estimates it; the estimate is the actual value times $[1-e,\,1+e]$, and the baseline case is $e = 0{,}50$.** The actual class risk is known only after the defects surface; the ordering is done beforehand. **TP10 (assumption) — the number of processes a test spins up is an observable proxy for the number of components it touches.** The "most components first" order is built with this proxy; the process counts are read from the closing tables. ```js // library/inventory.mjs — the first lesson's inventory and defect set, in short form. // Cost quantities from the M21/K03 and M21/K04 closing tables; conversion to minutes with TP1-TP2. const RAW = [ ["T01", "unit test suite", "unit", 0, 266, 0, "rule-boundary"], ["T02", "integration test", "integration", 1, 772, 0, "schema-mismatch"], ["T03", "database migration test", "integration", 1, 21, 0, "migration-data-loss"], ["T04", "contract test", "contract", 2, 100, 0, "contract-field"], ["T05", "end-to-end browser test", "end-to-end", 2, 16477, 0, "interface-state"], ["T06", "visual verification", "end-to-end", 1, 784, 1, "visual-deviation"], ["T07", "load test", "performance", 9, 55000, 0, "data-growth"], ["T08", "static security scan", "security", 0, 75, 7, "sql-concatenation"], ["T09", "dependency scan", "security", 0, 10, 5, "known-vulnerability"], ["T10", "accessibility audit", "security", 0, 14, 6, "accessibility"], ["T11", "fault injection", "resilience", 1, 13, 5, "degraded-response"], ["T12", "recovery verification", "resilience", 2, 7, 10, "recovery-gap"], ]; export const TESTS = RAW.map(([code, name, level, processes, calls, manual, cls]) => ({ code, name, level, processes, calls, manual, cls, min: 0.5 * processes + 0.0002 * calls + 6 * manual, })); export const UNCOVERED = ["off-scenario", "semantic-drift"]; const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` #### Same Budget, Five Orders An order is a test list; items are taken in sequence until the budget fills, and what does not fit is skipped. Five orders are compared: descending by estimated risk (S1), descending by estimated risk divided by minutes (S2), cheapest first (S3, the first lesson's order), random (S4), most components touched first (S5). ```js // library/risk.mjs — same budget, five orders, same defect set. Input: library/inventory.mjs import { TESTS, UNCOVERED, defects } from "./inventory.mjs"; const BUDGET = 120, SEED = 20260731, N = 60, MULTIPLIER = 6, CANDIDATES = 20; // TP3, TP5, TP6 const DEFECTS = defects(SEED, N); const CLASSES = [...new Set([...TESTS.map((t) => t.cls), ...UNCOVERED])]; const v = (x, n = 1) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); const makeRandom = (seed) => { let x = seed; return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; }; // TP8: risk score = likelihood x impact; class risk is the risk total of that class's defects. const ACTUAL = new Map(CLASSES.map((s) => [s, DEFECTS.filter((k) => k.cls === s) .reduce((a, k) => a + k.likelihood * k.impact, 0)])); // TP9: the plan does not know the actual risk, it estimates; the estimate drifts by [1-e, 1+e]. const estimate = (e, r) => new Map(CLASSES.map((s) => [s, ACTUAL.get(s) * (1 - e + 2 * e * r())])); // From an ordered list, take what fits the budget; what does not fit is skipped, order stays. const select = (order) => { const c = []; let t = 0; for (const x of order) if (t + x.min <= BUDGET) { c.push(x); t += x.min; } return c; }; function measure(c) { const moment = new Map(); let t = 0; for (const x of c) { t += x.min; moment.set(x.cls, t); } const caught = DEFECTS.filter((k) => moment.has(k.cls)), escaped = DEFECTS.filter((k) => moment.has(k.cls) === false); return { n: c.length, code: c.map((x) => x.code).join(" "), time: t, caught: caught.length, escaped: escaped.length, rate: (100 * escaped.length) / DEFECTS.length, risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), feedback: caught.reduce((a, k) => a + moment.get(k.cls), 0) / (caught.length || 1), fix: escaped.reduce((a, k) => a + k.fix, 0) }; } const riskOrder = (T) => [...TESTS].sort((a, b) => T.get(b.cls) - T.get(a.cls)); const densityOrder = (T) => [...TESTS].sort((a, b) => T.get(b.cls) / b.min - T.get(a.cls) / a.min); const TOTAL = [...ACTUAL.values()].reduce((a, x) => a + x, 0); const BASE = UNCOVERED.reduce((a, s) => a + ACTUAL.get(s), 0); console.log(`class risk (TP8) and estimate (TP9, e = 50%) — seed ${SEED}, ${N} defects, ${CLASSES.length} classes`); console.log("class defects actual estimate test min estimate/min"); const T50 = estimate(0.5, makeRandom(8125)); for (const s of CLASSES) { const t = TESTS.find((x) => x.cls === s); console.log(s.padEnd(21) + g(DEFECTS.filter((k) => k.cls === s).length, 5) + g(ACTUAL.get(s), 8) + g(v(T50.get(s)), 8) + " " + (t ? t.code : "none") + g(t ? v(t.min, 2) : "-", 7) + g(t ? v(T50.get(s) / t.min, 2) : "-", 11)); } console.log(`total risk ${TOTAL}; the two classes no test sees ${BASE} points -> reachable risk ${TOTAL - BASE}`); const S = { S1: riskOrder(T50), S2: densityOrder(T50), S3: [...TESTS].sort((a, b) => a.min - b.min), S5: [...TESTS].sort((a, b) => b.processes - a.processes || b.calls - a.calls) }; const LABEL = { S1: "S1 risk descending (estimate)", S2: "S2 estimate/min descending", S3: "S3 cheapest first", S5: "S5 most components first" }; const r4 = makeRandom(4242), draws = []; for (let i = 0; i < 200; i += 1) { const p = [...TESTS]; for (let j = p.length - 1; j > 0; j -= 1) { const q = Math.floor(r4() * (j + 1)); [p[j], p[q]] = [p[q], p[j]]; } draws.push(measure(select(p))); } const avg = (f) => draws.reduce((a, o) => a + f(o), 0) / draws.length; // The last column is the decision's return on investment: CANDIDATES releases of run time plus the MULTIPLIER-scaled fix cost of escaped defects. console.log(`\nsame budget TP3 = ${BUDGET} min, same defect set; last column ${CANDIDATES} releases with TP5-TP6`); console.log("order test time escaped leak% escaped risk feedback total min"); const w = (x) => (Number.isInteger(x) ? String(x) : v(x, 2)); const printRow = (name, o) => console.log(name.padEnd(30) + g(w(o.n), 5) + g(v(o.time), 7) + g(w(o.escaped), 8) + g(v(o.rate), 8) + g(v(o.risk), 14) + g(v(o.feedback), 10) + g(v(CANDIDATES * o.time + MULTIPLIER * o.fix, 0), 11)); for (const k of ["S1", "S2", "S3", "S5"]) printRow(LABEL[k], measure(select(S[k]))); printRow("S4 random (200 permutations)", { n: avg((o) => o.n), time: avg((o) => o.time), escaped: avg((o) => o.escaped), rate: avg((o) => o.rate), risk: avg((o) => o.risk), feedback: avg((o) => o.feedback), fix: avg((o) => o.fix) }); console.log(`S4 spread: escaped risk best ${Math.min(...draws.map((o) => o.risk))}, ` + `worst ${Math.max(...draws.map((o) => o.risk))}; ${draws.filter((o) => o.risk <= measure(select(S.S2)).risk).length} of 200 permutations are as good as S2`); for (const k of ["S1", "S5"]) console.log(k + " selected: " + measure(select(S[k])).code); console.log("\nestimate error, TP9's sensitivity — 200 draws per e, average escaped risk"); console.log("e S1 average S2 average draws where S2 > S3 worst S2"); const S3R = measure(select(S.S3)).risk; for (const e of [0, 0.25, 0.5, 1]) { const r = makeRandom(9001); let a = 0, b = 0, beaten = 0, worst = 0; for (let i = 0; i < 200; i += 1) { const T = estimate(e, r); const o1 = measure(select(riskOrder(T))), o2 = measure(select(densityOrder(T))); a += o1.risk; b += o2.risk; worst = Math.max(worst, o2.risk); if (o2.risk > S3R) beaten += 1; } console.log(g(Math.round(100 * e) + "%", 4) + g(v(a / 200), 12) + g(v(b / 200), 13) + g(beaten, 21) + g(worst, 12)); } console.log(`comparison baseline: S3 cheapest first, escaped risk ${S3R}`); ``` ``` class risk (TP8) and estimate (TP9, e = 50%) — seed 20260731, 60 defects, 14 classes class defects actual estimate test min estimate/min rule-boundary 7 63 40.8 T01 0.05 766.99 schema-mismatch 7 65 60.2 T02 0.65 92.04 migration-data-loss 3 15 21.6 T03 0.50 42.85 contract-field 7 75 101.7 T04 1.02 99.68 interface-state 4 39 24.0 T05 4.30 5.58 visual-deviation 2 23 33.2 T06 6.66 4.99 data-growth 6 51 64.5 T07 15.50 4.16 sql-concatenation 3 51 57.3 T08 42.02 1.36 known-vulnerability 3 33 28.4 T09 30.00 0.95 accessibility 4 32 32.5 T10 36.00 0.90 degraded-response 3 21 12.9 T11 30.50 0.42 recovery-gap 1 9 5.3 T12 61.00 0.09 off-scenario 4 33 49.1 none - - semantic-drift 6 60 78.7 none - - total risk 570; the two classes no test sees 93 points -> reachable risk 477 same budget TP3 = 120 min, same defect set; last column 20 releases with TP5-TP6 order test time escaped leak% escaped risk feedback total min S1 risk descending (estimate) 9 106.7 17 28.3 156.0 48.9 8434 S2 estimate/min descending 9 100.7 18 30.0 155.0 18.2 9574 S3 cheapest first 9 89.2 18 30.0 185.0 16.5 9704 S5 most components first 9 119.7 20 33.3 197.0 61.4 11484 S4 random (200 permutations) 8.48 111.7 21.36 35.6 207.1 75.8 11220 S4 spread: escaped risk best 155, worst 377; 17 of 200 permutations are as good as S2 S1 selected: T04 T07 T02 T08 T01 T06 T10 T05 T03 S5 selected: T07 T05 T04 T12 T06 T02 T03 T01 T09 estimate error, TP9's sensitivity — 200 draws per e, average escaped risk e S1 average S2 average draws where S2 > S3 worst S2 0% 155.0 155.0 0 155 25% 155.7 156.4 0 174 50% 161.8 161.6 4 186 100% 175.5 167.3 20 204 comparison baseline: S3 cheapest first, escaped risk 185 ``` #### Counting Numbers versus Counting Risk Of sixty defects, the caught counts are 43, 42, 42, 40, averaging 38.64; the escaped defect **count** barely separates the five orders: 17, 18, 18, 20, averaging 21.36. The escaped defects' **risk total**, though, does separate them: 156, 155, 185, 197, averaging 207.1. What makes the difference is that one counts every defect as one, the other counts by weight. S2 and S3 escape the same number of defects, but S3's escapees carry 30 more risk points. The comparison's floor is in the table too: of the total 570 risk points, 93 belong to the two classes no test sees, so the range the order can actually move is 477 points. S2 misses 62 points of that range, S3 misses 92 — a 1.48x difference from reordering alone. A sentence like "escaped risk 155," written without looking at the floor, hides this ratio. #### Risk Order versus Density Order S1 and S2 use the same estimate and arrive at nearly the same escaped risk (156 against 155). The distinction lies elsewhere: S1's feedback time is 48.9 minutes, S2's is 18.2 — a 2.7x difference. The reason is visible in the orders chosen. S1 places the 42.02-minute static security scan fourth, because that class's estimated risk is high; S2 pushes the same test to the end, because once divided by minutes its estimate/min value is 1.36. A high-risk but expensive item placed near the front of the order delays the reporting moment of every test that follows it. **Risk order alone is the wrong order; the divisor is minutes.** T01's estimate/min value is 766.99, T12's is 0.09 — a difference no risk ordering by itself can see. S5 shows this from the other side: the most-components-first order puts the load test and the recovery verification among its items, spending 76.5 of the 120 minutes on these two tests alone, and escapes 20 defects. Touching a lot of code is not the same as seeing a lot of risk. S4, in turn, shows that the order itself is a gain: the average of two hundred permutations is 207.1, the worst is 377, and only 17 are as good as S2. A plan with no order written down is, on average, a random plan. #### The Estimate's Margin of Error The ordering rests on the estimate, not the actual risk, and the estimate is wrong. The sensitivity scan draws two hundred samples at each error level. At zero error the two orders meet at 155 — this is the upper bound. When the error rises to ±50%, S2's average becomes 161.6, and only four of the two hundred draws are worse than the cheapest-first order's 185. At ±100% error the average stays at 167.3, the number of beaten draws rises to 20, and the worst draw is 204. The gain erodes slowly, because the order uses the estimate's **rank**, not its **value**: even when a class's risk is misestimated by a factor of two, its place in the list mostly stays the same. The rule that follows from this is writing down the estimate's source — the class distribution in the defect history, change intensity, the number of affected users. A risk order with no written source is a preference list; the ±100% row gives the price of that preference. #### The Return on the Decision The table's last column — twenty releases of run time plus the escaped defects' six-fold fix cost — tests the ordering with a third quantity: S1 is cheapest at 8,434 minutes, S2 at 9,574, S3 at 9,704, random at 11,220, most-components at 11,484. S1's total cost is lower even though its escaped risk is higher than S2's, because the risk score does not measure fix cost: in the defect set, fix minutes are drawn independent of likelihood and impact. No single order can optimize all three quantities at once; which one gets optimized is written into the plan. What a red result means also comes from the order. When the tests of the classes with the highest risk — the contract test and the integration test — turn red, the release candidate does not advance; the items further down the queue open a record. The order itself is also a maintenance item: as the defect history changes, class risk changes, and a risk order that is not re-derived turns into a habit after a few releases and its gain drops to the values in the ±100% row. #### Summary - Risk score is the product of likelihood and impact (TP8); the plan does not know class risk, it estimates it (TP9). - Escaped defect count separates the five orders between 17 and 21.36, escaped risk between 155 and 207.1; unweighted counting hides the ordering's gain. - Of the total 570 risk points, 93 belong to the two classes no test sees; the range the order can move is 477, and within that range the difference between S2 and S3 is 1.48x. - Risk order pushes an expensive item to the front and stretches feedback to 48.9 minutes; once the same estimate is divided by minutes, escaped risk stays at 155 and feedback drops to 18.2 minutes. - When estimate error rises to ±100%, the density order's average becomes 167.3 and 20 of the two hundred draws lose to the cheapest-first order; the gain erodes slowly. - Risk score does not measure fix cost: the order with the lowest escaped risk may not be the cheapest order. #### Next Step All five orders chose from within the same twelve tests, and none of them touched the 93-point floor. 33 of those points belong to the `off-scenario` class: four defects that surface on paths no written scenario passes through. The way to see this class is not to change the order but to change the test form — a session whose script is not written in advance, whose direction is decided during the test itself. The next lesson turns that session into a budget item: how the narrowness of scope changes what the session finds, and how much of what it finds converts into the scripted set. ### Exploratory Testing Source: https://academia.sh/en/courses/testing-process/exploratory-testing Turning testing whose script is not written in advance into a budget item: dividing the session's charter between width and depth, the defect class the scripted set cannot see becoming visible only through depth, the rate at which a found defect converts into a script, and measuring escaped defect and net cost across a twenty-release horizon. All five orders in the previous lesson chose from within the same twelve tests, and none of them touched the 93-point floor. One of the two classes producing that floor is `off-scenario`: defects that surface on paths no written scenario passes through. A class cannot be found by reordering as long as no test is written to see it. **Exploratory testing** is a test form whose script is not written in advance, whose direction is decided by what is seen during the test. Being free does not mean unstructured: the tester writes a **charter** — which area, for how long, looking for what. This lesson builds that charter as a budget distribution; the resource distributed is human attention, and it cannot be spent in two directions at once. #### Width and Depth Draw From the Same Budget The system's testable surface has two dimensions: how many **paths** are walked (width) and how far each path is followed (depth). A session's cost is the product of the two, so at a fixed budget one is traded for the other. **TP11 (assumption) — the per-release exploratory session budget is 120 minutes**; this is the same size as the first lesson's automated test budget (TP3), and the choice is tested with a sensitivity scan. **TP12 (assumption) — testing one path to one depth level takes 6 minutes.** The budget is thus 20 path-depth units. **TP13 (assumption) — every defect sits on one path at one depth; the two classes that fall outside scenarios are deep by definition (3–5), the rest are spread across 1–5.** This is a constraint that comes from the class's own definition: a defect sitting on a shallow path that written scenarios already pass through would already be visible in the scripted set. **TP14 (assumption) — a found class is converted into the scripted set starting the next release**; for an untested class this is 45 minutes to write plus 5 minutes of run time per release, for a class that already has a test in the inventory there is no writing, and the cost is that test's per-release minutes. ```js // library/inventory.mjs — the fields this lesson needs from the first lesson's inventory: each // test's defect class and cost quantities (processes, calls, manual steps; from the M21/K03 // and K04 closing tables). Conversion to minutes with the first lesson's TP1 and TP2, unchanged. const RAW = [["T01", "rule-boundary", 0, 266, 0], ["T02", "schema-mismatch", 1, 772, 0], ["T03", "migration-data-loss", 1, 21, 0], ["T04", "contract-field", 2, 100, 0], ["T05", "interface-state", 2, 16477, 0], ["T06", "visual-deviation", 1, 784, 1], ["T07", "data-growth", 9, 55000, 0], ["T08", "sql-concatenation", 0, 75, 7], ["T09", "known-vulnerability", 0, 10, 5], ["T10", "accessibility", 0, 14, 6], ["T11", "degraded-response", 1, 13, 5], ["T12", "recovery-gap", 2, 7, 10]]; export const TESTS = RAW.map(([code, cls, processes, calls, manual]) => ({ code, cls, min: 0.5 * processes + 0.0002 * calls + 6 * manual })); const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` #### Four Scopes, Twenty Releases Four distributions divide the same 120 minutes: twenty paths to depth one (E1), ten paths to depth two (E2), five paths to depth four (E3), four paths to depth five (E4). Which paths a scope covers is a matter of information — the plan does not know where the defect is, so path selection is seeded and random, and the average of two hundred draws is taken. The measurement runs across a twenty-release horizon: three defects surface each release, one session runs, and the found class enters the scripted set starting the next release. ```js // library/exploration.mjs — measures the exploratory session as a budget distribution. // Input: library/inventory.mjs (test set and defect set inherited from the first lesson). import { TESTS, defects } from "./inventory.mjs"; const SEED = 20260731, N = 60, PATHS = 20, UNIT = 6, SESSION = 120, DRAWS = 200; const MULTIPLIER = 6, CANDIDATES = 20, SHARE = N / CANDIDATES; // TP5, TP6: 3 defects/release // The second lesson's density order's nine budget-fitting items; the remaining classes are unscripted. const SCRIPTED = new Set(["T01", "T02", "T03", "T04", "T05", "T06", "T07", "T08", "T09"] .map((k) => TESTS.find((t) => t.code === k).cls)); // Unscripted classes split in two: ones with a test in the inventory that misses the budget (HAS_TEST) // and the two no test sees at all (DEEP). const HAS_TEST = new Map(TESTS.filter((t) => SCRIPTED.has(t.cls) === false).map((t) => [t.cls, t])); const DEEP = new Set(["off-scenario", "semantic-drift"]); const v = (x, n = 1) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); const makeRandom = (t) => { let x = t; return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; }; // TP13: every defect sits on one path at one depth. The two off-scenario classes are deep (3-5) // by definition, the rest are 1-5. Path and depth come from seed 31417. const r0 = makeRandom(31417); const DEFECTS = defects(SEED, N).map((k) => ({ ...k, path: 1 + Math.floor(r0() * PATHS), depth: DEEP.has(k.cls) ? 3 + Math.floor(r0() * 3) : 1 + Math.floor(r0() * 5) })); const UNSCRIPTED = DEFECTS.filter((k) => SCRIPTED.has(k.cls) === false); // A session walks p paths to depth d; its cost is p*d*UNIT minutes. A defect is found if its // path was walked and its depth does not exceed d; the find moment comes from that path's order. const walk = (r, p) => { const paths = Array.from({ length: PATHS }, (_, j) => j + 1); for (let j = PATHS - 1; j > 0; j -= 1) { const q = Math.floor(r() * (j + 1)); [paths[j], paths[q]] = [paths[q], paths[j]]; } return paths.slice(0, p); }; // Horizon: CANDIDATES releases, SHARE defects per release, and one session. A found class is // converted to the scripted set starting the next release (TP14), and no defect from it escapes after. function horizon(p, d, seed, budget = SESSION) { const r = makeRandom(seed), pathCount = Math.min(p, Math.floor(budget / (UNIT * d))), o = []; for (let c = 0; c < DRAWS; c += 1) { const converted = new Map(); let found = 0, moment = 0, escaped = []; for (let s = 1; s <= CANDIDATES; s += 1) { const place = new Map(walk(r, pathCount).map((y, i) => [y, (i + 1) * d * UNIT])); const newlyFound = []; for (const k of DEFECTS.slice((s - 1) * SHARE, s * SHARE)) { if (SCRIPTED.has(k.cls) || converted.has(k.cls)) continue; if (place.has(k.path) && k.depth <= d) { found += 1; moment += place.get(k.path); newlyFound.push(k.cls); } else escaped.push(k); } for (const x of newlyFound) if (converted.has(x) === false) converted.set(x, s); } const cost = [...converted].reduce((a, [x, s]) => a + (CANDIDATES - s) * (HAS_TEST.has(x) ? HAS_TEST.get(x).min : 5) + (HAS_TEST.has(x) ? 0 : 45), 0); o.push({ found, moment: moment / (found || 1), converted, cost, untested: [...converted.keys()].filter((x) => DEEP.has(x)).length, outOfBudget: [...converted.keys()].filter((x) => HAS_TEST.has(x)).length, escaped: escaped.length, risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), fix: escaped.reduce((a, k) => a + k.fix, 0) }); } const avg = (f) => o.reduce((a, x) => a + f(x), 0) / DRAWS; return { p: pathCount, d, budget, found: avg((x) => x.found), moment: avg((x) => x.moment), untested: avg((x) => x.untested), outOfBudget: avg((x) => x.outOfBudget), escaped: avg((x) => x.escaped), risk: avg((x) => x.risk), cost: avg((x) => x.cost), fix: avg((x) => x.fix) }; } const BASELINE_FIX = UNSCRIPTED.reduce((a, k) => a + k.fix, 0); const net = (o) => MULTIPLIER * (BASELINE_FIX - o.fix) - CANDIDATES * o.budget - o.cost; console.log(`defect set seed ${SEED}, ${N} defects; path and depth seed 31417, ${PATHS} paths`); console.log(`scripted set is the second lesson's nine items -> ${SCRIPTED.size} classes closed; ` + `${UNSCRIPTED.length} unscripted defects across the horizon, ${UNSCRIPTED.filter((k) => DEEP.has(k.cls)).length} in the two untested classes`); console.log("depth distribution (defect count): " + [1, 2, 3, 4, 5] .map((d) => d + ":" + DEFECTS.filter((k) => k.depth === d).length).join(" ")); console.log(`\nsession budget TP11 = ${SESSION} min = ${SESSION / UNIT} path-depth units; ` + `${CANDIDATES} releases, average of ${DRAWS} draws`); console.log("scope path depth found out-of-budget untested conversion find moment"); const D = [[20, 1], [10, 2], [5, 4], [4, 5]].map(([p, d], i) => horizon(p, d, 5501 + i)); D.forEach((o, i) => console.log(`E${i + 1} ${o.p} paths x depth ${o.d}`.padEnd(24) + g(o.p, 6) + g(o.d, 10) + g(v(o.found, 2), 10) + g(v(o.outOfBudget, 2), 15) + g(v(o.untested, 2), 10) + g(v((o.outOfBudget + o.untested) / o.found, 2), 12) + g(v(o.moment), 13))); console.log("\nend of horizon — the second lesson's density order alone was letting 18 defects escape"); console.log("scope escaped defect escaped risk session min conversion min gain from escaped defect net min"); D.forEach((o, i) => console.log(`E${i + 1}`.padEnd(9) + g(v(o.escaped, 2), 16) + g(v(o.risk), 15) + g(CANDIDATES * o.budget, 14) + g(v(o.cost, 0), 16) + g(v(MULTIPLIER * (BASELINE_FIX - o.fix), 0), 26) + g(v(net(o), 0), 9))); console.log(`without a session, baseline: 18 escaped, 155 risk, fix ${BASELINE_FIX} min (${MULTIPLIER * BASELINE_FIX} min with TP5)`); console.log("\nTP11's sensitivity — E4's depth (5) across three budgets; the 120 min row is E4 itself"); console.log("budget min paths walked found untested escaped defect net min"); for (const b of [60, 120, 240]) { const o = horizon(PATHS, 5, 5504, b); console.log(g(b, 10) + g(o.p, 15) + g(v(o.found, 2), 8) + g(v(o.untested, 2), 11) + g(v(o.escaped, 2), 17) + g(v(net(o), 0), 9)); } ``` ``` defect set seed 20260731, 60 defects; path and depth seed 31417, 20 paths scripted set is the second lesson's nine items -> 9 classes closed; 18 unscripted defects across the horizon, 10 in the two untested classes depth distribution (defect count): 1:18 2:10 3:13 4:8 5:11 session budget TP11 = 120 min = 20 path-depth units; 20 releases, average of 200 draws scope path depth found out-of-budget untested conversion find moment E1 20 paths x depth 1 20 1 3.00 3.00 0.00 1.00 64.4 E2 10 paths x depth 2 10 2 2.04 2.04 0.00 1.00 63.0 E3 5 paths x depth 4 5 4 2.53 1.35 1.12 0.98 73.3 E4 4 paths x depth 5 4 5 2.75 1.29 1.42 0.98 74.8 end of horizon — the second lesson's density order alone was letting 18 defects escape scope escaped defect escaped risk session min conversion min gain from escaped defect net min E1 12.00 114.0 2400 1281 2610 -1071 E2 13.85 126.5 2400 867 1881 -1386 E3 11.32 104.1 2400 729 2831 -298 E4 10.39 95.2 2400 735 3280 145 without a session, baseline: 18 escaped, 155 risk, fix 1260 min (7560 min with TP5) TP11's sensitivity — E4's depth (5) across three budgets; the 120 min row is E4 itself budget min paths walked found untested escaped defect net min 60 2 1.57 0.90 13.81 202 120 4 2.75 1.42 10.39 145 240 8 4.09 1.79 6.27 -923 ``` #### The Class Width Cannot See E1 walks all twenty paths and finds everything it can find: three unscripted defects sitting at depth 1. These three are a whole number, unchanged from draw to draw — at full width no randomness remains. Against that, E1's `untested` column is **0.00**, and stays 0.00 no matter how many draws are run. The reason is not a probability but a constraint: the two off-scenario classes' defects never sit at a depth shallower than 3, because if they did, written scenarios would already pass through there. E4 looks at only four paths with the same budget and finds fewer defects (2.75 against 3.00) — but 1.42 of what it finds come from the untested classes. **Width finds more defects, depth finds a different kind of defect.** The difference between the two distributions is not a difference in efficiency but a difference in reach; E1's shortfall is not that it works less, but that there is a place it never looks. The find moment shows the price of this reach: between 63.0 and 74.8 minutes. The previous lesson's density order was reporting a defect at an average of 18.2 minutes; the exploratory session reports with a delay close to four times that, and the delay grows with depth, because reaching the end of a deep path takes time. #### Conversion Rate and Two Kinds of Finding The rate at which a found defect converts into the scripted set falls between 0.98 and 1.00 across all four distributions: nearly every defect the session finds leads to a new scripted test. This is not as good news as it looks, because what converts is of two kinds. An **out-of-budget class** is one that has a test in the inventory but fell out of the plan because it does not fit 120 minutes — all three classes E1 finds are of this kind. A finding like this brings no new information; it re-asks the first lesson's budget decision, and its conversion brings back that decision's cost too: E1's conversion cost is 1,281 minutes, because the converted tests run every remaining release. An **untested class**, by contrast, is one with no counterpart in the inventory; its conversion is 45 minutes to write plus 5 minutes per release, and it is the small part of E4's 735-minute conversion cost. The distinction changes the decision. Finding an out-of-budget class is the plan's outcome, not the session's, and the answer to it is a budget discussion, not a session. Finding an untested class, though, is the session's own product, and it leaves a permanent test behind. #### The Number at the End of the Horizon At the end of twenty releases, the previous lesson's order alone was letting 18 defects escape, at 155 risk points. The exploratory session lowers that number in every distribution: to 13.85 for E2, 12.00 for E1, 11.32 for E3, 10.39 for E4. Escaped risk drops from 155 to 95.2 — meaning the session does not just let fewer defects escape, what escapes is also lighter, because the two untested classes sat at the heavy end of what escaped. The net column ties this to a decision. The twenty-release session cost is 2,400 minutes, and only E4 turns positive: 145 minutes. E1 lowers escaped defect to 12 but loses 1,071 minutes, because everything it finds requires an expensive test to be brought back into the budget. **A distribution lowering the escaped defect count does not justify it by itself; the cost of that reduction is counted too.** The TP11 scan gives this its limit. When the budget drops to 60 minutes, escaped defect rises to 13.81 but net rises to 202; raised to 240 minutes, escaped defect drops to 6.27 and net becomes −923. Escaped defect keeps falling with the budget, net does not. A plan that wants zero escaped defects exhausts the budget; this is this course's repeating trade-off, in this lesson's shape. What a red result means is also different here. An exploratory session does not give a pass or a fail, it gives a list of findings; every row in that list either opens a record or gives birth to a test. The authority to stop a release does not sit with the session — the session's product is a scripted test that can stop a release in later releases. #### Summary - A session's charter is a budget: the product of width and depth is fixed (TP11–TP12), one is traded for the other. - Full width (E1) is certain to find the three defects at depth 1 but never sees the two untested classes; this is not a probability but a constraint from the class's own definition (TP13). - Deep scope (E4) finds fewer defects (2.75 against 3.00) but 1.42 of what it finds come from untested classes; the difference is one of reach, not efficiency. - Conversion rate is 0.98–1.00, but most of what converts is out-of-budget classes: not new information, but the budget decision being re-asked. - Across twenty releases, escaped defect drops from 18 to 10.39, escaped risk from 155 to 95.2; at a session cost of 2,400 minutes, only deep scope turns positive (145 minutes). - When the budget is raised to 240 minutes, escaped defect drops to 6.27 and net becomes −923: escaped defect keeps dropping with the budget, net does not. #### Next Step So far, three lessons have distributed testing itself: which test, in what order, at what scope. All three share a silent assumption — that a defect is a defect, that a piece of work being done is clear. Yet the exploratory session's findings in the `semantic-drift` class open exactly this up to dispute: the tester counts it a defect, the developer counts it expected behavior, and the product side agrees with neither. In this case, escaped defect count is not a measurement, it is a disagreement. The next lesson takes up the definition of done and acceptance criteria: who writes the criterion, when it closes, and how many units of reopened work a criterion left ambiguous produces. ### Acceptance Criteria Source: https://academia.sh/en/courses/testing-process/acceptance-criteria Writing the definition of done as a data structure: the acceptance criterion's subject, measure, and limit slots, the reopened-work count and discovery phase that change with who fills a slot and when, the product side's attention being a limited budget, and a criterion written at closeout only confirming observed behavior. The previous lesson's exploratory session produced findings in the `semantic-drift` class, but whether these findings were defects was not a measurement question: the tester counts a defect, the developer counts expected behavior, and the product side agrees with neither. In a case like this, escaped defect count is not a measure, it is a disagreement. What turns disagreement into a measure is the **acceptance criterion**: a condition written in advance that says when a work item is done. Turning an acceptance criterion into an automated test was built in the Unit Testing and Test-Driven Development course; the question here is different — **who** writes the criterion and **when** it closes. This lesson builds those two questions as a budget distribution; the resource distributed is the product side's attention, and it is measured by **rework**, meaning work that is opened again. (In the Resilience and Reliability course, a failback is returning to the primary node after a failover; what is meant here by rework is a piece of work counted as closed being reopened.) #### The Criterion's Three Slots An acceptance criterion fills three slots: **subject** says what is observed, **measure** says how it is observed, **limit** says what value is accepted. A **definition of done** is the state where all of these criteria are met. Not every criterion needs all three slots, and which are required comes from the criterion's type: an existence criterion needs subject and measure ("a delay notification is sent"), a numeric criterion also needs a limit ("five days after the due date"), a semantic criterion needs a limit **and** requires that limit not be derived from observation ("the notification goes to the member who holds the loan"). The last item is this lesson's sharpest point. If a criterion is written at closeout by looking at the system's observed behavior, the limit is set equal to that behavior; the criterion never turns red, because what it tests is its own source. **TP15 (assumption) — 20 work items, 4 acceptance criteria per item, 80 criteria in total; the type distribution comes from seed 6060.** The defect set's 60 defects are distributed to items three at a time. **TP16 (assumption) — the time the product side can set aside for writing criteria across the horizon is 300 minutes; reviewing one criterion takes 12 minutes.** This is the distributed resource, and its sensitivity is scanned. **TP17 (assumption) — a piece of rework's cost depends on the phase where it surfaces:** 5 minutes at write time, 20 during development, 60 at acceptance review, 180 after release. ```js // library/defects.mjs — the first lesson's defect set, unchanged (seed 20260731, 60 defects, // 14 classes). This lesson does not use the inventory's cost columns, only the defect set. const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` #### Five Policies Five policies distribute the criterion to different hands and different moments. A slot left missing surfaces at the phase when the side that will fill it first looks at the criterion: the product side fills a missing limit, the tester fills a missing measure. ```js // library/criteria.mjs — builds the acceptance criterion as a three-slot data structure and // measures the reopened-work return on who writes it and when. Input: library/defects.mjs import { defects } from "./defects.mjs"; const ITEMS = 20, PER_ITEM = 4, SEED = 20260731, N = 60, PRODUCT_MIN = 12, PRODUCT_BUDGET = 300; const COST = [5, 20, 60, 180]; // TP17: reopened work's minutes per phase const PHASE = ["write time", "development", "acceptance review", "after release"]; let MULTIPLIER = 6; // TP5: escaped defect's fix multiplier const g = (x, n) => String(x).padStart(n); // TP15: 20 work items x 4 acceptance criteria = 80 criteria. Criterion type sets required slots: // existence (subject+measure), numeric (subject+measure+limit), semantic (all three; limit not from observation). let x = 6060; const r = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const TYPE = ["existence", "numeric", "semantic"]; const CRITERIA = Array.from({ length: ITEMS * PER_ITEM }, (_, i) => { const p = r(); return { id: i, item: Math.floor(i / PER_ITEM), type: TYPE[p < 0.45 ? 0 : p < 0.8 ? 1 : 2] }; }); const REQUIRED = { existence: ["subject", "measure"], numeric: ["subject", "measure", "limit"], semantic: ["subject", "measure", "limit"] }; // Defects are distributed to items three at a time. A semantic-drift defect by definition sits // under a semantic criterion — what makes it a defect is not a rule in the code but what the // criterion says; if the item has no semantic criterion, one of its criteria is marked semantic. const DEFECTS = defects(SEED, N).map((k, i) => ({ ...k, item: Math.floor(i / (N / ITEMS)) })); const SEMANTIC = DEFECTS.filter((k) => k.cls === "semantic-drift"); for (const k of SEMANTIC) { const itemCriteria = CRITERIA.filter((o) => o.item === k.item); k.criterion = itemCriteria.find((o) => o.type === "semantic") || Object.assign(itemCriteria[Math.floor(r() * itemCriteria.length)], { type: "semantic" }); } // Policy: which slots are filled for each criterion, whether product fills the limit slot, whether // the limit is derived from observation, and the phase at which the side covering the gap first reads it. function evaluate(name, slots, productReviews, fromObservation, productPhase, testerPhase, writeMin) { const status = CRITERIA.map((o) => { const filled = slots(o); const missing = REQUIRED[o.type].filter((y) => filled.includes(y) === false); const judges = missing.length === 0 && (o.type !== "semantic" || fromObservation === false); return { ...o, judges, hasLimit: filled.includes("limit"), phase: missing.length ? productPhase : judges ? -1 : testerPhase }; }); const open = status.filter((o) => o.judges === false); const escaped = SEMANTIC.filter((k) => status[k.criterion.id].judges === false); const productHour = productReviews ? PRODUCT_MIN * status.filter((o) => o.hasLimit).length : 0; return { name, productHour, writeMin: writeMin * CRITERIA.length + productHour, reopened: open.length, item: new Set(open.map((o) => o.item)).size, reworkMin: open.reduce((a, o) => a + COST[o.phase], 0), distribution: [0, 1, 2, 3].map((s) => open.filter((o) => o.phase === s).length), escaped: escaped.length, risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), fix: MULTIPLIER * escaped.reduce((a, k) => a + k.fix, 0) }; } const TWO = ["subject", "measure"], THREE = ["subject", "measure", "limit"]; const POLICIES = () => [evaluate("Y1 developer writes, product does not review", () => TWO, false, false, 2, 2, 5), evaluate("Y2 product reviews every criterion, upfront", () => THREE, true, false, 0, 1, 5), evaluate("Y3 tester writes at closeout", () => THREE, false, true, 3, 2, 8), evaluate("Y4 no criteria, one-line definition", () => [], false, false, 3, 3, 1), evaluate("Y5 product reviews only semantic criteria", (o) => (o.type === "semantic" ? THREE : TWO), true, false, 2, 2, 5)]; const total = (o) => o.writeMin + o.reworkMin + o.fix; const fitsBudget = (t) => { const u = POLICIES().map((o, i) => [t[i], o.productHour <= PRODUCT_BUDGET, i]) .filter(([, s]) => s); return "Y" + (u.sort((a, b) => a[0] - b[0])[0][2] + 1); }; console.log(`${ITEMS} work items x ${PER_ITEM} acceptance criteria = ${CRITERIA.length} criteria (seed 6060); type: ` + TYPE.map((t) => `${t} ${CRITERIA.filter((o) => o.type === t).length}`).join(", ")); console.log(`defect set seed ${SEED}, ${N} defects; ${SEMANTIC.length} semantic drift, ` + `${SEMANTIC.reduce((a, k) => a + k.likelihood * k.impact, 0)} risk points; ` + `across ${new Set(SEMANTIC.map((k) => k.item)).size} separate work items`); console.log(`TP16 product budget ${PRODUCT_BUDGET} min (${PRODUCT_MIN} min per criterion); ` + `TP17 rework: ` + PHASE.map((a, i) => `${a} ${COST[i]} min`).join(", ")); console.log("\npolicy product min budget reopened item discovery phase"); for (const o of POLICIES()) console.log(o.name.padEnd(46) + g(o.productHour, 14) + g(o.productHour <= PRODUCT_BUDGET ? "fits" : "over", 8) + g(o.reopened, 10) + g(o.item, 6) + " " + (o.reopened ? PHASE[o.distribution.indexOf(Math.max(...o.distribution))] : "-")); console.log("\ncost on the same defect set"); console.log("policy write min rework min escaped escaped risk fix min total min"); for (const o of POLICIES()) console.log(o.name.padEnd(46) + g(o.writeMin, 11) + g(o.reworkMin, 13) + g(o.escaped, 9) + g(o.risk, 14) + g(o.fix, 9) + g(total(o), 11)); console.log("\nTP16's sensitivity to product budget — product reviews semantic criteria first, then numeric ones"); console.log("product min criteria reviewed reopened escaped total min"); const order = [...CRITERIA].sort((a, b) => TYPE.indexOf(b.type) - TYPE.indexOf(a.type)); for (const b of [0, 204, 300, 960]) { const covered = new Set(order.slice(0, Math.floor(b / PRODUCT_MIN)).map((o) => o.id)); const o = evaluate("", (t) => (covered.has(t.id) ? THREE : TWO), true, false, 2, 2, 5); console.log(g(b, 7) + g(covered.size, 16) + g(o.reopened, 13) + g(o.escaped, 8) + g(total(o), 12)); } console.log("\nTP5's sensitivity: which policy is cheapest as the escaped defect multiplier changes"); console.log("mult " + ["Y1", "Y2", "Y3", "Y4", "Y5"].map((a) => g(a, 8)).join("") + " cheapest cheapest within budget"); for (const mult of [1, 6, 20]) { MULTIPLIER = mult; const t = POLICIES().map(total); console.log(g(mult, 3) + t.map((n) => g(n, 8)).join("") + g("Y" + (t.indexOf(Math.min(...t)) + 1), 10) + g(fitsBudget(t), 22)); } ``` ``` 20 work items x 4 acceptance criteria = 80 criteria (seed 6060); type: existence 34, numeric 29, semantic 17 defect set seed 20260731, 60 defects; 6 semantic drift, 60 risk points; across 5 separate work items TP16 product budget 300 min (12 min per criterion); TP17 rework: write time 5 min, development 20 min, acceptance review 60 min, after release 180 min policy product min budget reopened item discovery phase Y1 developer writes, product does not review 0 fits 46 20 acceptance review Y2 product reviews every criterion, upfront 960 over 0 0 - Y3 tester writes at closeout 0 fits 17 14 acceptance review Y4 no criteria, one-line definition 0 fits 80 20 after release Y5 product reviews only semantic criteria 204 fits 29 19 acceptance review cost on the same defect set policy write min rework min escaped escaped risk fix min total min Y1 developer writes, product does not review 400 2760 6 60 2970 6130 Y2 product reviews every criterion, upfront 1360 0 0 0 0 1360 Y3 tester writes at closeout 640 1020 6 60 2970 4630 Y4 no criteria, one-line definition 80 14400 6 60 2970 17450 Y5 product reviews only semantic criteria 604 1740 0 0 0 2344 TP16's sensitivity to product budget — product reviews semantic criteria first, then numeric ones product min criteria reviewed reopened escaped total min 0 0 46 6 6130 204 17 29 0 2344 300 25 21 0 1960 960 80 0 0 1360 TP5's sensitivity: which policy is cheapest as the escaped defect multiplier changes mult Y1 Y2 Y3 Y4 Y5 cheapest cheapest within budget 1 3655 1360 2155 14975 2344 Y2 Y3 6 6130 1360 4630 17450 2344 Y2 Y5 20 13060 1360 11560 24380 2344 Y2 Y5 ``` #### The Cheapest Policy on Paper Is Not in the Budget Y2 writes every criterion together with the product side and produces no rework in the table: at 1,360 minutes it is the cheapest of the five policies. But the product column says 960 minutes, and TP16's budget is 300 — the policy **cannot be carried out.** The best of the five rows is not an option; it is a ceiling. Among those that fit the budget, Y5 stands out: the product side reviews only the seventeen semantic criteria, spends 204 minutes, and drops escaped defect to zero. Y1 uses none of that budget and lets six defects through to release; its total cost is 6,130 minutes, 2.6 times Y5's. The difference is where 204 minutes of attention gets placed. The product budget scan gives the shape of this distribution. The first 204 minutes remove **all** of the escaped defect (6 → 0), because those minutes go to semantic criteria. The remaining 756 minutes save no further defect; they only drop rework from 29 to 0 and pull the total from 2,344 to 1,360. The same resource's first part and last part buy different things: the first part buys escaped defect, the last part buys rework. #### The Criterion Written at Closeout Y3 looks reasonable in the table: it spends no product budget, produces only seventeen rework items, and at 4,630 minutes is cheaper than Y1. Yet it still lets six defects through to release. The reason is not that the criterion is missing — all three of its slots are filled. The reason is that the limit is **derived from observation**: the tester writes the criterion at closeout by watching the running system, and records what the system does as acceptable. The criterion does not turn red, because it is derived from the very behavior it tests. **The number of criteria written is not a quality indicator; the criterion's source is the indicator.** Y4 writes no criteria at all and leaves all eighty of the eighty criteria to be disputed after release: 14,400 minutes of rework, all twenty of the twenty items reopened. The discovery phase column is the one thing that separates this policy from the others — not the count, the moment. The same disagreement costs 60 minutes at acceptance review, 180 minutes after release. #### The Return on the Decision In this lesson, the return on the decision is a directly measured quantity. Y1 reopens all twenty of the twenty items, Y5 nineteen of them, Y3 fourteen, Y2 none. The number of reopened items changes little across policies; what changes is **which phase** the reopening happens at, and how many defects reach release. The TP5 scan ties the decision to a context. When an escaped defect's fix cost equals a caught one's (multiplier 1), the cheapest policy that fits the budget is Y3 — a criterion written at closeout is defensible while defects are cheap. At multipliers 6 and 20, Y5 wins and widens the gap. So the answer to who writes the criterion depends on what an escaped defect costs the team; the policy cannot be chosen without knowing that number. #### Summary - An acceptance criterion is three slots: subject, measure, limit. The criterion's type determines which slots are required; a semantic criterion additionally requires the limit not be derived from observation. - The product side's attention is a limited resource (TP16, 300 minutes); the policy that reviews every criterion asks for 960 minutes and does not fit the budget — the cheapest row in the table is not an option, it is a ceiling. - The same budget's first 204 minutes remove all escaped defect, the remaining 756 minutes only lower rework; a resource's first and last parts buy different things. - A criterion written at closeout fills all three slots but never turns red because it takes its limit from observed behavior, and it lets six defects through to release. - When no criterion is written, the disagreement does not disappear, only its phase shifts: the same rework is 60 minutes at acceptance review, 180 minutes after release. - Policy choice depends on the escaped defect multiplier: at multiplier 1, writing at closeout is cheapest within budget; at multipliers 6 and 20, targeted product attention is. #### Next Step Every criterion here was written by parties inside the team: the developer, the tester, the product side. All three stand on the side designing the system, and all three use the same vocabulary. Yet the expectation that must ultimately be met by the criterion sits outside the team, and that expectation was written into no slot — how many steps a library clerk expects a loan transaction to take, when a member finds a delay notification meaningful. The next lesson manages that verification: how acceptance scope is distributed across user tasks, how the returning feedback is separated into defect, request, and misunderstanding, and what each class costs the team. ### User Acceptance Testing Source: https://academia.sh/en/courses/testing-process/user-acceptance-testing Managing end-user verification as a distribution decision: splitting acceptance scope four ways across user tasks, separating the returning feedback into defect, request, and misunderstanding, each class's cost measured in minutes, and why feedback volume is not a quality measure. Every criterion in the previous lesson was written by parties inside the team. All three stand on the side designing the system and use the same vocabulary; yet the expectation the criterion must ultimately meet sits outside the team. How many steps a library clerk expects a loan transaction to take, or when a member finds a delay notification meaningful, was written into no slot. **User acceptance testing** has the end user answer this gap. This lesson builds it as a distribution decision: the resource distributed is the user's time, the distribution unit is the **user task**, and what comes back is not of one kind. Part of the feedback is a defect, part is a new **request**, and part is a **misunderstanding** that misreads the system's correct behavior — and the three cost differently. #### What Reaches the Acceptance Stage Acceptance testing is the last gate: a defect an earlier gate already saw does not reach here. The second lesson's nine budget-fitting items closed nine classes; the remaining five classes' 18 defects reach this stage. The fourth lesson's Y5 policy also left behind a state: twenty-nine numeric criteria have an empty limit. These two inheritances determine feedback's class. When a user observes a criterion, the outcome splits three ways: at **a criterion with an empty limit**, the user states their own limit, and this is a request; at **a complete criterion with a defect underneath**, they report a defect; at **a complete, flawless existence criterion whose measure is not visible to the user**, they compare what they see to their own expectation and produce a misunderstanding. **TP18 (assumption) — 20 work items are distributed across 12 user tasks, each task getting at least one item; usage share comes from seed 4712.** **TP19 (assumption) — the acceptance budget is 600 user minutes, observing one criterion takes 8 minutes**; the budget is thus 75 criterion observations, and does not cover all eighty criteria. **TP20 (assumption) — 30% of requests are accepted and turn into work**, the rest stay on record. **TP21 (assumption) — deciding on a request takes 30 minutes, doing an accepted request takes 120, resolving a misunderstanding takes 20 minutes; a defect's cost is the fix minutes in the defect set.** ```js // library/defects.mjs — the first lesson's defect set, unchanged (seed 20260731, 60 defects, // 14 classes). This lesson does not use the inventory's cost columns, only the defect set. const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` #### Four Scope Distributions The same 600 minutes is split by four rules: equal (K1), by usage share (K2), toward the task with the most incomplete criteria (K3), and only toward the last four new tasks (K4). ```js // library/acceptance.mjs — distributes acceptance scope across user tasks and separates the // returning feedback into defect / request / misunderstanding. Input: library/defects.mjs import { defects } from "./defects.mjs"; const ITEMS = 20, PER_ITEM = 4, TASKS = 12, SEED = 20260731, N = 60, MULTIPLIER = 6; const BUDGET = 600, CRITERION_MIN = 8; // TP19: acceptance budget, user minutes per criterion let ACCEPT_RATE = 0.3; // TP20: rate at which a request converts into work const REQUEST_MIN = 30, MISUNDERSTANDING_MIN = 20, NEW_WORK_MIN = 120; // TP21: cost per class const v = (x, n = 1) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); const makeRandom = (t) => { let x = t; return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; }; // The fourth lesson's criterion set, unchanged: 80 criteria, seed 6060, three types. const r = makeRandom(6060); const TYPE = ["existence", "numeric", "semantic"]; const CRITERIA = Array.from({ length: ITEMS * PER_ITEM }, (_, i) => { const p = r(); return { id: i, item: Math.floor(i / PER_ITEM), type: TYPE[p < 0.45 ? 0 : p < 0.8 ? 1 : 2] }; }); // Defects that reach the acceptance stage: the five classes the second lesson's nine // budget-fitting items miss, i.e. that lesson's 18 escaped defects. What the scripted set caught does not arrive here. const REACHED = new Set(["accessibility", "degraded-response", "recovery-gap", "off-scenario", "semantic-drift"]); const DEFECTS = defects(SEED, N).map((k, i) => ({ ...k, item: Math.floor(i / (N / ITEMS)) })) .filter((k) => REACHED.has(k.cls)); for (const k of DEFECTS.filter((k) => k.cls === "semantic-drift")) { const itemCriteria = CRITERIA.filter((o) => o.item === k.item); if (itemCriteria.some((o) => o.type === "semantic") === false) Object.assign(itemCriteria[Math.floor(r() * itemCriteria.length)], { type: "semantic" }); } // The state the fourth lesson's Y5 policy left behind: numeric criteria have an empty limit. for (const o of CRITERIA) o.complete = o.type !== "numeric"; // TP18: 20 work items are distributed across 12 user tasks (each task gets at least one item); // a task's usage share comes from the same seed. const r2 = makeRandom(4712); const SHARE = Array.from({ length: TASKS }, () => 1 + Math.floor(r2() * 9)); const SHARE_TOTAL = SHARE.reduce((a, x) => a + x, 0); const ITEM_TASK = Array.from({ length: ITEMS }, (_, i) => i % TASKS); for (let j = ITEMS - 1; j > 0; j -= 1) { const q = Math.floor(r2() * (j + 1)); [ITEM_TASK[j], ITEM_TASK[q]] = [ITEM_TASK[q], ITEM_TASK[j]]; } const taskOf = (o) => ITEM_TASK[o.item]; // Every defect sits under one criterion of its item; semantic drift under a semantic criterion. const r3 = makeRandom(9313); for (const k of DEFECTS) { const candidates = CRITERIA.filter((o) => o.item === k.item && (k.cls === "semantic-drift" ? o.type === "semantic" : true)); k.criterion = candidates[Math.floor(r3() * candidates.length)]; } const FLAWED = new Set(DEFECTS.map((k) => k.criterion.id)); // A task with m minutes: m/CRITERION_MIN criteria are observed. Feedback follows the criterion's // state: empty limit -> request (user states their own limit), flawed complete criterion -> defect, // flawless existence criterion -> misunderstanding (its measure is invisible to the user, who // compares it to their own expectation), flawless semantic criterion -> no feedback. function evaluate(distribution) { const findings = { defect: [], request: 0, misunderstanding: 0 }; let observed = 0, moment = 0, count = 0; for (let task = 0; task < TASKS; task += 1) { const list = CRITERIA.filter((o) => taskOf(o) === task); const seen = Math.min(list.length, Math.floor(distribution[task] / CRITERION_MIN)); observed += seen; list.slice(0, seen).forEach((o, i) => { const when = (i + 1) * CRITERION_MIN; if (o.complete === false) { findings.request += 1; moment += when; count += 1; } else if (FLAWED.has(o.id)) { const matches = DEFECTS.filter((k) => k.criterion.id === o.id); findings.defect.push(...matches); moment += when * matches.length; count += matches.length; } else if (o.type === "existence") { findings.misunderstanding += 1; moment += when; count += 1; } }); } const escaped = DEFECTS.filter((k) => findings.defect.includes(k) === false); // A found defect's fix is paid once, an escaped defect's fix is paid at the TP5 multiplier. const o = { observed, moment: moment / (count || 1), defect: findings.defect.length, request: findings.request, misunderstanding: findings.misunderstanding, escaped: escaped.length, escapedRisk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), defectMin: findings.defect.reduce((a, k) => a + k.fix, 0), escapedMin: MULTIPLIER * escaped.reduce((a, k) => a + k.fix, 0), requestMin: REQUEST_MIN * findings.request + ACCEPT_RATE * findings.request * NEW_WORK_MIN, misunderstandingMin: MISUNDERSTANDING_MIN * findings.misunderstanding }; return { ...o, total: o.defectMin + o.escapedMin + o.requestMin + o.misunderstandingMin }; } const equal = () => Array.from({ length: TASKS }, () => BUDGET / TASKS); const byShare = () => SHARE.map((p) => (BUDGET * p) / SHARE_TOTAL); const byGap = () => { const e = Array.from({ length: TASKS }, (_, task) => CRITERIA.filter((o) => taskOf(o) === task && o.complete === false).length); const t = e.reduce((a, x) => a + x, 0); return e.map((x) => (BUDGET * x) / t); }; const newOnly = () => Array.from({ length: TASKS }, (_, task) => (task >= TASKS - 4 ? BUDGET / 4 : 0)); const D = [["K1 equal distribution", equal()], ["K2 by usage share", byShare()], ["K3 to most-incomplete criteria", byGap()], ["K4 only four new tasks", newOnly()]]; console.log(`${CRITERIA.length} criteria (seed 6060) distributed across ${TASKS} user tasks (seed 4712)`); console.log("criteria per task: " + Array.from({ length: TASKS }, (_, task) => CRITERIA.filter((o) => taskOf(o) === task).length).join(" ") + "; usage share: " + SHARE.map((p) => v((100 * p) / SHARE_TOTAL, 0) + "%").join(" ")); console.log(`the fourth lesson's Y5 state: ${CRITERIA.filter((o) => o.complete === false).length} criteria have an empty limit; ` + `${FLAWED.size} criteria have a defect underneath`); console.log(`\nacceptance budget TP19 = ${BUDGET} user min, ${CRITERION_MIN} min per criterion = ${BUDGET / CRITERION_MIN} criteria`); console.log("distribution observed defect request misunderstanding defect share find moment"); for (const [name, d] of D) { const o = evaluate(d); const t = o.defect + o.request + o.misunderstanding; console.log(name.padEnd(32) + g(o.observed, 8) + g(o.defect, 8) + g(o.request, 9) + g(o.misunderstanding, 19) + g(t ? v(o.defect / t, 2) : "-", 14) + g(v(o.moment), 13)); } console.log("\nclass costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21)"); console.log("distribution defect min request min misunderstanding min escaped escaped risk escaped min total min"); for (const [name, d] of D) { const o = evaluate(d); console.log(name.padEnd(32) + g(o.defectMin, 12) + g(v(o.requestMin, 0), 13) + g(o.misunderstandingMin, 23) + g(o.escaped, 9) + g(o.escapedRisk, 14) + g(o.escapedMin, 13) + g(v(o.total, 0), 11)); } console.log("\nTP20's sensitivity: what share of requests convert into work"); console.log("accept rate " + D.map(([name]) => g(name.slice(0, 2), 9)).join("") + " cheapest"); for (const p of [0, 0.3, 0.6]) { ACCEPT_RATE = p; const t = D.map(([, d]) => evaluate(d).total); console.log(g((100 * p) + "%", 11) + t.map((n) => g(v(n, 0), 9)).join("") + g(D[t.indexOf(Math.min(...t))][0].slice(0, 2), 10)); } ``` ``` 80 criteria (seed 6060) distributed across 12 user tasks (seed 4712) criteria per task: 8 8 8 8 8 8 8 8 4 4 4 4; usage share: 4% 8% 10% 12% 3% 10% 10% 3% 12% 12% 12% 6% the fourth lesson's Y5 state: 29 criteria have an empty limit; 15 criteria have a defect underneath acceptance budget TP19 = 600 user min, 8 min per criterion = 75 criteria distribution observed defect request misunderstanding defect share find moment K1 equal distribution 64 7 21 30 0.12 25.2 K2 by usage share 54 8 21 22 0.16 26.0 K3 to most-incomplete criteria 60 6 23 24 0.11 28.7 K4 only four new tasks 16 6 4 8 0.33 19.6 class costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21) distribution defect min request min misunderstanding min escaped escaped risk escaped min total min K1 equal distribution 675 1386 600 11 88 3510 6171 K2 by usage share 720 1386 440 10 82 3240 5786 K3 to most-incomplete criteria 555 1518 480 12 102 4230 6783 K4 only four new tasks 570 264 160 12 89 4140 5134 TP20's sensitivity: what share of requests convert into work accept rate K1 K2 K3 K4 cheapest 0% 5415 5030 5955 4990 K4 30% 6171 5786 6783 5134 K4 60% 6927 6542 7611 5278 K4 ``` #### Feedback Volume Is Not a Quality Measure The **defect share** of the feedback the four distributions produce is between 0.11 and 0.33. That is, in the best case one report in three, in the worst case one in nine, is an actual defect; the rest is request and misunderstanding. K1 produces fifty-eight reports, and only seven of these are defects. Summarizing an acceptance round as "fifty-eight findings" does not tell the team it is talking about seven defects. Without classification this distinction is invisible: all three classes fall into the same list and get written the same way. Tying the distinction to the criterion takes it out of being a judgment call: feedback arriving at a criterion with an empty limit is a request, because there is no unmet promise — the user's limit is being stated there for the first time. #### Which Distribution Buys What K3 is the distribution that brings the most requests (23). This is not surprising: targeting incomplete criteria has users observe, by definition, criteria whose limit has not been written. Its escaped defect is also highest (12), and its total is the most expensive (6,783 minutes). **Targeting uncertainty finds requests, not defects.** K2 distributes by usage share, finds the most defects (8), and lets the fewest escape (10, escaped risk 82). A defect sitting in a task the user uses heavily gets found when tested. Against that, K2 is lowest in the number of criteria it manages to observe (54), because the bulk of the budget goes to a small number of tasks. K4 looks cheapest (5,134 minutes) and has the highest defect share (0.33). The reason is not an advantage: because it observes only sixteen criteria, it also produces no noise. Request and misunderstanding cost stays at 424 minutes, but it lets twelve defects escape. **The total cost column alone is misleading, because producing no feedback is cheap.** The two columns must be read together: K2 is cheap on escaped defect, K4 is cheap on processed feedback, and the two do not measure the same thing. The TP20 scan confirms this. Even if none of the requests turn into work (acceptance rate 0%), the ranking does not change; raised to 60%, K4 pulls further ahead, because it already has few requests to process. What actually determines an acceptance round's cost is not the number of defects found but the **number of processed reports.** #### The Return on the Decision Acceptance testing's red is not a single decision, it is three separate queues. A defect stops the release and goes to a fix. A request goes to the product side's decision, and 30% of it turns into work; the rest stays on record. A misunderstanding goes not to the code but to interface copy or documentation, and feeds back into the fourth lesson's criterion slots — a misunderstanding is the mark of a criterion whose measure is not visible to the user. The finding's average moment is between 19.6 and 28.7 minutes, but this number is deceptively small: the acceptance window opens **after** development is finished. The second lesson's scripted set reported a defect at an average of 18.2 minutes, the third lesson's exploratory session at 74.8 minutes; acceptance testing's finding, though, only surfaces once the 600-minute window opens. Along the same defect chain, the reporting moment grows, and its cost grows with it. #### Summary - Only defects the earlier gates did not see reach the acceptance stage: 18 defects from the second lesson's five unclosed classes. - Feedback is three classes, and the criterion's state determines the class: an empty-limit criterion produces a request, a flawed complete criterion produces a defect, a flawless criterion whose measure is invisible to the user produces a misunderstanding. - Defect share is between 0.11 and 0.33; summarizing an acceptance round by finding count does not tell the team how many defects were found. - The distribution targeting uncertainty produces the most requests and the most escaped defect (23 requests, 12 escaped, 6,783 minutes); the distribution by usage share finds the most defects (8) and escapes the fewest. - The distribution with the lowest total cost is the one that produces the fewest reports (K4, 5,134 minutes, 12 escaped); producing no feedback is cheap, so total cost is not read alone. - Acceptance testing's red goes to three separate queues: fix, product decision, and interface copy; a misunderstanding feeds back into the criterion slots. #### Next Step This topic has left four decisions behind. The plan wrote down scope, criteria, and responsibility; the risk order split the same budget by likelihood and impact; the exploratory session reached, through depth instead of width, the class the scripted set cannot see; the acceptance criterion and user acceptance testing carried the definition of done outside the team. What the four share is this: all of them are a distribution **on paper.** A plan is written but does not run; a risk order is a list, but a change can still be shipped without anyone looking at it; a charter opens only if someone decides to sit down; an acceptance criterion closes only if someone reads it. For this distribution to happen on its own, on every change, without depending on anyone remembering — a mechanism is needed, and that mechanism was never built. The Automation Infrastructure topic starts from exactly this point. ## Automation Infrastructure Placing tests into a pipeline and turning a result into a decision: the trade-off staged test gates strike between feedback time and escaped defects, the cost of environment variety, secrets and data safety in the test environment, making results readable for the team, interpreting escaped defects together with rework and cycle time metrics, and tracing recurring defects to their source. ### Testing in Continuous Integration Source: https://academia.sh/en/courses/testing-process/testing-in-continuous-integration Placing the tests left behind by four courses into a pipeline: the trade-off staging and gating strikes between feedback time and escaped defects, fail fast's effect on the run budget, the wait cost of nightly and release gates, and tying quarantine to the gate decision. The previous topic left a distribution behind: scope was written, a risk order was built, an exploratory session's charter was set, acceptance criteria were tied to a definition of done. What all of these share is this — the distribution is **on paper**. A plan is only an intention until it is wired into a mechanism that runs on its own with every change; otherwise who runs which test, and when, is left to human memory. This lesson builds that mechanism. The chain that takes a change and runs it through the tests in sequence is called a **pipeline**; each part of the pipeline is a **stage**, and the point where a stage's outcome forces a decision is a **gate**. The pipe from the Shell Programming course connects one process's output to another's input; the pipeline here is the ordered set of tests a change must pass — it carries a decision, not data. The question can be written as a single number: for a given test running at a given stage, where does the trade-off between feedback time and **escaped defect** (the same concept as the previous courses' escaped real defect) fall. #### Test Inventory Four courses wrote tests across thirty lessons, and each lesson's wrap-up table left behind that test's run-independent cost. Those numbers are collected here into nine **test teams**. Process count, network requests, data access, and manual review steps are read directly from the wrap-up tables: the integration team's 816 accesses is the sum of the rows for The Scope of Integration Testing (4 queries), Test Environment Management (6 steps), Test Data Management (772 queries), Database Testing (21 queries, 13 migration steps), and Mocking External Services (0); the security team's 20 manual steps is the sum of the manual-review counts from four security lessons. **TP6 — run cost is model minutes: standing up one process is 1 minute, a thousand network requests is 1 minute, a thousand data accesses is 1 minute, one manual review step is 15 minutes; the floor is 1 minute.** A manual step weighs fifteen times the others, because manual work cannot be placed at a gate — it waits its turn. **TP7 — occurrences per defect class in a hundred changes.** Classes the lower layers see are written more often, because every change touches code; capacity and recovery classes are rare. Two classes have an empty caught-by list: these come from the source lessons' "missed" column and are defects no team can see. ```js // inventory.mjs -- the team list, run cost, and defect set of tests written across four courses const roundUp = (n) => Math.ceil(n / 1000); // TP6: process 1 min, thousand requests 1 min, thousand accesses 1 min, manual step 15 min; floor 1 min. export const cost = (t) => Math.max(1, t.process + roundUp(t.request) + roundUp(t.access) + t.manual * 15); // process / request / access / manual counts read from the M21/K03 and K04 wrap-up tables. export const teams = [ { name: 'unit', process: 0, request: 0, access: 9, manual: 0 }, { name: 'contract', process: 6, request: 126, access: 0, manual: 0 }, { name: 'integration', process: 1, request: 0, access: 816, manual: 0 }, { name: 'end-to-end', process: 12, request: 24, access: 25511, manual: 0 }, { name: 'perf-run', process: 44, request: 78939, access: 0, manual: 0 }, { name: 'perf-sample', process: 0, request: 0, access: 536000, manual: 0 }, { name: 'security-auto', process: 9, request: 8, access: 183, manual: 0 }, { name: 'security-manual', process: 0, request: 0, access: 0, manual: 20 }, { name: 'resilience', process: 0, request: 639, access: 20, manual: 0 }, ]; // TP7: occurrences per defect class in a hundred changes. export const defects = [ { defectClass: 'boundary comparison', catches: ['unit'], count: 13 }, { defectClass: 'schema mismatch', catches: ['integration'], count: 8 }, { defectClass: 'card status not updating', catches: ['end-to-end'], count: 7 }, { defectClass: 'status code mapping', catches: ['end-to-end'], count: 6 }, { defectClass: 'breaking change', catches: ['unit', 'contract'], count: 6 }, { defectClass: 'request shape drift', catches: ['contract'], count: 5 }, { defectClass: 'accessibility criterion', catches: ['security-manual'], count: 5 }, { defectClass: 'field lost from contract', catches: ['contract'], count: 4 }, { defectClass: 'missing warning on rejected request', catches: ['end-to-end'], count: 4, quarantine: true }, { defectClass: 'latency regression', catches: ['perf-run'], count: 4 }, { defectClass: 'data loss in migration', catches: ['integration'], count: 3 }, { defectClass: 'concatenated query', catches: ['security-auto'], count: 3 }, { defectClass: 'semantic mismatch', catches: [], count: 3 }, { defectClass: 'check-then-act race', catches: ['end-to-end'], count: 2 }, { defectClass: 'distribution-dependent report defect', catches: ['integration'], count: 2 }, { defectClass: 'process-lifetime state', catches: [], count: 2 }, { defectClass: 'percentile read from few samples', catches: ['perf-sample'], count: 1 }, { defectClass: 'recovery gap', catches: ['resilience'], count: 1 }, { defectClass: 'capacity limit', catches: ['resilience'], count: 1 }, ]; ``` ```js // cost.mjs -- run cost per team and the total of the defect set import { teams, defects, cost } from './inventory.mjs'; const s = (x, n) => String(x).padStart(n); console.log(`${'team'.padEnd(20)}${s('process', 9)}${s('request', 9)}${s('access', 8)}${s('manual', 8)}${s('minutes', 9)}`); for (const t of teams) { console.log(`${t.name.padEnd(20)}${s(t.process, 9)}${s(t.request, 9)}${s(t.access, 8)}${s(t.manual, 8)}${s(cost(t), 9)}`); } const total = teams.reduce((a, t) => a + cost(t), 0); const defectTotal = defects.reduce((a, k) => a + k.count, 0); const unseen = defects.filter((k) => k.catches.length === 0); console.log(`\nteam ${teams.length}, full team ${total} minutes`); console.log(`defect class ${defects.length}, ${defectTotal} defects in a hundred changes, ` + `${100 - defectTotal} clean changes`); console.log(`classes no team sees ${unseen.length}, ` + `${unseen.reduce((a, k) => a + k.count, 0)} defects`); ``` ``` team process request access manual minutes unit 0 0 9 0 1 contract 6 126 0 0 7 integration 1 0 816 0 2 end-to-end 12 24 25511 0 39 perf-run 44 78939 0 0 123 perf-sample 0 0 536000 0 536 security-auto 9 8 183 0 11 security-manual 0 0 0 20 300 resilience 0 639 20 0 2 team 9, full team 1021 minutes defect class 19, 80 defects in a hundred changes, 20 clean changes classes no team sees 2, 5 defects ``` The full team is 1021 minutes, that is, seventeen hours. This number has already made the decision: running the full team on every change is not an option. The pyramid's arithmetic was done in The Test Pyramid lesson; here the pyramid is converted into time — the unit team is 1 minute, the end-to-end team 39, the performance team's sampling run alone is 536 minutes. #### Gate Arrangements An arrangement sets the order of stages and each stage's frequency: on every change, nightly, or per release. Frequency produces a wait. **TP8 — wait times: 240 minutes for the nightly gate, 1200 minutes for the release gate.** Changes spread evenly across the workday; the nightly run fires once at day's end, so the average wait is half a workday. Release is once a week, so the average wait is two and a half workdays. **TP9 — a hundred changes spread across twenty workdays: five changes a day, four releases.** This assumption sets the run count and directly affects the machine cost. **TP10 — a runner works 480 minutes a day.** The required runner count follows from this and is the arrangement's hardware cost. **Fail fast** is when a stage turning red keeps the following stages from running at all. The measurement below toggles this as an option. ```js // gate.mjs -- comparing seven gate arrangements over the same defect set import { teams, defects, cost } from './inventory.mjs'; const minute = Object.fromEntries(teams.map((t) => [t.name, cost(t)])); const WAIT = { change: 0, nightly: 240, release: 1200 }; // TP8 const RUNS = { change: 100, nightly: 20, release: 4 }; // TP9 const A = ['unit', 'contract']; const B = ['integration', 'end-to-end']; const C = ['perf-run', 'security-auto', 'resilience']; const D = ['perf-sample', 'security-manual']; const arrangements = [ { name: 'single gate', failFast: true, stages: [['all', 'change', [...A, ...B, ...C, ...D]]] }, { name: 'three stages', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'change', [...C, ...D]]] }, { name: 'three stages, no cut', failFast: false, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'change', [...C, ...D]]] }, { name: 'heavy stage nightly', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', [...C, ...D]]] }, { name: 'three frequencies', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', D]] }, { name: 'manual off pipeline', failFast: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', ['perf-sample']]] }, { name: 'three frequencies, quarantine', failFast: true, quarantine: true, stages: [['fast', 'change', A], ['mid', 'change', B], ['heavy', 'nightly', C], ['release', 'release', D]] }, ]; const measure = (d) => { const stage = d.stages.map(([name, frequency, tk]) => ({ name, frequency, tk, duration: tk.reduce((a, x) => a + minute[x], 0) })); const priorDuration = (i) => { const f = stage[i].frequency; const moreFrequent = stage.filter((x) => RUNS[x.frequency] > RUNS[f]).reduce((a, x) => a + x.duration, 0); const same = stage.slice(0, i + 1).filter((x) => x.frequency === f).reduce((a, x) => a + x.duration, 0); return moreFrequent + WAIT[f] + same; }; let missed = 0, weight = 0, totalFeedback = 0, worst = 0, cancelled = 0; for (const k of defects) { const visible = (t) => k.catches.includes(t) && (d.quarantine !== true || k.quarantine !== true); const i = stage.findIndex((x) => x.tk.some(visible)); if (i === -1) { missed += k.count; continue; } const feedback = priorDuration(i); weight += k.count; totalFeedback += feedback * k.count; worst = Math.max(worst, feedback); // fail fast: change stages after a red stage do not run if (d.failFast && stage[i].frequency === 'change') { cancelled += k.count * stage.slice(i + 1).filter((x) => x.frequency === 'change').reduce((a, x) => a + x.duration, 0); } } const machine = (stage.reduce((a, x) => a + x.duration * RUNS[x.frequency], 0) - cancelled) / 100; return { machine, runner: Math.ceil((machine * 5) / 480), feedback: totalFeedback / weight, worst, missed }; }; const s = (x, n) => String(x).padStart(n); console.log(`${'arrangement'.padEnd(32)}${s('min/change', 12)}${s('runner', 8)}${s('avg feedback', 14)}${s('worst', 9)}${s('missed/100', 12)}`); for (const d of arrangements) { const r = measure(d); console.log(`${d.name.padEnd(32)}${s(r.machine.toFixed(1), 12)}${s(r.runner, 8)}` + `${s(r.feedback.toFixed(1), 14)}${s(r.worst, 9)}${s(r.missed, 12)}`); } const duration = (g) => g.reduce((a, x) => a + minute[x], 0); console.log(`\nstage durations: fast ${duration(A)}, mid ${duration(B)}, nightly ${duration(C)}, release ${duration(D)} minutes`); console.log(`night window 480 minutes; heavy stage in full ${duration([...C, ...D])} minutes, ` + `${duration([...C, ...D]) <= 480 ? 'fits' : 'does not fit'}`); ``` ``` arrangement min/change runner avg feedback worst missed/100 single gate 1021.0 11 1021.0 1021 5 three stages 426.3 5 228.1 1021 5 three stages, no cut 1021.0 11 228.1 1021 5 heavy stage nightly 231.9 3 276.1 1261 5 three frequencies 98.2 2 252.6 2221 5 manual off pipeline 86.2 1 107.7 1921 10 three frequencies, quarantine 98.2 2 264.0 2221 9 stage durations: fast 8, mid 41, nightly 136, release 836 minutes night window 480 minutes; heavy stage in full 972 minutes, does not fit ``` #### Reading the Table The first two rows give what splitting into stages buys. In the single-gate arrangement, every defect surfaces after 1021 minutes, because the result only comes out once the whole team finishes. Once the same team is split into three stages, average feedback drops to 228.1 minutes — escaped defects do not change, the missed defect set is the same. Splitting alone is a free gain, with no loss. The third row isolates fail fast's share. The same arrangement asks for 1021 minutes and eleven runners per change with the cut off; 426.3 minutes and five runners with it on. Feedback time is identical in both rows: fail fast does not tell anyone anything sooner, it only cancels the 1013 minutes of work that would have run after a red change. The gain is fifty-eight percent and comes directly from defect frequency — in a defect-free repository, fail fast buys nothing. The fourth and fifth rows show frequency's cost. Moving the heavy stage to the nightly gate leaves 231.9 minutes per change, but the last row reports a problem: the heavy stage in full is 972 minutes and does not fit the 480-minute night window. A nightly gate has to finish before night ends. So the heavy stage splits in two — the performance run, the automated security scan, and the resilience tests go nightly (136 minutes, fits the window), performance sampling and the manual security review go to the release gate. The three-frequency arrangement drops to 98.2 minutes per change and two runners; in exchange, the worst feedback rises from 1021 to 2221 minutes. The sixth row is this course's rule in its plainest form. When the manual security review is dropped from the pipeline entirely, 86.2 minutes and a single runner per change suffice, and average feedback drops to 107.7 minutes — **and escaped defects rise from five to ten.** The average improving is not a gain, it is a measurement trap: the latest-caught class is no longer caught at all, so it has also dropped out of the average's denominator. Read on its own, feedback time always makes deleting a test look good. The last row ties quarantine to the gate decision. When the end-to-end test quarantined in the Flakiness Management lesson is dropped from the pipeline, run cost does not change (98.2 minutes is the same, because the team's duration is set by other tests) but the defect class it protected **escapes**: from five to nine. Quarantine looks like a maintenance decision; in the table, it is a gate decision. Across the first five rows, escaped defects stay fixed at five. These five defects belong to two classes that no arrangement catches — semantic mismatch and process-lifetime state, both from the source lessons' "missed" column. **No arrangement asks for zero escaped defects;** even the most expensive one misses five, because no test in the inventory was written to see that class. #### The Return on the Decision A gate's value is in what its red result does. The fast and mid stages **block merging**: if the result is red, the change does not enter the main branch, and its owner is still at that work. The nightly gate cannot block merging, because the change has already entered; a red result **opens a record**, and that record's owner is the set of changes that went into the nightly run — no single one of them is identified, and that is the 240-minute wait's second cost. The release gate **stops the publish**, and the decision's owner is not one person but the team approving the release. These three returns are the real criterion for stage placement: a test is placed at the earliest stage where the decision its result corresponds to can actually be made. The manual security review sitting at the release gate is not a shortcoming — it is manual work's nature: it waits its turn and cannot be placed at an automated gate. #### Summary - Four courses' thirty lessons were collected into nine test teams; the full team is 1021 model minutes, and every cost was read from the source lessons' wrap-up tables. - Splitting the team into three stages dropped average feedback from 1021 to 228.1 minutes and did not change escaped defects. - Fail fast, without changing feedback at all, dropped the machine cost per change from 1021 to 426.3 minutes and the runner count from eleven to five. - The heavy stage in full is 972 minutes and does not fit the 480-minute night window; split in two, the cost per change drops to 98.2 minutes and the worst feedback rises to 2221. - Dropping the manual review from the pipeline lowered average feedback to 107.7 minutes and raised escaped defects from five to ten; dropping the quarantined test moved escaped defects to nine. - No arrangement zeroed out escaped defects: two defect classes are not seen by any test in the inventory. #### Next Step Every row in this table rests on an assumption: a stage's duration is just the tests' own run cost. But every team besides the fast stage wants a test environment — the integration team a dependent service, the end-to-end team a live system, the performance team a production-like environment. These environments are not unlimited. When two changes arrive at once, one waits, and its wait is added to the stage's duration; adding environments lowers the wait but the cost grows linearly. The next lesson counts how many environments to keep at which stage, the wait a shared environment produces, and a reserved environment's cost. ### Management of Test Environments Source: https://academia.sh/en/courses/testing-process/management-of-test-environments Adding the environment dimension to the previous lesson's gate arrangement: the visibility limit an environment tier gives a team, a shared environment's queue time, a reserved environment's idle cost, and the escaped defects eight environment arrangements produce on the same defect set. Every row in the previous lesson's table rested on a silent assumption: a stage's duration is just the run cost of the tests running there. That amounts to counting the environment as free. But every team besides the fast stage wants a **test environment**, and an environment cuts cost from two places — **queueing**, because it exists in limited number, and **idle waiting**, because it stays reserved continuously. This lesson's question is not how an environment is set up; that mechanism was built and counted step by step in the Test Environment Management lesson (four setup steps, two teardown steps). The question here is distribution: how many environments, at which stage, of which tier. The resource distributed is **environment-minutes**; its two returns are queue time and **escaped defects**. #### Environment Tier and Fidelity An environment's distinguishing feature is not its price but its **fidelity**: which property of production it carries. Four tiers are distinguished. The `none` tier is not an environment at all, it is the runner itself; only teams that use nothing but their own processes belong here. The `ephemeral` tier is set up and torn down per run, lives for a single process, and fills with seeded data. The `shared` tier is a full system standing continuously: multiple services, persistent state, a real restart. The `production-like` tier adds production-scale data and a real resource limit on top of that. Fidelity sets a team's visibility limit. The end-to-end team cannot test card status in a single-process-lifetime environment, because the flow is spread across multiple services. So one more column is added to each row of the defect set: the minimum environment tier required for that class to be visible. The column's values were read from the source lessons' measurements — the distribution-dependent report defect is visible with seeded data, so `ephemeral`; the capacity limit only shows up under a real resource limit, so `production-like`. **TP11 — fidelity is ordered: `none` < `ephemeral` < `shared` < `production-like`.** A higher tier sees everything a lower tier sees. This is a simplification that reduces fidelity to one dimension. **TP12 — a reserved environment pays cost all day long: a shared environment costs 1440 model minutes a day, a production-like environment four times that.** Its cost is measured by its existence, not its use. The ephemeral environment is the opposite: it only pays for the minute it is used, adding a 3-minute setup and teardown per run. ```js // matrix.mjs -- environment tiers, stage list, and the defect set's tier column // Stage durations, defect classes and counts read from the previous lesson's inventory. export const LEVEL = { none: 0, ephemeral: 1, shared: 2, 'production-like': 3 }; // TP11 export const DAILY = { shared: 1440, 'production-like': 5760 }; // TP12 export const SETUP = 3; // an ephemeral environment's per-run setup + teardown cost // [name, frequency, duration, baseline, teams]; baseline is the per-stage feedback time // from the previous lesson's "three frequencies" row. export const stages = [ ['fast', 'change', 8, 8, ['unit', 'contract']], ['mid', 'change', 41, 49, ['integration', 'end-to-end']], ['nightly', 'nightly', 136, 425, ['perf-run', 'security-auto', 'resilience']], ['release', 'release', 836, 2221, ['perf-sample', 'security-manual']], ].map(([name, frequency, duration, baseline, teams]) => ({ name, frequency, duration, baseline, teams })); // [defectClass, catching team, count in a hundred changes, required minimum environment tier] export const defects = [ ['boundary comparison', 'unit', 13, 0], ['schema mismatch', 'integration', 8, 1], ['card status not updating', 'end-to-end', 7, 2], ['status code mapping', 'end-to-end', 6, 1], ['breaking change', 'contract', 6, 0], ['request shape drift', 'contract', 5, 0], ['accessibility criterion', 'security-manual', 5, 2], ['field lost from contract', 'contract', 4, 0], ['missing warning on rejected request', 'end-to-end', 4, 2], ['latency regression', 'perf-run', 4, 3], ['data loss in migration', 'integration', 3, 1], ['concatenated query', 'security-auto', 3, 0], ['semantic mismatch', null, 3, 0], ['check-then-act race', 'end-to-end', 2, 2], ['distribution-dependent report defect', 'integration', 2, 1], ['process-lifetime state', null, 2, 0], ['percentile read from few samples', 'perf-sample', 1, 3], ['recovery gap', 'resilience', 1, 2], ['capacity limit', 'resilience', 1, 3], ].map(([defectClass, team, count, tier]) => ({ defectClass, team, count, tier })); // An arrangement gives each pool an environment tier, a count, and the stages it serves. const E = (stages) => ({ tier: 'ephemeral', count: Infinity, stages }); const S = (count, stages) => ({ tier: 'shared', count, stages }); const P = (count, stages) => ({ tier: 'production-like', count, stages }); export const arrangements = [ { name: 'all ephemeral', pools: [E(['mid', 'nightly', 'release'])] }, { name: 'ephemeral mid', pools: [E(['mid']), S(1, ['nightly', 'release'])] }, { name: 'one shared', pools: [S(1, ['mid', 'nightly', 'release'])] }, { name: 'two shared', pools: [S(2, ['mid', 'nightly', 'release'])] }, { name: 'four shared', pools: [S(4, ['mid', 'nightly', 'release'])] }, { name: 'separate heavy environment', pools: [S(2, ['mid']), S(1, ['nightly', 'release'])] }, { name: 'production-like release', pools: [S(2, ['mid']), S(1, ['nightly']), P(1, ['release'])] }, { name: 'production-like heavy', pools: [S(2, ['mid']), P(1, ['nightly', 'release'])] }, ]; ``` #### Queue A reserved environment carries a single run at a time. When two changes arrive at once, one waits, and its wait is added to that change's feedback time. The queue's size depends not on average load but on how irregular the arrivals are; that is why arrivals are placed not at even intervals but with a seeded sequence. **TP13 — five changes a day are placed across the workday's 480 minutes with a seed; environments run without interruption.** The previous lesson's TP9 spread a hundred changes across twenty workdays; the addition here is where those five changes land within the day. The nightly run enters the queue at minute 480 of every day, the release run at the same minute of four release days — the release run competes for the same environment as the nightly run, at the same moment. ```js // queue.mjs -- a seeded arrival sequence and a k-environment first-come-first-served queue export const DAYS = 20, WORKDAY = 480, FULL_DAY = 1440, DAILY_CHANGES = 5; // TP13: linear congruential generator (from an earlier course); seed is visible, sequence is reproducible. export function makeGenerator(seed) { let state = seed; return () => { state = (state * 1103515245 + 12345) % 2147483648; return state / 2147483648; }; } export function arrivals(seed = 20260731) { const random = makeGenerator(seed); const list = []; for (let day = 0; day < DAYS; day += 1) { const minutes = []; for (let i = 0; i < DAILY_CHANGES; i += 1) minutes.push(Math.floor(random() * WORKDAY)); for (const m of minutes.sort((a, b) => a - b)) list.push(day * FULL_DAY + m); } return list; } // jobs: { t, duration, stage }. If count is Infinity the environment is always ready. export function measureQueue(jobs, count) { const sorted = [...jobs].sort((a, b) => a.t - b.t); const freeAt = count === Infinity ? null : new Array(count).fill(0); const wait = new Map(); let busy = 0; for (const job of sorted) { let start = job.t; if (freeAt !== null) { let least = 0; for (let i = 1; i < freeAt.length; i += 1) if (freeAt[i] < freeAt[least]) least = i; start = Math.max(job.t, freeAt[least]); freeAt[least] = start + job.duration; } const w = wait.get(job.stage) ?? []; w.push(start - job.t); wait.set(job.stage, w); busy += job.duration; } const summary = new Map(); for (const [stage, w] of wait) { summary.set(stage, { avg: w.reduce((a, x) => a + x, 0) / w.length, worst: Math.max(...w) }); } const occupancy = count === Infinity ? 0 : busy / (count * DAYS * FULL_DAY); return { summary, occupancy }; } ``` #### Eight Arrangements The measurement holds the previous lesson's three-frequency gate arrangement fixed and only changes the environment distribution. If a stage's environment tier is below the required fidelity, the defect escapes even if a catching team is assigned to it. ```js // measure.mjs -- eight environment arrangements over the same team list: queue, cost, escaped defects import { LEVEL, DAILY, SETUP, stages, defects, arrangements } from './matrix.mjs'; import { arrivals, measureQueue, DAYS, WORKDAY, FULL_DAY } from './queue.mjs'; const ARRIVALS = arrivals(); const RELEASE_DAYS = [4, 9, 14, 19]; // TP9: four releases in twenty workdays const jobList = (stage, extra) => { const duration = stage.duration + extra; if (stage.frequency === 'change') return ARRIVALS.map((t) => ({ t, duration, stage: stage.name })); const days = stage.frequency === 'release' ? RELEASE_DAYS : [...Array(DAYS).keys()]; return days.map((d) => ({ t: d * FULL_DAY + WORKDAY, duration, stage: stage.name })); }; export function measure(arrangement) { const tierOf = new Map(); // stage name -> environment tier for (const p of arrangement.pools) for (const s of p.stages) tierOf.set(s, p.tier); let cost = 0, occupancy = 0; const wait = new Map(); for (const p of arrangement.pools) { const extra = p.tier === 'ephemeral' ? SETUP : 0; const jobs = p.stages.flatMap((name) => jobList(stages.find((s) => s.name === name), extra)); const r = measureQueue(jobs, p.count); for (const [stage, o] of r.summary) wait.set(stage, o); if (p.count === Infinity) cost += jobs.reduce((a, x) => a + x.duration, 0); else { cost += p.count * DAILY[p.tier] * DAYS; occupancy = Math.max(occupancy, r.occupancy); } } let extra = 0, queue = 0; // accumulates in stage order const feedback = new Map(); for (const s of stages) { if (tierOf.get(s.name) === 'ephemeral') extra += SETUP; queue += wait.get(s.name)?.avg ?? 0; feedback.set(s.name, s.baseline + extra + queue); } let missed = 0, weight = 0, total = 0; for (const k of defects) { const s = stages.find((x) => x.teams.includes(k.team)); const level = s === undefined ? 0 : LEVEL[tierOf.get(s.name) ?? 'none']; if (s === undefined || level < k.tier) { missed += k.count; continue; } weight += k.count; total += feedback.get(s.name) * k.count; } return { envs: arrangement.pools.reduce((a, p) => a + (p.count === Infinity ? 0 : p.count), 0), cost: cost / 100, occupancy, wait, queue: wait.get('mid')?.avg ?? 0, worst: Math.max(0, ...[...wait.values()].map((o) => o.worst)), feedback: total / weight, missed, }; } const s = (x, n) => String(x).padStart(n); console.log(`${'arrangement'.padEnd(28)}${s('envs', 6)}${s('env-min/chg', 14)}${s('occupancy', 11)}` + `${s('mid queue', 13)}${s('worst', 9)}${s('avg feedback', 14)}${s('missed/100', 12)}`); for (const d of arrangements) { const r = measure(d); console.log(`${d.name.padEnd(28)}${s(r.envs, 6)}${s(r.cost.toFixed(0), 14)}` + `${s(r.envs === 0 ? '-' : `${(r.occupancy * 100).toFixed(0)}%`, 11)}` + `${s(r.queue.toFixed(1), 13)}${s(r.worst.toFixed(0), 9)}` + `${s(r.feedback.toFixed(1), 14)}${s(r.missed, 12)}`); } const oneShared = measure(arrangements.find((d) => d.name === 'one shared')); console.log(`\nqueue per stage in the one-shared environment (minutes):`); for (const [stage, o] of oneShared.wait) { console.log(` ${stage.padEnd(9)} average ${s(o.avg.toFixed(1), 6)}, worst ${s(o.worst, 5)}`); } const load = 100 * 41 + 20 * 136 + 4 * 836; console.log(`\narrival sequence seed 20260731; ${ARRIVALS.length} changes, ${DAYS} workdays, ` + `first arrival minute ${ARRIVALS[0] % FULL_DAY}, workday ${WORKDAY} minutes`); console.log(`one shared environment's load ${load} minutes, reserved duration ${DAYS * FULL_DAY} minutes`); const seen = defects.filter((k) => k.team !== null); const baseline = seen.reduce((a, k) => a + stages.find((x) => x.teams.includes(k.team)).baseline * k.count, 0) / seen.reduce((a, k) => a + k.count, 0); console.log(`average feedback with environment queue counted as zero: ${baseline.toFixed(1)} minutes`); ``` ``` arrangement envs env-min/chg occupancy mid queue worst avg feedback missed/100 all ephemeral 0 105 - 0.0 0 50.1 30 ephemeral mid 1 332 21% 0.0 136 262.9 24 one shared 1 288 35% 6.4 172 228.3 11 two shared 2 576 18% 0.0 38 212.5 11 four shared 4 1152 9% 0.0 0 211.6 11 separate heavy environment 3 864 21% 0.0 136 221.4 11 production-like release 4 2016 12% 0.0 0 240.3 10 production-like heavy 3 1728 21% 0.0 136 263.5 5 queue per stage in the one-shared environment (minutes): mid average 6.4, worst 50 nightly average 18.6, worst 79 release average 145.0, worst 172 arrival sequence seed 20260731; 100 changes, 20 workdays, first arrival minute 74, workday 480 minutes one shared environment's load 10164 minutes, reserved duration 28800 minutes average feedback with environment queue counted as zero: 252.6 minutes ``` #### Reading the Table The last line ties back to the previous lesson: with environment queue counted as zero, average feedback is 252.6 minutes — the same number as the previous lesson's three-frequency row. That number was hiding an environment assumption. In the one-shared-environment arrangement, occupancy is 35 percent; the environment is idle two-thirds of the time. Yet the queue is not zero, and where it builds up cannot be read from the average: the mid stage waits 6.4 minutes on average, 50 in the worst case; the nightly stage 18.6 and 79; the release stage 145.0 and 172. The queue is largest at the least frequently running stage: the release run competes for the same environment at the same minute as the nightly run and waits behind its 136-minute run. A second environment resolves this directly — the release wait drops from 145 to 7.5, the mid stage's to zero. In exchange, cost rises from 288 to 576 environment-minutes, and escaped defects do not change — they stay at eleven. **The queue is solved with money; fidelity is not.** Four environments take this one step further: the queue zeroes out at every stage, occupancy drops to 9 percent, cost rises to 1152 environment-minutes, and average feedback falls from 212.5 to only 211.6. The third and fourth environments buy 0.9 minutes for 576 environment-minutes; at 18 percent occupancy, replication buys nothing. The first two rows look the other way. The all-ephemeral arrangement holds no reserved environment at all, is the cheapest row at 105 environment-minutes per change, and raises escaped defects from five to thirty. Even when the ephemeral environment is kept only at the mid stage, escaped defects are twenty-four, because the end-to-end team runs at that stage and cannot see card status, the warning on a rejected request, or the check-then-act race in a single-process-lifetime environment. Its cost is 332 environment-minutes — more expensive than one shared environment, with more than double its escaped defects. An arrangement that loses on both axes at once shows up in the table but not in a narrative. The average feedback column repeats the previous lesson's trap: the all-ephemeral arrangement gives the table's best feedback at 50.1 minutes, because the late-caught classes are no longer caught at all and have dropped out of the average's denominator. The last three rows ask where fidelity should go. When the production-like environment is placed at the release gate, escaped defects drop from eleven to only ten; the only class recovered is the percentile read from few samples. When the same environment covers the nightly stage instead, escaped defects drop to five — latency regression and capacity limit are the teams running there — and with three environments instead of four, at 1728 environment-minutes instead of 2016. Fidelity matches the team running at a gate, not how late that gate is; putting the most expensive environment at the last gate invests the money in the wrong place. Escaped defects do not drop below five. Those five defects belong to two classes that no team in the previous lesson's inventory sees; even a production-like environment does not catch process-lifetime state, because no test was written to catch it. Environment fidelity widens a test's visibility limit; it does not write the test itself. #### The Return on the Decision An environment decision's return shows up in two places. The first is the **queue**: while a change waits for an environment, its owner moves on to other work, and by the time a red result arrives, the context is gone — this is why queue time is counted in the same column as run time. The second is the **environment's own red**: when an environment breaks, every test in that stage drops, and the dropped tests report an environment, not a defect. This result's owner is not the person who made the change but the team holding the environment; the decision is not to quarantine the test but to bring the environment back. Being unable to tell the two kinds of red apart is the shared environment's uncounted third cost. #### Summary - An environment splits into four tiers; the tier sets the highest defect class the team running at that stage can see, and it does not write the test itself. - With environment queue counted as zero, average feedback is 252.6 minutes: the previous lesson's number rested on a hidden environment assumption. - One shared environment produced a queue even at 35 percent occupancy; the queue is largest at the least frequently running stage (release 145.0 minutes, mid 6.4 minutes). - A second environment nearly zeroes the queue, raises cost from 288 to 576 environment-minutes, and does not change escaped defects; a fourth environment buys 0.9 minutes for 576 environment-minutes. - The ephemeral environment is the cheapest row (105 environment-minutes) and raises escaped defects from five to thirty; the arrangement that keeps the ephemeral environment at the mid stage is both more expensive and misses more defects. - Moving the production-like environment to the nightly stage instead of the release gate dropped escaped defects from ten to five and lowered cost from 2016 to 1728 environment-minutes. #### Next Step The production-like environment leaves one question open: what goes into that environment for it to resemble production. The data side was measured in the Test Data Management lesson. What was not counted is access credentials, identity, signing keys, and external-service tokens; all of these are secrets, and every secret handed to a test produces two things — a leak surface and a catchable defect class. The next lesson counts this distribution: which secrets enter the environment, how much the exposure surface shrinks when authorization scope is narrowed, and how many steps a time-limited privilege leaves manual. ### Secrets and Data Management Source: https://academia.sh/en/courses/testing-process/secrets-and-data-management The distribution of secrets entering the test environment: leak points measured in run artifacts, least privilege's effect on exposure surface, the manual step count a time-limited privilege leaves behind, and the escaped defects seven authorization arrangements produce on the same defect set. The previous lesson tied the production-like environment to a cost but did not ask what goes into that environment. The data side was measured in the Test Data Management lesson: masking looks at the content of personal fields, and what property it preserves was counted there. Access credentials, identity, signing keys, and external-service tokens were left outside that measurement. These are **secrets**, and they cannot be masked — a masked version of a secret does not work, so it either goes in or it does not. The resource this lesson distributes is **authorization**: how many secrets, with how much scope, for how long, enter the test environment. The distribution has two returns — every secret that does not enter misses a defect class, every secret that does enter opens a **leak surface**. #### Leak Surface Leak surface is not an estimate, it is a countable quantity: how many of the artifacts a run leaves behind carry the secret itself. The run below actually calls the notification boundary and triggers the defect measured in the previous course — the unknown recipient. Two token forms are tried: an indefinitely valid persistent token and a token issued only for that run. ```js // leak.mjs -- how many places a token appears in run artifacts, and its value after the run import { createServer } from 'node:http'; const valid = new Set(['B-persistent-9f21', 'B-run-4a7c']); const members = new Set(['U-17']); const server = createServer((req, res) => { const token = req.headers['x-token'] ?? ''; const recipient = new URL(req.url, 'http://y').searchParams.get('recipient'); const [status, body] = !valid.has(token) ? [403, `{"error":"invalid token"}`] : members.has(recipient) ? [202, '{"status":"queued"}'] // the diagnostic body echoes the request's context back: this is the third leak point : [404, `{"error":"unknown recipient: ${recipient}","token":"${token}"}`]; res.writeHead(status, { 'content-type': 'application/json' }); res.end(body); }); server.listen(8941, '127.0.0.1'); const BASE = 'http://127.0.0.1:8941/notify'; const call = (token, recipient) => fetch(`${BASE}?recipient=${recipient}`, { headers: { 'x-token': token } }); // The run triggers the defect measured in the Test Environment Management lesson: unknown recipient. async function run(token) { const artifact = [`STEP loan-flow x-token=${token}`]; // 1: run record const res = await call(token, 'U-99'); artifact.push(`ERROR ${await res.text()}`); // 2: the other side's diagnostic body artifact.push(`RETRY 1/1 x-token=${token} outcome=${res.status}`); // 3: retry summary return { artifact, leaked: artifact.filter((s) => s.includes(token)).length }; } const persistent = await run('B-persistent-9f21'); const runScoped = await run('B-run-4a7c'); valid.delete('B-run-4a7c'); // validity window closed const after = async (token) => (await call(token, 'U-17')).status; const persistentAfter = await after('B-persistent-9f21'); const runAfter = await after('B-run-4a7c'); server.close(); console.log(`persistent token: ${persistent.artifact.length} artifacts, value present in ${persistent.leaked}`); console.log(`run-scoped token: ${runScoped.artifact.length} artifacts, value present in ${runScoped.leaked}`); console.log(`leak point count is the same in both forms: ${persistent.leaked === runScoped.leaked}`); console.log(`post-run request, persistent token: ${persistentAfter}`); console.log(`post-run request, run-scoped token: ${runAfter}`); console.log(`artifact not in the team's code: ${runScoped.artifact[1]}`); ``` ``` persistent token: 3 artifacts, value present in 3 run-scoped token: 3 artifacts, value present in 3 leak point count is the same in both forms: true post-run request, persistent token: 202 post-run request, run-scoped token: 403 artifact not in the team's code: ERROR {"error":"unknown recipient: U-99","token":"B-run-4a7c"} ``` Three results are worth reading. First, a single failed call leaves three artifacts, and all three carry the token itself. Second, only two of these three points are in the team's own code; the third is the other side's diagnostic body, and no scrubbing rule the team writes can anticipate that body. Third, the two token forms leak **exactly** the same number of points — the leak point count depends on the shape of the error path, not the nature of the token. The only thing that changes is the value after the run: the persistent token still gets 202, the run-scoped token 403. The lever is not concealment, it is **scope** and **duration**. #### Secret Catalog The test environment wants five secrets, and each opens specific defect classes. The numbers come from the previous lesson's defect set: all thirty-two defects caught at the mid stage are tied to a secret, four at the nightly stage, one at the release stage. **TP14 — the access counts under production scope are an assumption; the ones under narrow scope are a measurement.** Narrow scope is the subset the secret actually touches, and it was read from the test set the Test Data Management lesson measured: 772 rows, 200 members. The numbers on the production side are assumptions about the same system's production scale and carry an order of magnitude. **TP15 — in the time-limited arrangement, the validity window is 44 minutes, and the setup's manual steps are counted per secret.** The window is the sum of the mid stage's 41-minute run and the ephemeral environment's 3-minute setup; it cannot be shorter than the run, because authorization has to end when the run ends. The manual steps are three items: opening one account per secret, writing one scope list under narrow scope, setting up one mechanism for automated issuance. These are paid once per period, while a per-run manual step is paid again on every run. ```js // secrets.mjs -- the secret catalog and authorization arrangements // Per-stage feedback time and the caught defect count not tied to a secret; both read from // the previous lesson's "production-like heavy" row. export const stages = { fast: { feedback: 8, free: 28 }, mid: { feedback: 49, free: 0 }, nightly: { feedback: 425, free: 5 }, release: { feedback: 2357, free: 5 }, }; export const MISSED_FLOOR = 5; // two classes no test sees export const MANUAL = 15; // TP6: one manual step is 15 minutes export const RUNS = { mid: 100, nightly: 20, release: 4 }; // [name, stage, defect count opened, is write required, prod access, narrow-scope access] export const secrets = [ ['catalog ID', 'mid', 13, true, 240000, 772], ['notification token', 'mid', 10, false, 18000, 200], ['payment key', 'mid', 9, true, 9000, 41], ['performance environment ID', 'nightly', 4, true, 240000, 772], ['metrics store ID', 'release', 1, true, 4800, 24], ].map(([name, stage, count, write, prod, narrow]) => ({ name, stage, count, write, prod, narrow })); // An arrangement gives, for each secret: does it enter the environment, how many records can it // access, can it write; validity window (minutes), setup manual steps, per-run manual steps. export const arrangements = [ { name: 'production identity', enters: 1, access: (g) => g.prod, writes: 1, window: Infinity, setup: 0, perRunManual: 0 }, { name: 'separate account', enters: 1, access: () => 772, writes: 1, window: Infinity, setup: 5, perRunManual: 0 }, { name: 'read only', enters: 1, access: () => 772, writes: 0, window: Infinity, setup: 5, perRunManual: 0 }, { name: 'narrow scope', enters: 1, access: (g) => g.narrow, writes: 1, window: Infinity, setup: 10, perRunManual: 0 }, { name: 'narrow scope, time-limited', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 10, perRunManual: 1 }, { name: 'time-limited, automated issuance', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 11, perRunManual: 0 }, { name: 'mock', enters: 0, access: () => 0, writes: 0, window: 0, setup: 0, perRunManual: 0 }, ]; ``` #### Seven Authorization Arrangements The **least privilege** principle is tied to a number here: **exposure surface** is the total number of records the leaked secrets can reach. If a secret does not get the write permission it requires, the defect classes it opens escape; the same happens if it does not enter the environment at all. ```js // scope.mjs -- seven authorization arrangements: leak surface, exposure, manual steps, escaped defects import { stages, secrets, arrangements, MISSED_FLOOR, MANUAL, RUNS } from './secrets.mjs'; const LEAK_POINTS = 3; // measured value: the error path produces three artifacts function measure(d) { const entering = d.enters === 1 ? secrets : []; const exposed = entering.reduce((a, g) => a + d.access(g), 0); const manualRuns = d.perRunManual * Object.values(RUNS).reduce((a, x) => a + x, 0); let extra = 0; const feedback = {}; for (const [name, s] of Object.entries(stages)) { if (name !== 'fast') extra += d.perRunManual * MANUAL; feedback[name] = s.feedback + extra; } let missed = MISSED_FLOOR, weight = 0, total = 0; for (const [name, s] of Object.entries(stages)) { weight += s.free; total += feedback[name] * s.free; } for (const g of secrets) { if (d.enters === 0 || (g.write && d.writes === 0)) { missed += g.count; continue; } weight += g.count; total += feedback[g.stage] * g.count; } return { entering: entering.length, leak: entering.length * LEAK_POINTS, exposed, window: d.window, manual: d.setup + manualRuns, feedback: total / weight, missed, }; } const s = (x, n) => String(x).padStart(n); console.log(`${'arrangement'.padEnd(35)}${s('enters', 8)}${s('leak', 7)}${s('exposure', 12)}` + `${s('window', 11)}${s('manual steps', 14)}${s('avg feedback', 14)}${s('missed/100', 12)}`); for (const d of arrangements) { const r = measure(d); console.log(`${d.name.padEnd(35)}${s(r.entering, 8)}${s(r.leak, 7)}${s(r.exposed, 12)}` + `${s(r.window === Infinity ? 'unlimited' : r.window, 11)}${s(r.manual, 14)}` + `${s(r.feedback.toFixed(1), 14)}${s(r.missed, 12)}`); } const prod = secrets.reduce((a, g) => a + g.prod, 0); const narrow = secrets.reduce((a, g) => a + g.narrow, 0); const writers = secrets.filter((g) => g.write); console.log(`\nsecrets ${secrets.length}, requiring write ${writers.length}, ` + `defects they open ${writers.reduce((a, g) => a + g.count, 0)}`); console.log(`prod scope ${prod} records, narrow scope ${narrow} records, ` + `narrowing ${(prod / narrow).toFixed(0)}x`); console.log(`run count ${Object.values(RUNS).reduce((a, x) => a + x, 0)}; ` + `one manual step per run costs ${Object.values(RUNS).reduce((a, x) => a + x, 0) * MANUAL} minutes`); ``` ``` arrangement enters leak exposure window manual steps avg feedback missed/100 production identity 5 15 511800 unlimited 0 263.5 5 separate account 5 15 3860 unlimited 5 263.5 5 read only 5 15 3860 unlimited 5 304.7 32 narrow scope 5 15 1809 unlimited 10 263.5 5 narrow scope, time-limited 5 15 1809 44 134 277.1 5 time-limited, automated issuance 5 15 1809 44 11 263.5 5 mock 0 0 0 0 0 371.9 42 secrets 5, requiring write 4, defects they open 27 prod scope 511800 records, narrow scope 1809 records, narrowing 283x run count 124; one manual step per run costs 1860 minutes ``` #### Reading the Table The first and fourth rows are this lesson's central measurement. Between production identity and narrow scope, escaped defects (five) and average feedback (263.5 minutes) are identical to the digit; the only place they diverge is exposure surface — 511,800 records against 1,809, two hundred eighty-three times over. **Least privilege's cost in escaped defects is zero.** What is paid is the ten manual setup steps, and these are paid once per period. What makes narrowing scope hard is not a measurable trade-off but the deferral of a one-time task. This number has a condition, and it ties to the second half of this lesson's title. The narrow scope's 1,809 records assumes the data in the environment is the **masked** test set. If a copy of production was loaded into the environment, then no matter how narrow the secret's scope is, exposure surface equals the environment's own size: 240,000 records sit there, and every token that can reach that environment reaches them. The two decisions cannot be made separately for this reason — unmasked data defeats least privilege, and narrow authorization does not stand in for masking either. The third row is narrowing done on the wrong axis. Read-only authority sounds safe, but four of the five secrets require write: the catalog ID changes schema in the migration test, the payment key updates card status, the metrics IDs write results. The twenty-seven defects these four open escape, and escaped defects rise from five to thirty-two. Exposure surface does not shrink at all — it stays at 3,860, because the value still enters the environment and still leaks from three points. This row loses on both axes at once. Here the average feedback column shows the **opposite** of the trap in the previous two lessons. Under read-only, the average rises from 263.5 to 304.7, and to 371.9 under mock. The reason is that the missed defects are the cheap ones caught: once the mid stage's forty-nine- minute defects drop out of the denominator, only the expensive ones caught remain, and the average rises. The same column can lie by improving in one lesson and worsening in another; in both, what exposes the lie is the escaped defects column. The last row is the only arrangement that truly zeroes the leak surface. No secret enters, the leak point count is zero, exposure surface is zero — and escaped defects are forty-two. Mocking the external service was built as a test form in the Mocking External Services lesson; here it shows up as an **authorization decision**, and its cost is thirty-seven defects. An arrangement that wants to zero out the leak surface ends up closing off the test itself. The fifth and sixth rows isolate time-limited authorization's cost. The window dropping from unlimited to 44 minutes changes neither the leak point count (fifteen) nor exposure surface (1,809); the only thing that changes is how useful the leaked value is after the run — that is what the 403 in the run above is. Issuing the token manually adds one step to each of the hundred twenty-four runs: the manual step count rises from ten to a hundred thirty-four, average feedback from 263.5 to 277.1 minutes. Automating issuance gives the same window with eleven manual steps and without changing feedback at all. The difference is a single setup step, and it removes 1,860 minutes of manual work. #### The Return on the Decision When a secret is seen to have leaked, the decision made depends on the window column in the table. If the window is unlimited, the decision is a revoke-and-reissue operation; every system using that value has to be found, and the work falls to the team holding the production side. If the window is 44 minutes, the decision is limited to opening a record, because the value is already invalid by the time the record opens. Same event, same leak point, two different decisions. The second return is in the pipeline itself. An authorization rejection (403) and the defect itself (404) look like the same red to a stage. Unless they are separated, an expired token looks like a **flaky test** and gets quarantined; and a quarantined test, as the previous lesson measured, drops a defect class from the pipeline. A classification that tells an authorization error apart from a defect error is, for this reason, not a security measure but a gate decision. #### Summary - A secret cannot be masked: it either enters the test environment or it does not; every value that enters produces a leak surface, every value that does not produces an escaped defect class. - A single failed call left three artifacts, and all three carried the token; one of the three was not in the team's code, but in the other side's diagnostic body. - The leak point count does not depend on the token's form; the persistent and time-limited tokens leaked the same number of points, and after the run one got 202, the other 403. - Narrowing scope dropped exposure surface from 511,800 records to 1,809 (283 times over) and did not change escaped defects at all; its cost is the ten manual setup steps. - Pulling back to read-only left four of the five secrets non-functional: escaped defects rose from five to thirty-two, and exposure surface did not shrink at all. - Issuing time-limited authorization manually added one step to a hundred twenty-four runs (134 manual steps, 277.1-minute feedback); automating issuance gave the same window with 11 steps and 263.5 minutes. #### Next Step Three lessons counted three distributions: run minutes, environment-minutes, and authorization. The numbers all three produce sit in the same place — inside a run. No one reads the three artifacts above; no one watches the gate arrangement's average feedback, the environment queue, and the exposure surface on a screen. For a measurement to change a decision, it first has to be readable, and readability has a budget too: how many lines, to whom, how often. The next lesson takes on this distribution — it counts which of the hundreds of lines a run leaves behind reach the team, and which decision the ones that do not reach anyone silently cancel. ### Reporting Source: https://academia.sh/en/courses/testing-process/reporting Making a run's result readable for the team: measuring the same nightly run across four reporting arrangements in lines read, steps opened, and noise items, comparing time-to-decision against the attention budget, an unresolved root turning into an escaped defect, and scanning the flaky section's culling threshold together with its source. The previous lessons put the pipeline, the environment, and secrets in order. The pipeline now produces a result on every change and every night. This lesson's question is not the result itself, but what it tells whom: a run result counts as not having run at all until it turns into a decision, because no one acts on it otherwise. This is not a writing task, it is a distribution task. The resource distributed is **attention**: how many lines get read before a result turns into a decision, how many steps it takes to find the cause, how many items produce no action at all. The same run result is presented in four arrangements, and these three numbers are measured. #### A Run's Raw Result The result to be measured comes from a nightly run: the day's five changes are taken together, and seven teams run, all but the two teams at the release gate. The red items inside the result come from four sources, and the report's real job is telling them apart — **root** (the first test a defect drops), **derived** (the other tests the same root drops), **environment** (a failure that belongs to the mechanism, not the code under test), and **flaky** (a test that drops without the code changing). ```js // run.mjs -- a nightly run's raw result: test states, red items, roots let s = 20260731; // seed const random = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); // TP16: seven teams' test counts; flaky tests are spread at step intervals and drop at rate, // history is the probability of dropping from a REAL defect. The release gate is not in this run. export const teams = [ { name: 'unit', tests: 240, step: 0, rate: 0.00, history: 0.010 }, { name: 'contract', tests: 42, step: 21, rate: 0.03, history: 0.008 }, { name: 'integration', tests: 96, step: 12, rate: 0.06, history: 0.015 }, { name: 'end-to-end', tests: 34, step: 3, rate: 0.15, history: 0.012 }, { name: 'perf-run', tests: 12, step: 6, rate: 0.10, history: 0.010 }, { name: 'security-auto', tests: 58, step: 19, rate: 0.04, history: 0.006 }, { name: 'resilience', tests: 14, step: 7, rate: 0.05, history: 0.010 }, ]; // M21/K03: the decision looks at the MEASURED rate over 200 runs; the measured rate is the sum of two sources. export const tests = []; for (const t of teams) { for (let i = 0; i < t.tests; i += 1) { const flaky = t.step > 0 && i % t.step === 0 ? t.rate : 0; let dF = 0, dR = 0; for (let k = 0; k < 200; k += 1) { if (random() < flaky) dF += 1; if (random() < t.history) dR += 1; } tests.push({ id: `${t.name}/${i + 1}`, team: t.name, flaky, dF, dR, measured: (dF + dR) / 200, red: false, root: null }); } } // TP17: this run's red roots. Class name and catching team are from automation-infrastructure/01's // defect set; affected test count comes from the shared surface the defect touches. export const roots = [ { name: 'schema mismatch', team: 'integration', affected: 23, real: true }, { name: 'boundary comparison', team: 'unit', affected: 3, real: true }, { name: 'status code mapping', team: 'end-to-end', affected: 5, real: true }, { name: 'dependent service did not start', team: 'resilience', affected: 14, real: false }, ]; for (const t of tests) if (t.flaky > 0 && random() < t.flaky) { t.red = true; t.root = 'flaky'; } for (const k of roots) { for (const t of tests.filter((x) => x.team === k.team).slice(0, k.affected)) { t.red = true; t.root = k.name; } } export const reds = tests.filter((t) => t.red); export const realRoots = roots.filter((k) => k.real); ``` This run has three real defects and one environment failure; the number of items that drop is far larger, because the single defect touching a shared schema drops all twenty-three tests that use that schema at once. The report does not know this; all it sees are red items. #### Four Reporting Arrangements The harness below models reading as a cost (TP18). A line is three seconds, a step is three minutes; a step is any work that does not end with reading the line — opening a detail, rerunning, finding the owner. In an ungrouped report, the reader only merges the same signature by the third item. **TP19 — the attention one person gives to a run result is 30 minutes, that is, one sixteenth of a workday;** its sensitivity is scanned in the second half of the table. A root that is not resolved carries over to the next nightly run and adds 240 minutes per TP8; a root that is never resolved escapes. ```js // report.mjs -- time-to-decision for the same run result across four reporting arrangements import { teams, tests, reds, roots, realRoots } from './run.mjs'; // TP18: a line is 3 sec, a step is 3 min, the raw log has 12 detail lines per red item, the // same root merges by the third item. TP19: attention given to one run result is 30 min. const SEC = 3, STEP = 3, DETAIL = 12, RECOGNIZE = 3, BUDGET = 30; const WAIT = 240; // automation-infrastructure/01 TP8: nightly wait // An arrangement's reading flow: a sequence of {lines, step, root resolved} events. function flow(arrangement, remaining) { const live = (t) => t.red && (t.root === 'flaky' || remaining.includes(t.root)); const events = [], seen = {}; const raw = (detailed) => { for (const tm of teams) { if (detailed) events.push({ lines: 2, step: 0 }); for (const t of tests.filter((x) => x.team === tm.name)) { if (live(t) === false) { if (detailed) events.push({ lines: 1, step: 0 }); continue; } const n = (seen[t.root] = (seen[t.root] ?? 0) + 1); const group = tests.filter((x) => live(x) && x.root === t.root).length; events.push({ lines: detailed ? 1 + DETAIL : 1, step: n <= RECOGNIZE ? 1 : 0, done: n === Math.min(RECOGNIZE, group) ? t.root : null }); } } }; if (arrangement === 'raw log') { events.push({ lines: 1, step: 1 }); raw(true); } if (arrangement === 'flat list') { events.push({ lines: 1, step: 0 }); raw(false); events.push({ lines: 1, step: 0 }); } if (arrangement === 'grouped') { for (const k of roots.filter((x) => x.real && remaining.includes(x.name))) events.push({ lines: 3, step: 1, done: k.name }); // header + root test + "n more items" events.push({ lines: 1, step: 1 }, { lines: 1, step: 1 }, { lines: 2, step: 0 }); // environment, flaky, summary } if (arrangement === 'summary only') { events.push({ lines: 3, step: 2 }); raw(true); } return events; } const read = (arrangement, remaining, budget) => { let minutes = 0, lines = 0, step = 0; const done = []; for (const e of flow(arrangement, remaining)) { const cost = minutes + (e.lines * SEC) / 60 + e.step * STEP; if (cost > budget) break; minutes = cost; lines += e.lines; step += e.step; if (e.done && remaining.includes(e.done)) done.push(e.done); } return { minutes, lines, step, done }; }; // An unresolved root carries over to the next run (+240 min); a root never resolved escapes. function measure(arrangement, budget = BUDGET) { let remaining = realRoots.map((k) => k.name), cycle = 0, first = null; const feedback = []; while (remaining.length > 0 && cycle < 20) { cycle += 1; const r = read(arrangement, remaining, budget); first ??= r; if (r.done.length === 0) break; for (const _ of r.done) feedback.push(WAIT * cycle + r.minutes); remaining = remaining.filter((a) => r.done.includes(a) === false); } const avg = feedback.length ? feedback.reduce((a, c) => a + c, 0) / feedback.length : NaN; return { ...first, firstCycle: first.done.length, cycle, missed: remaining.length, feedback: avg }; } const ARRANGEMENTS = ['raw log', 'flat list', 'grouped', 'summary only']; const noise = { 'raw log': tests.length - 3, 'flat list': reds.length - 3, grouped: 2, 'summary only': tests.length - 3 }; const p = (x, n) => String(x).padStart(n); const total = flow('raw log', realRoots.map((k) => k.name)).reduce((a, e) => a + e.lines, 0); console.log(`${tests.length} tests, ${reds.length} red items, ${roots.length} roots ` + `(${realRoots.length} real defects + 1 environment), ${reds.filter((t) => t.root === 'flaky').length} independent flaky drops`); console.log(`raw log ${total} lines = ${((total * SEC) / 60).toFixed(1)} min, ` + `${((total * SEC) / 60 / BUDGET).toFixed(1)}x the budget\n`); console.log(`${'arrangement'.padEnd(14)}${p('lines', 7)}${p('steps', 7)}${p('minutes', 9)}${p('noise', 9)}` + `${p('1st cycle', 11)}${p('cycles', 8)}${p('missed root', 13)}${p('avg feedback', 14)}`); for (const d of ARRANGEMENTS) { const r = measure(d); console.log(`${d.padEnd(14)}${p(r.lines, 7)}${p(r.step, 7)}${p(r.minutes.toFixed(1), 9)}` + `${p(noise[d], 9)}${p(`${r.firstCycle}/3`, 11)}${p(r.cycle, 8)}${p(r.missed, 13)}` + `${p(Number.isNaN(r.feedback) ? '-' : r.feedback.toFixed(1), 14)}`); } console.log(`\nTP19 sensitivity (roots resolved in cycle 1 / roots missed)`); console.log(`${'budget'.padEnd(8)}${ARRANGEMENTS.map((d) => p(d, 14)).join('')}`); for (const b of [15, 30, 60]) console.log(`${`${b} min`.padEnd(8)}${ARRANGEMENTS.map((d) => { const r = measure(d, b); return p(`${r.firstCycle} / ${r.missed}`, 14); }).join('')}`); ``` ``` 496 tests, 46 red items, 4 roots (3 real defects + 1 environment), 1 independent flaky drops raw log 895 lines = 44.8 min, 1.5x the budget arrangement lines steps minutes noise 1st cycle cycles missed root avg feedback raw log 325 4 28.3 493 1/3 3 1 389.0 flat list 32 9 28.6 43 3/3 1 0 268.6 grouped 13 5 15.7 2 3/3 1 0 255.7 summary only 299 5 30.0 493 1/3 2 2 270.0 TP19 sensitivity (roots resolved in cycle 1 / roots missed) budget raw log flat list grouped summary only 15 min 1 / 2 1 / 0 3 / 0 0 / 3 30 min 1 / 1 3 / 0 3 / 0 1 / 2 60 min 2 / 0 3 / 0 3 / 0 2 / 0 ``` The raw log is 895 lines; reading all of it is one and a half times the budget. The reader exhausts the budget at 325 lines and resolves only one root that night. Even read three nights running, the third root still does not show up: the end-to-end team sits at the end of the log, behind the detail lines of forty-six red items. **A defect being present in the report is not the same as it having a place in the report.** The flat red list and the grouped report give the same decision: all three roots close on the first night, nothing escapes. The difference is in cost. The flat list has 32 lines to read, the grouped report 13; but the flat list opens 9 steps, the grouped report 5. This is where the numbers pull apart: 32 lines is 1.6 minutes, 9 steps is 27 minutes. **A report's cost is not the reading cost, it is the opening cost** — and opening cost only drops with grouping, not with shortening. The flat list consumes ninety-five percent of the budget; the grouped report, half. This share's weight shows up in the sensitivity table. At a fifteen-minute budget, the grouped report still surfaces all three roots, the flat list only one. There is no symmetry in the other direction either: raising the budget to sixty minutes gives the raw log two roots in the first cycle, while the grouped report does three in fifteen. **Raising attention does not do what grouping does,** because most of the attention spent goes not to the decision itself but to recognizing the same root again and again. The noise column gives the source of the difference: 493 items in the raw log resolve into no decision, 43 in the flat list, 2 in the grouped report — the environment and flaky sections. In the summary-only arrangement, the report is three lines and says nothing; the reader goes back to the raw log, and two roots escape. **A short report is not a readable report.** #### The Flaky Section's Threshold Part of the grouped report's gain comes from putting flaky items into a separate section. This is a threshold decision: **a test whose measured drop rate crosses the threshold counts as flaky and is pulled out of the main section.** The threshold's source is the Flakiness Management lesson; there the same number was scanned as a quarantine threshold and dropped the test from the pipeline. The decision here is lighter — the test keeps running — but its cost is measured the same way. ```js // threshold.mjs -- the flaky section's culling threshold: noise cut against real red hidden import { tests, reds, realRoots } from './run.mjs'; const p = (x, n) => String(x).padStart(n); const total = (f) => tests.reduce((a, t) => a + f(t), 0); console.log(`${'threshold'.padStart(9)}${p('separated', 11)}${p('to section', 12)}` + `${p('hidden items', 14)}${p('hidden roots', 14)}` + `${p('noise cut', 11)}${p('real hidden', 13)}`); for (const e of [0.005, 0.01, 0.015, 0.025, 0.05, 0.1]) { const separated = new Set(tests.filter((t) => t.measured >= e).map((t) => t.id)); const dropped = reds.filter((t) => separated.has(t.id)); const items = dropped.filter((t) => realRoots.some((k) => k.name === t.root)).length; const hiddenRoots = realRoots.filter((k) => reds.filter((t) => t.root === k.name).every((t) => separated.has(t.id))).length; console.log(`${`${(e * 100).toFixed(1)}%`.padStart(9)}${p(separated.size, 11)}${p(dropped.length, 12)}` + `${p(items, 14)}${p(hiddenRoots, 14)}` + `${p(total((t) => (separated.has(t.id) ? t.dF : 0)), 11)}` + `${p(total((t) => (separated.has(t.id) ? t.dR : 0)), 13)}`); } console.log(`\ncolumns: the first four are over THIS run's ${reds.length} red items, ` + `the last two are over the 200-run history`); console.log(`of the ${total((t) => t.dF + t.dR)} historical drops, ` + `${total((t) => t.dF)} were from flakiness, ${total((t) => t.dR)} from a real defect; ` + `the report cannot tell them apart, it only sees the combined rate`); ``` ``` threshold separated to section hidden items hidden roots noise cut real hidden 0.5% 431 45 31 3 541 1013 1.0% 296 35 23 0 541 878 1.5% 187 26 18 0 541 660 2.5% 61 14 9 0 541 241 5.0% 25 7 4 0 508 59 10.0% 13 3 2 0 364 26 columns: the first four are over THIS run's 46 red items, the last two are over the 200-run history of the 1554 historical drops, 541 were from flakiness, 1013 from a real defect; the report cannot tell them apart, it only sees the combined rate ``` The first row shows how far the threshold can go. At 0.5 percent, 431 of 496 tests count as flaky, 45 of 46 red items drop to the second section, and all three real roots are hidden at once. The report looks spotless and says nothing: a report that zeroes out the noise may have zeroed out the signal too. The reason is written in the last two columns. Of the 1554 historical drops, 541 come from flakiness, 1013 from a real defect; the report cannot tell the two apart, it only sees the combined rate. This is why a low threshold hides the tests that have surfaced the most defects. The 2.5 percent threshold is the lowest point where it can still tell the two apart: 61 tests separate out, 14 items drop to the second section, nine of them from a real defect, but **no root is hidden entirely** — every root keeps at least one item in the main section. In exchange, all 541 historical flaky drops are cut. Raising the threshold beyond this only loses: the noise cut falls to 508 at 5 percent, to 364 at 10 percent — the noise flows back into the main section. #### The Return on the Decision Every section of a report corresponds to an action; a section with no corresponding action is cut. The root section **opens a record**, and its owner is the owner of the change that touched that defect; since a nightly run has no single owner, this section has to show the root, counting items is not enough. The environment section is the pipeline operator's business and says nothing about the code under test. The flaky section stops no one; it is a maintenance list and feeds the quarantine decision. Not showing derived items is also a decision, not a concealment: twenty-three tests dropped from a single defect, so a single record opens. Shown separately, the record count would not change, but the attention spent would multiply by twenty-three. #### Summary - The same run result was presented in four arrangements: 496 tests, 46 red items, 3 real roots, 1 environment failure, 1 independent flaky drop. - The grouped report resolved all three roots in 13 lines, 5 steps, and 15.7 minutes; the flat list gave the same decision in 32 lines, 9 steps, and 28.6 minutes — the difference comes from the steps opened (9 steps is 27 minutes, 32 lines is 1.6 minutes). - In the raw log, one root was never resolved on any night and escaped; in the summary-only arrangement, two roots escaped. Average feedback time is 255.7 minutes in the grouped report, 389.0 in the raw log. - Raising the attention budget did not stand in for grouping: at 60 minutes, the raw log surfaced two roots; the grouped report surfaced three in 15. - When the flaky section's threshold was lowered to 0.5 percent, all three roots were hidden; at 2.5 percent, all flaky drops were cut while no root was hidden. #### Next Step Every measure here belongs to a single run and gives a single night's decision. No matter how well a report shows the root in its own run, it cannot answer this question: is this team testing better over time, or are the same defects arriving under different names. Schema mismatch showing up tonight is an event; showing up every week is a process defect, and a single run's report makes the two look the same. The next lesson moves the measure to the period level: escaped defects, rework, and cycle time are computed from the same data, and each is asked what decision it changes. ### Quality Metrics Source: https://academia.sh/en/courses/testing-process/quality-metrics Computing escaped defects, rework, and cycle time from the same event set: measuring a single pipeline-cheapening decision across two periods, scanning six metrics' thresholds together with their sources, counting which metric changes a decision, and the attention cost of a metric that changes none. The previous lesson made a single run's result readable and resolved that run's root into a decision. One question a report cannot answer remained: is this team testing better over time. Schema mismatch showing up one night is an event; showing up every week is a process defect, and a single run's report makes the two look the same. This lesson moves the measure to the period level. What is measured here is not the system, it is the team. The running system's response time, error rate, and saturation are the Performance Anti-Patterns and Monitoring course's subject; the metrics here measure the testing itself. The resource distributed is again **attention**, and its quantity is the metric count: every metric wants to be collected and read once per period, and in exchange is expected to produce a decision. If it does not, it is noise. #### Three Metrics, One Data Set **Defect leakage rate** is the ratio of defects that reach production without being caught at any pipeline stage to total defects; the escaped defects in its numerator are the same concept as M21/K03's escaped real defects. **Rework** is work redone because of a defect, and it is measured in hours; this is a different concept from the **failback** the Resilience and Reliability course describes — the return to the primary copy after a takeover. **Cycle time** is the time from a change being submitted to it entering the main branch; the time up to release is **lead time**, and this lesson does not measure it. What the three share is this: all of them come out of a single event set — which defect was caught at which stage. Stage order, feedback minutes, and the defect set are taken from the Testing in Continuous Integration lesson's three-frequency arrangement. ```js // period.mjs -- five metrics for two periods over the same defect set // TP9 (automation-infrastructure/01): 100 changes, 20 workdays. Class names and per-hundred- // change counts are from the same lesson's TP7 defect set; stage is the place of the team // catching that class in the three-frequency arrangement. Total 80 defects, 20 clean changes. export const CLASSES = [ ['boundary comparison', 13, 'fast'], ['breaking change', 6, 'fast'], ['request shape drift', 5, 'fast'], ['field lost from contract', 4, 'fast'], ['schema mismatch', 8, 'mid'], ['card status not updating', 7, 'mid'], ['status code mapping', 6, 'mid'], ['missing warning on rejected request', 4, 'mid'], ['data loss in migration', 3, 'mid'], ['check-then-act race', 2, 'mid'], ['distribution-dependent report defect', 2, 'mid'], ['latency regression', 4, 'nightly'], ['concatenated query', 3, 'nightly'], ['recovery gap', 1, 'nightly'], ['capacity limit', 1, 'nightly'], ['accessibility criterion', 5, 'release'], ['percentile read from few samples', 1, 'release'], ['semantic mismatch', 3, 'escapes'], ['process-lifetime state', 2, 'escapes'], ]; // Feedback minutes read from automation-infrastructure/01's three-frequency arrangement. // TP20: rework hours by stage -- a late-caught defect grows from lost context, retesting, and // incident handling in production. TP21: an escaped defect is reported in production after one // release cycle (2400 min); a change's baseline cycle time is 480 min. export const FEEDBACK = { fast: 8, mid: 49, nightly: 425, release: 2221, escapes: 2400 }; export const REWORK = { fast: 0.25, mid: 1, nightly: 3, release: 6, escapes: 16 }; export const BASELINE = 480, DAY = 480, CHANGES = 100; export const CAPACITY = 800; // TP22: 5 people x 20 workdays x 8 hours // Period 2's only difference is automation-infrastructure/01's "manual off pipeline" // arrangement: the manual security review leaves the release gate, and accessibility criterion is no longer caught. export const period = (second) => CLASSES.map(([name, n, a]) => [name, n, second && name === 'accessibility criterion' ? 'escapes' : a]); export function measure(k) { const defectCount = k.reduce((a, [, n]) => a + n, 0); const missed = k.filter(([, , a]) => a === 'escapes').reduce((a, [, n]) => a + n, 0); const rework = k.reduce((a, [, n, x]) => a + n * REWORK[x], 0); const feedback = k.reduce((a, [, n, x]) => a + n * FEEDBACK[x], 0) / defectCount; // Cycle time: clean and escaped changes converge at the baseline; a caught defect waits on // its stage's feedback and its rework. const d = []; for (let i = 0; i < CHANGES - defectCount; i += 1) d.push(BASELINE); for (const [, n, x] of k) for (let i = 0; i < n; i += 1) d.push(x === 'escapes' ? BASELINE : BASELINE + FEEDBACK[x] + REWORK[x] * 60); d.sort((a, b) => a - b); const percentile = (q) => d[Math.ceil((q / 100) * d.length) - 1] / DAY; return { defectCount, missed, leak: missed / defectCount, rework, reworkShare: rework / CAPACITY, feedback, median: percentile(50), p90: percentile(90), caught: defectCount - missed }; } ``` Cycle time's calculation turns on one detail that decides the whole outcome: **an escaped defect does not lengthen cycle time.** No one sees it, so the change merges on time; its cost is paid in production, in a different column. This line is why the table further down shows cycle time moving in the wrong direction. #### Two Periods, One Decision There is exactly one difference between the two periods compared. The first period uses the three-frequency arrangement. In the second period, the team decides to cheapen the pipeline and removes the manual security review from the release gate — the Testing in Continuous Integration lesson measured that this arrangement drops from 98.2 to 86.2 minutes per change. The same lesson also measured that this decision raises escaped defects from five to ten; the question here is whether the metrics can see that. ```js // decision.mjs -- five metrics' value in two periods, and which one changed a decision import { CLASSES, REWORK, CAPACITY, CHANGES, period, measure } from './period.mjs'; const PERIODS = [measure(period(false)), measure(period(true))]; const p = (x, n) => String(x).padStart(n); const b = (x, n) => x.toFixed(n); // Threshold and its source. A metric with no threshold has no decision tied to it either. const METRICS = [ { name: 'leakage rate', get: (x) => x.leak * 100, format: (v) => `${b(v, 2)}%`, threshold: 8, decision: 'add a test to inventory' }, { name: 'rework share', get: (x) => x.reworkShare * 100, format: (v) => `${b(v, 2)}%`, threshold: 25, decision: 'move the class to an earlier stage' }, { name: 'cycle time p90', get: (x) => x.p90, format: (v) => `${b(v, 2)} days`, threshold: 3.5, decision: 'change stage placement' }, { name: 'cycle time median', get: (x) => x.median, format: (v) => `${b(v, 2)} days`, threshold: null, decision: 'none' }, { name: 'avg feedback', get: (x) => x.feedback, format: (v) => `${b(v, 1)} min`, threshold: 480, decision: 'change gate frequency' }, { name: 'caught defects', get: (x) => x.caught, format: (v) => String(v), threshold: null, decision: 'none' }, ]; console.log(`${CHANGES} changes, ${CLASSES.length} defect classes, ${PERIODS[0].defectCount} defects, ` + `${CHANGES - PERIODS[0].defectCount} clean changes; team capacity ${CAPACITY} hours`); console.log(`\n${'metric'.padEnd(20)}${p('period 1', 11)}${p('period 2', 11)}${p('threshold', 11)}` + `${p('P1 over', 9)}${p('P2 over', 9)}${p('changed', 9)}${' linked decision'}`); let changing = 0; for (const g of METRICS) { const [v1, v2] = [g.get(PERIODS[0]), g.get(PERIODS[1])]; const [c1, c2] = [g.threshold !== null && v1 >= g.threshold, g.threshold !== null && v2 >= g.threshold]; const changed = g.threshold !== null && c1 !== c2; if (changed) changing += 1; console.log(`${g.name.padEnd(20)}${p(g.format(v1), 11)}${p(g.format(v2), 11)}` + `${p(g.threshold === null ? '-' : g.format(g.threshold), 11)}${p(g.threshold === null ? '-' : (c1 ? 'yes' : 'no'), 9)}` + `${p(g.threshold === null ? '-' : (c2 ? 'yes' : 'no'), 9)}${p(changed ? 'YES' : 'no', 9)}` + ` ${g.decision}`); } // TP23: a metric's per-period collection and review cost is 30 minutes. console.log(`\n${METRICS.length} metrics x 30 min = ${METRICS.length * 30} min/period review; ` + `changing a decision ${changing}, noise ${(METRICS.length - changing) * 30} min`); // The decision also asks "which class": rework hours broken down by class. console.log(`\ntop three classes eating rework hours ` + `(total ${b(PERIODS[0].rework, 0)} -> ${b(PERIODS[1].rework, 0)} hours)`); console.log(`${' '.repeat(3)}${p('period 1', 32)}${p('period 2', 32)}`); const rank = (second) => period(second).map(([name, n, a]) => [name, n * REWORK[a]]) .sort((x, z) => z[1] - x[1]).slice(0, 3); const [r1, r2] = [rank(false), rank(true)]; for (let i = 0; i < 3; i += 1) console.log(`${`${i + 1}.`.padEnd(3)}${p(`${r1[i][0]} ${b(r1[i][1], 0)} hrs`, 32)}` + `${p(`${r2[i][0]} ${b(r2[i][1], 0)} hrs`, 32)}`); ``` ``` 100 changes, 19 defect classes, 80 defects, 20 clean changes; team capacity 800 hours metric period 1 period 2 threshold P1 over P2 over changed linked decision leakage rate 6.25% 12.50% 8.00% no yes YES add a test to inventory rework share 22.75% 29.00% 25.00% no yes YES move the class to an earlier stage cycle time p90 2.26 days 1.23 days 3.50 days no no no change stage placement cycle time median 1.05 days 1.05 days - - - no none avg feedback 386.8 min 398.0 min 480.0 min no no no change gate frequency caught defects 75 70 - - - no none 6 metrics x 30 min = 180 min/period review; changing a decision 2, noise 120 min top three classes eating rework hours (total 182 -> 232 hours) period 1 period 2 1. semantic mismatch 48 hrs accessibility criterion 80 hrs 2. process-lifetime state 32 hrs semantic mismatch 48 hrs 3. accessibility criterion 30 hrs process-lifetime state 32 hrs ``` #### Thresholds and Their Sources A metric's value alone does not carry a decision; what turns it into one is a threshold, and a threshold has to have a source. - **Leakage rate, 8 percent.** Its source is the inventory's blind spot: the Testing in Continuous Integration lesson counted two defect classes that no test team sees, five of eighty defects. That is leakage rate's floor — 6.25 percent. The threshold cannot be set below that floor, because an unreachable threshold produces no decision in any period, it only leaves a permanent alarm. - **Rework share, 25 percent.** Its source is team capacity: a quarter of 800 hours is 200 hours, and going above that means a quarter of a period going to fixes instead of new work. - **Cycle time p90, 3.5 workdays.** Its source is the release gate itself: that gate's average wait is 1200 minutes, that is, 2.5 workdays. If the threshold were set below that, every change caught at the release gate would cross it, and the metric would report the gate arrangement's own outcome as if it were a defect. - **Average feedback, 480 minutes.** One workday: the criterion that a defect reaches its owner within the same day. - **Cycle time median and caught defect count have no threshold**, because they have no decision tied to them. #### The Metric That Changed a Decision Two of the six metrics changed a decision. Leakage rate crossed its threshold, rising from 6.25 percent to 12.50 percent; rework share crossed its threshold too, rising from 22.75 percent to 29.00 percent. The remaining four either never crossed their threshold in either period or had none. The two metrics moved in the same direction but do not show the same thing. Leakage rate counts defects, rework share weighs them in hours, and their rankings differ. In the first period, the class eating the most rework hours is semantic mismatch: 48 hours, but only three defects. Three defects out of eighty never rise to the top of any count-based list; weighed in hours, it ranks first. In the second period, accessibility criterion jumps from 30 hours to 80 and takes first place — all five of its defects are now found in production, and fixing them takes sixteen hours instead of six. The decision, for this reason, comes out not as "do something" but as "do this." The remaining four metrics are this lesson's real finding. **Cycle time p90 dropped from 2.26 days to 1.23** — that is, this metric improved while quality got worse. The reason is written in a single line of code: an escaped defect does not lengthen cycle time. The five changes that used to wait at the release gate no longer wait there, because there is no longer a test there to hold them. The median, meanwhile, did not move at all, staying at 1.05 days. **Deleting a test always improves cycle time.** Average feedback time only worsened by about three percent, from 386.8 minutes to 398.0 — while leakage doubled. The reason is how close the two numbers are: the release gate used to report a defect in 2221 minutes, an escaped defect in production is reported in 2400. **The average cannot tell catching apart from missing.** Caught defect count also dropped from 75 to 70; read on its own, this row looks like "fewer defects," but what it says is fewer catches. The attention arithmetic gives the price of this: six metrics ask for 180 minutes of collection and review per period, and 120 minutes of that changes no decision. **A metric that changes no decision is noise** — and its cost is not just that 120 minutes. In the same period, cycle time and caught defect count look like they improved together; a team looking at just those two could read the decision that doubled leakage as a success. #### The Return on the Decision Leakage rate's decision changes the inventory: a new test is written for the blind-spot class, and that test's run cost is added to the next period's pipeline time. This worsens cycle time. **Metrics are not independent of each other**; while one is improved, another pays for it, and this is why no single metric is targeted alone. Rework share's decision moves a defect class to an earlier stage, and its owner is whoever built the gate arrangement. Cycle time's decision changes stage placement. A metric with no decision tied to it has no owner; it gets read out as a number in the period meeting, no one acts on it, and its cost is paid again every period. #### Summary - Leakage rate, rework share, and cycle time were computed from the same defect set; rework here is a separate concept from the failback in the Resilience and Reliability course — the return to the primary copy after a takeover. - A single pipeline-cheapening decision raised leakage rate from 6.25 percent to 12.50 percent and rework share from 22.75 percent to 29.00 percent; both crossed their threshold and produced a decision. - The same decision dropped cycle time p90 from 2.26 days to 1.23, because an escaped defect does not lengthen cycle time; the median did not change at all, and caught defects dropped from 75 to 70. - Average feedback time only worsened by three percent: because the time to notice in production (2400 min) is close to the release gate's wait (2221 min), the average conceals the miss. - Two of the six metrics changed a decision; the remaining four ate 120 minutes of attention per period, and two pointed the wrong way. **A metric that changes no decision is noise.** #### Next Step Metrics say one thing and do not say another. What they say: schema mismatch came up eight times this period, boundary comparison thirteen times, and most rework hours pile up in a few classes. What they do not say: why. All eight schema mismatches were fixed one by one, each one paying its own rework hours again; if there is a single shared cause under those eight, no metric shows it, because metrics count, they do not explain. A class recurring is not a defect, it is a symptom, and going from symptom to cause needs a method. The next lesson builds that method. ### Root Cause Analysis Source: https://academia.sh/en/courses/testing-process/root-cause-analysis Tracing a recurring defect class to its source: building the causal chain as a data structure, comparing the intervention that fixes the symptom with the one that fixes the cause on the same defect set, coverage growing while applicability drops with depth in the chain, and the review threshold's source. Metrics count a class coming up again and again but do not say why. If eight schema mismatches were fixed one by one, whether there is a single shared cause under those eight shows up in no metric. A class recurring is not a defect, it is a symptom. **Root cause analysis** is the method that goes from symptom to cause. Its tool is the **causal chain**: a sequence of links that starts from a defect and moves forward by asking "what caused this" at every step — known by the common name **five whys**. The chain is stopped somewhere, and an action is written there. The action that closes the symptom is called **corrective action**, the action that cuts the class's recurrence is called **preventive action**. This lesson's question is **where to stop** in the chain; the resource distributed is review hours. #### The Chain Is a Data Structure The chain's first two links are class-specific: why this defect occurred, why that cause existed. From the third on, links are shared — several classes feed from the same process gap. Node count drops with depth, class-per-node rises. In exchange, a deep link falls outside the team's control: what needs to change is not in the code, it is in the organization. **TP24 (assumption)** — reviewing one link takes 25 minutes, the review budget across the horizon is 400 minutes, and a review lands with a delay of one release per depth. The budget's source is the 120 minutes of per-period attention in the Quality Metrics lesson. **TP25 (assumption)** — preventive action's cost at link k is 60, 240, 720, and 2,400 minutes; a deeper link concerns more parties. **TP26 (assumption)** — the action's applicability share drops with depth: 0.90 / 0.70 / 0.40 / 0.00. The fifth link is outside the team. **TP27 (assumption)** — the root cause threshold is 3 recurrences. The threshold's source is a measurement: the per-class recurrence average is 60/14 = 4.29, and threshold 3 also covers one step below that. ```js // library/defects.mjs -- the first lesson's defect set, exactly (seed 20260731, 60 defects, // 14 classes). This lesson does not use the inventory's cost columns, only the defect set. const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, defectClass = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { defectClass = s; break; } } list.push({ no: i, defectClass, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` #### Five Interventions, One Defect Set The model runs twenty releases. When a class recurs as many times as the threshold, a review opens for its k-th link; once it lands, every class that node feeds has its recurrence cut by the applicability share. I1 opens no review at all. ```js // library/chain.mjs -- builds the causal chain as a data structure and compares the // intervention that fixes the symptom with the one that fixes the cause, on the same defect set. Input: library/defects.mjs import { defects } from "./defects.mjs"; const SEED = 20260731, N = 60, RELEASES = 20, PER_RELEASE = N / RELEASES, MULTIPLIER = 6; // TP5, TP6 const REVIEW = 25, BUDGET = 400; // TP24: review min per link, horizon's review budget const COST = [0, 0, 60, 240, 720, 2400]; // TP25: preventive action cost at link k const APPLICABILITY = [0, 0, 0.9, 0.7, 0.4, 0]; // TP26: the action's applicability share const v = (x, n = 2) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); // Causal chain: the first two links are class-specific, the third onward is shared across // classes. Nodes shrink with depth, because the same cause feeds more classes. const L3 = {}, L4 = {}; for (const [cause, sub] of [ ["schema-change-not-reviewed", ["schema-mismatch", "migration-data-loss", "contract-field"]], ["boundary-value-not-documented", ["rule-boundary", "interface-state"]], ["baseline-image-not-updated", ["visual-deviation"]], ["scale-assumption-not-documented", ["data-growth", "degraded-response", "recovery-gap"]], ["security-rule-missing-from-review", ["sql-concatenation", "known-vulnerability", "accessibility"]], ["acceptance-criterion-missing", ["off-scenario", "semantic-drift"]]]) for (const s of sub) L3[s] = cause; for (const [cause, sub] of [ ["no-check-by-change-type", ["schema-change-not-reviewed", "boundary-value-not-documented", "baseline-image-not-updated"]], ["assumption-not-documented", ["scale-assumption-not-documented", "security-rule-missing-from-review", "acceptance-criterion-missing"]]]) for (const s of sub) L4[s] = cause; const linkOf = (s, k) => (k <= 2 ? `${s}#${k}` : k === 3 ? L3[s] : k === 4 ? L4[L3[s]] : "process-owner-not-assigned"); // The five classes the second lesson's nine budget-fitting items miss escape to production. const ESCAPES = new Set(["accessibility", "degraded-response", "recovery-gap", "off-scenario", "semantic-drift"]); const DEFECTS = defects(SEED, N); const CLASSES = [...new Set(DEFECTS.map((k) => k.defectClass))]; // Horizon: RELEASES releases, PER_RELEASE defects per release. When a class recurs threshold // times, a review opens for its k-th link; a review takes one release per depth (TP24), and // once it lands, every class the node feeds has its recurrence cut by APPLICABILITY[k]. function horizon(depth, threshold) { const counter = new Map(), prevented = new Map(), open = new Set(), pending = []; let recur = 0, corrective = 0, preventive = 0, review = 0, missed = 0, when = 0, late = 0; for (let s = 1; s <= RELEASES; s += 1) { for (const item of pending) if (item.s === s) prevented.set(item.link, item.share); for (const k of DEFECTS.slice((s - 1) * PER_RELEASE, s * PER_RELEASE)) { const link = depth >= 2 ? linkOf(k.defectClass, depth) : null; const occurrence = 1 - (link ? prevented.get(link) || 0 : 0); if (occurrence <= 0) continue; // A review that has opened but has not yet landed still counts recurrence as delayed. if (link && open.has(link) && prevented.has(link) === false) late += occurrence; recur += occurrence; corrective += occurrence * k.fix; if (ESCAPES.has(k.defectClass)) missed += occurrence; const c = (counter.get(k.defectClass) || 0) + occurrence; counter.set(k.defectClass, c); if (depth >= 2 && c >= threshold && open.has(link) === false) { open.add(link); review += REVIEW * depth; preventive += COST[depth]; pending.push({ link, s: s + depth - 1, share: APPLICABILITY[depth] }); when += s + depth - 1; } } } const total = corrective + review + preventive + MULTIPLIER * missed * 60; // TP5: escaped defect's multiplier return { opened: open.size, review, preventive, corrective, recur, missed, late, when: open.size ? when / open.size : 0, total, fits: review <= BUDGET }; } console.log(`defect set seed ${SEED}, ${N} defects, ${CLASSES.length} classes, ${RELEASES} releases`); const counts = CLASSES.map((s) => DEFECTS.filter((k) => k.defectClass === s).length); console.log(`recurrence per class: minimum ${Math.min(...counts)}, maximum ${Math.max(...counts)}, average ${v(N / CLASSES.length)}`); console.log("\nchain structure -- nodes shrink with depth, applicability drops"); console.log(" link node class per node applicability prevent min expected prevented"); for (const k of [1, 2, 3, 4, 5]) { const d = new Set(CLASSES.map((s) => linkOf(s, k))); const coverage = CLASSES.length / d.size; console.log(g(k, 5) + g(d.size, 7) + g(v(coverage), 20) + g(k >= 2 ? v(APPLICABILITY[k]) : "-", 18) + g(k >= 2 ? COST[k] : "-", 13) + g(k >= 2 ? v(coverage * APPLICABILITY[k]) : "-", 24)); } const NAMES = ["I1 fix the symptom", "I2 stop at link 2", "I3 stop at link 3", "I4 stop at link 4", "I5 stop at link 5"]; console.log(`\nfive interventions on the same defect set (threshold TP27 = 3, review budget TP24 = ${BUDGET} min)`); console.log("intervention opened review budget preventive corrective recur delayed missed in effect total min"); for (let k = 1; k <= 5; k += 1) { const o = horizon(k, 3); console.log(NAMES[k - 1].padEnd(24) + g(o.opened, 8) + g(o.review, 10) + g(o.fits ? "fits" : "over", 8) + g(o.preventive, 12) + g(v(o.corrective, 0), 12) + g(v(o.recur), 10) + g(v(o.late), 11) + g(v(o.missed), 8) + g(o.opened ? v(o.when, 1) : "-", 11) + g(v(o.total, 0), 11)); } console.log("\nthreshold TP27's sensitivity -- stopping at link 3"); console.log(" threshold opened review min budget recur missed total min"); for (const e of [2, 3, 5]) { const o = horizon(3, e); console.log(g(e, 11) + g(o.opened, 8) + g(o.review, 13) + g(o.fits ? "fits" : "over", 8) + g(v(o.recur), 10) + g(v(o.missed), 8) + g(v(o.total, 0), 11)); } ``` ``` defect set seed 20260731, 60 defects, 14 classes, 20 releases recurrence per class: minimum 1, maximum 7, average 4.29 chain structure -- nodes shrink with depth, applicability drops link node class per node applicability prevent min expected prevented 1 14 1.00 - - - 2 14 1.00 0.90 60 0.90 3 6 2.33 0.70 240 1.63 4 2 7.00 0.40 720 2.80 5 1 14.00 0.00 2400 0.00 five interventions on the same defect set (threshold TP27 = 3, review budget TP24 = 400 min) intervention opened review budget preventive corrective recur delayed missed in effect total min I1 fix the symptom 0 0 fits 0 4125 60.00 0.00 18.00 - 10605 I2 stop at link 2 12 600 over 720 2870 42.00 1.00 14.40 12.5 9374 I3 stop at link 3 5 375 fits 1200 2813 40.40 8.00 15.20 10.8 9860 I4 stop at link 4 2 200 fits 1440 3075 44.40 7.00 15.60 8.0 10331 I5 stop at link 5 1 125 fits 2400 4125 60.00 11.00 18.00 9.0 13130 threshold TP27's sensitivity -- stopping at link 3 threshold opened review min budget recur missed total min 2 6 450 over 29.90 10.30 7707 3 5 375 fits 40.40 15.20 9860 5 4 300 fits 53.00 17.30 11204 ``` #### Symptom and Cause I1 fixes sixty defects one by one: 4,125 minutes of corrective action, sixty recurrences, eighteen escaped defects. No class stops, because every intervention only closes that one defect. I3 opens five reviews, drops recurrence to 40.40, escaped defects to 15.20, and pulls corrective cost down to 2,813 minutes; the 1,312 minutes in between are fixes never written. This does not mean preventive action is free. I3 pays 1,200 minutes for preventive action, 375 for review; its total is 9,860 minutes, 745 minutes better than I1's 10,605. **Cutting recurrence pays off in the share of the recurrence it cuts that would have escaped.** I2 gives the lowest total (9,374 minutes), but it opens twelve reviews and asks for 600 minutes; TP24's budget is 400. **The best row on paper does not fit the budget**; the cheapest intervention that fits is I3. #### Depth's Two Sides The structure table separates depth's two opposing effects. Class per node rises from 1.00 to 14.00 — a deeper cause feeds more classes — but applicability drops from 0.90 to 0.00. Their product is 0.90, 1.63, 2.80, and 0.00: highest at the fourth link, zero at the fifth. Still, this ranking does not decide alone. I4 is the link with the highest expected coverage, but it does worse than I3 over the horizon (44.40 recurrences against 40.40). The reason is delay: depth k delays the action's landing by k−1 releases, and the defect recurring while delayed is 1.00 for I2, 8.00 for I3, 11.00 for I5. **Going deeper is not just expensive, it is slow; slowness produces its own recurrence.** #### The Fifth Link's Cost I5 is this lesson's clearest number. It opens a single review, pays 125 minutes of review and 2,400 minutes of preventive action, and then nothing changes: recurrence 60.00, escaped defects 18.00 — both identical to I1's. Its total is 13,130 minutes, **2,525 minutes more expensive than reviewing nothing at all.** The chain is correct; the problem is not in the chain. "Process owner not assigned" genuinely feeds all fourteen classes. But the action written at that link is not one the team can carry out, and an action not carried out prevents zero defects. **Five whys is not a race to depth: the place to stop is the last link the team can actually change.** #### Threshold and the Return on the Decision The threshold scan repeats the same pattern. Threshold 2 gives the lowest total (7,707 minutes, 29.90 recurrences, 10.30 escaped), but it opens six reviews and asks for 450 minutes — it exceeds the budget. Threshold 5 fits but leaves recurrence at 53.00. **The threshold's source is not a requirement but a two-sided measurement**: the per-class recurrence average gives the floor, the review budget gives the ceiling. The decision's return is two queues. Corrective action closes a record and ends with the defect's owner. Preventive action opens a process change — a review step, a checklist item, a documented assumption — and its closing depends on an approval outside the team. A root cause record opened without an owner written down meets the same end as the planning topic's ownerless items: covered on paper, not run in practice. #### Summary - The causal chain is a data structure: the first two links are class-specific, the rest are shared, and node count drops from 14 to 1. - Fixing the symptom leaves sixty recurrences and eighteen escaped defects unchanged; stopping at link 3 drops recurrence to 40.40, escaped defects to 15.20. - Depth has two sides: class per node rises from 1.00 to 14.00 while applicability drops from 0.90 to 0.00. Depth is also delay — the defect recurring while delayed is 1.00 for I2, 8.00 for I3, 11.00 for I5. - Stopping at the fifth link pays 2,525 minutes for nothing: recurrence and escaped defects stay identical to not reviewing at all. - The threshold's source is a two-sided measurement: the per-class recurrence average sets the floor, the review budget sets the ceiling. #### Course Wrap-Up This course's eleven lessons followed the same pattern: a limited resource was distributed, and the escaped defects and feedback time the distribution produced were measured. | Lesson | Resource distributed | Distribution decision | Escaped defects / feedback time | |---|---|---|---| | Test Plan | 120 min test budget per release | nine tests fitting the budget | 18 escaped / 16.5 min (full coverage 10 / 40.8) | | Risk-Based Prioritization | same 120 min, order free | estimate/minute order | 18 escaped, escaped risk 155 / 18.2 min | | Exploratory Testing | 120 min session = width × depth | four paths, depth five | 10.39 escaped / 74.8 min | | Acceptance Criteria | 300 min product-side attention | seventeen semantic criteria (204 min) | 0 escaped / acceptance review stage | | User Acceptance Testing | 600 user-minutes, twelve tasks | by usage share | 10 escaped / 26.0 min to finding | | Testing in Continuous Integration | 1021 model min run | three stages and fail fast | 5 escaped / 228.1 min | | Management of Test Environments | environment count and environment-minutes | production-like environment to nightly stage | 5 escaped / 263.5 min | | Secrets and Data Management | secrets entering the test environment | narrowing scope from 511,800 to 1,809 records | 5 escaped, unchanged / 263.5 min | | Reporting | 30 min attention, steps opened | grouped report | 0 roots escaped / 255.7 min | | Quality Metrics | 180 min attention per period | two decision-changing metrics | leakage rate 6.25% → 12.50% / 386.8 → 398.0 min | | Root Cause Analysis | 400 min review across the horizon | stopping at the third link | 15.20 escaped / lands at release 10.8 | There are two zeros in the last column, and neither refutes the rule: Acceptance Criteria zeroes out escaped defects but moves the cost to the acceptance review stage, and Reporting zeroes out escaped **roots** — not escaped defects. The rule holds: **every distribution produces escaped defects.** There is no undivided state for dividing a limited resource; the plan that fits the budget misses eighteen defects, and the plan that does not fit does not run at all. A plan that wants zero escaped defects consumes the budget — in the Exploratory Testing lesson, doubling the session budget dropped escaped defects to 6.27 while the net benefit went negative, and that is this rule counted out. The only way to defend a distribution is to write out its escaped defects and its feedback time together. This course was the fifth and last of the Software Quality and Testing curriculum. Quality and Testing Fundamentals built the concepts: defect, class, level, and oracle. Unit Testing and Test-Driven Development worked the cheapest link — the fastest-running, narrowest-seeing test. Integration, Contract and End-to-End Testing ran real boundaries and asked whether the fake matched the real thing. Non-Functional Testing tested behavior under load, attack, and failure with thresholds. This course placed all of them within time and attention: which test runs when, what decision a red result produces. The shared conclusion fits in one sentence — a test suite is only meaningful within a budget, and the decision that divides that budget is made in the reader's own team. The numbers here show that decision's shape; they do not take its place.