Risk-Based Prioritization

The previous lesson built the budget-fitting plan by taking the cheapest tests in order; that ordering’s only justification was fitting as many items as possible into 120 minutes. Yet every defect has a likelihood and an impact, and a test’s cheapness has nothing to do with either. What would change if the same budget were distributed in a different order?

This lesson turns ordering into a decision. Risk-based prioritization distributes the limited budget by defect classes’ risk; risk here is the product of two quantities — a defect’s likelihood of appearing and the impact it has when it does. The distributed resource is again 120 minutes per release (TP3); what is measured is again escaped defect and feedback time.

The Risk Score and Its Estimate

A defect’s risk score is the product of likelihood and impact; a class risk is the sum of the risk scores of the defects in that class. The likelihood and impact values in the defect set come from the first lesson’s seed, not invented.

TP8 (assumption) — risk score is likelihood × impact, the two quantities weighted equally. TP9 (assumption) — the plan does not know the actual class risk, it estimates it; the estimate is the actual value times $[1-e,\,1+e]$, and the baseline case is $e = 0{,}50$. The actual class risk is known only after the defects surface; the ordering is done beforehand. TP10 (assumption) — the number of processes a test spins up is an observable proxy for the number of components it touches. The “most components first” order is built with this proxy; the process counts are read from the closing tables.

// library/inventory.mjs — the first lesson's inventory and defect set, in short form.
// Cost quantities from the M21/K03 and M21/K04 closing tables; conversion to minutes with TP1-TP2.
const RAW = [
  ["T01", "unit test suite", "unit", 0, 266, 0, "rule-boundary"],
  ["T02", "integration test", "integration", 1, 772, 0, "schema-mismatch"],
  ["T03", "database migration test", "integration", 1, 21, 0, "migration-data-loss"],
  ["T04", "contract test", "contract", 2, 100, 0, "contract-field"],
  ["T05", "end-to-end browser test", "end-to-end", 2, 16477, 0, "interface-state"],
  ["T06", "visual verification", "end-to-end", 1, 784, 1, "visual-deviation"],
  ["T07", "load test", "performance", 9, 55000, 0, "data-growth"],
  ["T08", "static security scan", "security", 0, 75, 7, "sql-concatenation"],
  ["T09", "dependency scan", "security", 0, 10, 5, "known-vulnerability"],
  ["T10", "accessibility audit", "security", 0, 14, 6, "accessibility"],
  ["T11", "fault injection", "resilience", 1, 13, 5, "degraded-response"],
  ["T12", "recovery verification", "resilience", 2, 7, 10, "recovery-gap"],
];
export const TESTS = RAW.map(([code, name, level, processes, calls, manual, cls]) => ({
  code, name, level, processes, calls, manual, cls, min: 0.5 * processes + 0.0002 * calls + 6 * manual,
}));
export const UNCOVERED = ["off-scenario", "semantic-drift"];
const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4],
  ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4],
  ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3],
  ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]];
export function defects(seed, n) {
  let x = seed;
  const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; };
  const total = WEIGHT.reduce((a, [, w]) => a + w, 0);
  const list = [];
  for (let i = 1; i <= n; i += 1) {
    let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0];
    for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } }
    list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5),
      impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) });
  }
  return list;
}

Same Budget, Five Orders

An order is a test list; items are taken in sequence until the budget fills, and what does not fit is skipped. Five orders are compared: descending by estimated risk (S1), descending by estimated risk divided by minutes (S2), cheapest first (S3, the first lesson’s order), random (S4), most components touched first (S5).

// library/risk.mjs — same budget, five orders, same defect set. Input: library/inventory.mjs
import { TESTS, UNCOVERED, defects } from "./inventory.mjs";

const BUDGET = 120, SEED = 20260731, N = 60, MULTIPLIER = 6, CANDIDATES = 20;   // TP3, TP5, TP6
const DEFECTS = defects(SEED, N);
const CLASSES = [...new Set([...TESTS.map((t) => t.cls), ...UNCOVERED])];
const v = (x, n = 1) => x.toFixed(n);
const g = (x, n) => String(x).padStart(n);
const makeRandom = (seed) => { let x = seed;
  return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; };

// TP8: risk score = likelihood x impact; class risk is the risk total of that class's defects.
const ACTUAL = new Map(CLASSES.map((s) => [s, DEFECTS.filter((k) => k.cls === s)
  .reduce((a, k) => a + k.likelihood * k.impact, 0)]));
// TP9: the plan does not know the actual risk, it estimates; the estimate drifts by [1-e, 1+e].
const estimate = (e, r) => new Map(CLASSES.map((s) => [s, ACTUAL.get(s) * (1 - e + 2 * e * r())]));

// From an ordered list, take what fits the budget; what does not fit is skipped, order stays.
const select = (order) => { const c = []; let t = 0;
  for (const x of order) if (t + x.min <= BUDGET) { c.push(x); t += x.min; } return c; };
function measure(c) {
  const moment = new Map(); let t = 0;
  for (const x of c) { t += x.min; moment.set(x.cls, t); }
  const caught = DEFECTS.filter((k) => moment.has(k.cls)),
    escaped = DEFECTS.filter((k) => moment.has(k.cls) === false);
  return { n: c.length, code: c.map((x) => x.code).join(" "), time: t, caught: caught.length,
    escaped: escaped.length, rate: (100 * escaped.length) / DEFECTS.length,
    risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0),
    feedback: caught.reduce((a, k) => a + moment.get(k.cls), 0) / (caught.length || 1),
    fix: escaped.reduce((a, k) => a + k.fix, 0) };
}
const riskOrder = (T) => [...TESTS].sort((a, b) => T.get(b.cls) - T.get(a.cls));
const densityOrder = (T) => [...TESTS].sort((a, b) => T.get(b.cls) / b.min - T.get(a.cls) / a.min);

const TOTAL = [...ACTUAL.values()].reduce((a, x) => a + x, 0);
const BASE = UNCOVERED.reduce((a, s) => a + ACTUAL.get(s), 0);
console.log(`class risk (TP8) and estimate (TP9, e = 50%) — seed ${SEED}, ${N} defects, ${CLASSES.length} classes`);
console.log("class                defects  actual  estimate  test     min  estimate/min");
const T50 = estimate(0.5, makeRandom(8125));
for (const s of CLASSES) {
  const t = TESTS.find((x) => x.cls === s);
  console.log(s.padEnd(21) + g(DEFECTS.filter((k) => k.cls === s).length, 5) + g(ACTUAL.get(s), 8) +
    g(v(T50.get(s)), 8) + "  " + (t ? t.code : "none") + g(t ? v(t.min, 2) : "-", 7) +
    g(t ? v(T50.get(s) / t.min, 2) : "-", 11));
}
console.log(`total risk ${TOTAL}; the two classes no test sees ${BASE} points -> reachable risk ${TOTAL - BASE}`);

const S = { S1: riskOrder(T50), S2: densityOrder(T50),
  S3: [...TESTS].sort((a, b) => a.min - b.min),
  S5: [...TESTS].sort((a, b) => b.processes - a.processes || b.calls - a.calls) };
const LABEL = { S1: "S1 risk descending (estimate)", S2: "S2 estimate/min descending", S3: "S3 cheapest first",
  S5: "S5 most components first" };
const r4 = makeRandom(4242), draws = [];
for (let i = 0; i < 200; i += 1) { const p = [...TESTS];
  for (let j = p.length - 1; j > 0; j -= 1) { const q = Math.floor(r4() * (j + 1));
    [p[j], p[q]] = [p[q], p[j]]; }
  draws.push(measure(select(p))); }
const avg = (f) => draws.reduce((a, o) => a + f(o), 0) / draws.length;

// The last column is the decision's return on investment: CANDIDATES releases of run time plus the MULTIPLIER-scaled fix cost of escaped defects.
console.log(`\nsame budget TP3 = ${BUDGET} min, same defect set; last column ${CANDIDATES} releases with TP5-TP6`);
console.log("order                          test   time  escaped   leak%  escaped risk  feedback  total min");
const w = (x) => (Number.isInteger(x) ? String(x) : v(x, 2));
const printRow = (name, o) => console.log(name.padEnd(30) + g(w(o.n), 5) + g(v(o.time), 7) +
  g(w(o.escaped), 8) + g(v(o.rate), 8) + g(v(o.risk), 14) + g(v(o.feedback), 10) +
  g(v(CANDIDATES * o.time + MULTIPLIER * o.fix, 0), 11));
for (const k of ["S1", "S2", "S3", "S5"]) printRow(LABEL[k], measure(select(S[k])));
printRow("S4 random (200 permutations)", { n: avg((o) => o.n), time: avg((o) => o.time),
  escaped: avg((o) => o.escaped), rate: avg((o) => o.rate), risk: avg((o) => o.risk),
  feedback: avg((o) => o.feedback), fix: avg((o) => o.fix) });
console.log(`S4 spread: escaped risk best ${Math.min(...draws.map((o) => o.risk))}, ` +
  `worst ${Math.max(...draws.map((o) => o.risk))}; ${draws.filter((o) => o.risk <=
  measure(select(S.S2)).risk).length} of 200 permutations are as good as S2`);
for (const k of ["S1", "S5"]) console.log(k + " selected: " + measure(select(S[k])).code);

console.log("\nestimate error, TP9's sensitivity — 200 draws per e, average escaped risk");
console.log("e      S1 average   S2 average   draws where S2 > S3   worst S2");
const S3R = measure(select(S.S3)).risk;
for (const e of [0, 0.25, 0.5, 1]) {
  const r = makeRandom(9001); let a = 0, b = 0, beaten = 0, worst = 0;
  for (let i = 0; i < 200; i += 1) { const T = estimate(e, r);
    const o1 = measure(select(riskOrder(T))), o2 = measure(select(densityOrder(T)));
    a += o1.risk; b += o2.risk; worst = Math.max(worst, o2.risk); if (o2.risk > S3R) beaten += 1; }
  console.log(g(Math.round(100 * e) + "%", 4) + g(v(a / 200), 12) + g(v(b / 200), 13) +
    g(beaten, 21) + g(worst, 12));
}
console.log(`comparison baseline: S3 cheapest first, escaped risk ${S3R}`);

class risk (TP8) and estimate (TP9, e = 50%) — seed 20260731, 60 defects, 14 classes
class                defects  actual  estimate  test     min  estimate/min
rule-boundary            7      63    40.8  T01   0.05     766.99
schema-mismatch          7      65    60.2  T02   0.65      92.04
migration-data-loss      3      15    21.6  T03   0.50      42.85
contract-field           7      75   101.7  T04   1.02      99.68
interface-state          4      39    24.0  T05   4.30       5.58
visual-deviation         2      23    33.2  T06   6.66       4.99
data-growth              6      51    64.5  T07  15.50       4.16
sql-concatenation        3      51    57.3  T08  42.02       1.36
known-vulnerability      3      33    28.4  T09  30.00       0.95
accessibility            4      32    32.5  T10  36.00       0.90
degraded-response        3      21    12.9  T11  30.50       0.42
recovery-gap             1       9     5.3  T12  61.00       0.09
off-scenario             4      33    49.1  none      -          -
semantic-drift           6      60    78.7  none      -          -
total risk 570; the two classes no test sees 93 points -> reachable risk 477

same budget TP3 = 120 min, same defect set; last column 20 releases with TP5-TP6
order                          test   time  escaped   leak%  escaped risk  feedback  total min
S1 risk descending (estimate)     9  106.7      17    28.3         156.0      48.9       8434
S2 estimate/min descending        9  100.7      18    30.0         155.0      18.2       9574
S3 cheapest first                 9   89.2      18    30.0         185.0      16.5       9704
S5 most components first          9  119.7      20    33.3         197.0      61.4      11484
S4 random (200 permutations)   8.48  111.7   21.36    35.6         207.1      75.8      11220
S4 spread: escaped risk best 155, worst 377; 17 of 200 permutations are as good as S2
S1 selected: T04 T07 T02 T08 T01 T06 T10 T05 T03
S5 selected: T07 T05 T04 T12 T06 T02 T03 T01 T09

estimate error, TP9's sensitivity — 200 draws per e, average escaped risk
e      S1 average   S2 average   draws where S2 > S3   worst S2
  0%       155.0        155.0                    0         155
 25%       155.7        156.4                    0         174
 50%       161.8        161.6                    4         186
100%       175.5        167.3                   20         204
comparison baseline: S3 cheapest first, escaped risk 185

Counting Numbers versus Counting Risk

Of sixty defects, the caught counts are 43, 42, 42, 40, averaging 38.64; the escaped defect count barely separates the five orders: 17, 18, 18, 20, averaging 21.36. The escaped defects’ risk total, though, does separate them: 156, 155, 185, 197, averaging 207.1. What makes the difference is that one counts every defect as one, the other counts by weight. S2 and S3 escape the same number of defects, but S3’s escapees carry 30 more risk points.

The comparison’s floor is in the table too: of the total 570 risk points, 93 belong to the two classes no test sees, so the range the order can actually move is 477 points. S2 misses 62 points of that range, S3 misses 92 — a 1.48x difference from reordering alone. A sentence like “escaped risk 155,” written without looking at the floor, hides this ratio.

Risk Order versus Density Order

S1 and S2 use the same estimate and arrive at nearly the same escaped risk (156 against 155). The distinction lies elsewhere: S1’s feedback time is 48.9 minutes, S2’s is 18.2 — a 2.7x difference. The reason is visible in the orders chosen. S1 places the 42.02-minute static security scan fourth, because that class’s estimated risk is high; S2 pushes the same test to the end, because once divided by minutes its estimate/min value is 1.36. A high-risk but expensive item placed near the front of the order delays the reporting moment of every test that follows it.

Risk order alone is the wrong order; the divisor is minutes. T01’s estimate/min value is 766.99, T12’s is 0.09 — a difference no risk ordering by itself can see.

S5 shows this from the other side: the most-components-first order puts the load test and the recovery verification among its items, spending 76.5 of the 120 minutes on these two tests alone, and escapes 20 defects. Touching a lot of code is not the same as seeing a lot of risk. S4, in turn, shows that the order itself is a gain: the average of two hundred permutations is 207.1, the worst is 377, and only 17 are as good as S2. A plan with no order written down is, on average, a random plan.

The Estimate’s Margin of Error

The ordering rests on the estimate, not the actual risk, and the estimate is wrong. The sensitivity scan draws two hundred samples at each error level. At zero error the two orders meet at 155 — this is the upper bound. When the error rises to ±50%, S2’s average becomes 161.6, and only four of the two hundred draws are worse than the cheapest-first order’s 185. At ±100% error the average stays at 167.3, the number of beaten draws rises to 20, and the worst draw is 204.

The gain erodes slowly, because the order uses the estimate’s rank, not its value: even when a class’s risk is misestimated by a factor of two, its place in the list mostly stays the same. The rule that follows from this is writing down the estimate’s source — the class distribution in the defect history, change intensity, the number of affected users. A risk order with no written source is a preference list; the ±100% row gives the price of that preference.

The Return on the Decision

The table’s last column — twenty releases of run time plus the escaped defects’ six-fold fix cost — tests the ordering with a third quantity: S1 is cheapest at 8,434 minutes, S2 at 9,574, S3 at 9,704, random at 11,220, most-components at 11,484. S1’s total cost is lower even though its escaped risk is higher than S2’s, because the risk score does not measure fix cost: in the defect set, fix minutes are drawn independent of likelihood and impact. No single order can optimize all three quantities at once; which one gets optimized is written into the plan.

What a red result means also comes from the order. When the tests of the classes with the highest risk — the contract test and the integration test — turn red, the release candidate does not advance; the items further down the queue open a record. The order itself is also a maintenance item: as the defect history changes, class risk changes, and a risk order that is not re-derived turns into a habit after a few releases and its gain drops to the values in the ±100% row.

Summary

• Risk score is the product of likelihood and impact (TP8); the plan does not know class risk, it estimates it (TP9).
• Escaped defect count separates the five orders between 17 and 21.36, escaped risk between 155 and 207.1; unweighted counting hides the ordering’s gain.
• Of the total 570 risk points, 93 belong to the two classes no test sees; the range the order can move is 477, and within that range the difference between S2 and S3 is 1.48x.
• Risk order pushes an expensive item to the front and stretches feedback to 48.9 minutes; once the same estimate is divided by minutes, escaped risk stays at 155 and feedback drops to 18.2 minutes.
• When estimate error rises to ±100%, the density order’s average becomes 167.3 and 20 of the two hundred draws lose to the cheapest-first order; the gain erodes slowly.
• Risk score does not measure fix cost: the order with the lowest escaped risk may not be the cheapest order.

Next Step

All five orders chose from within the same twelve tests, and none of them touched the 93-point floor. 33 of those points belong to the off-scenario class: four defects that surface on paths no written scenario passes through. The way to see this class is not to change the order but to change the test form — a session whose script is not written in advance, whose direction is decided during the test itself. The next lesson turns that session into a budget item: how the narrowness of scope changes what the session finds, and how much of what it finds converts into the scripted set.