Test Plan

The Non-Functional Testing course closed four courses’ accumulation with a single question: which test runs when. Some of these tests are cheap enough to run on every change, some are expensive enough to run once per release; how limited time is divided among them was addressed in no lesson.

Every lesson in this course follows the same shape: a limited resource is distributed, the escaped defect it produces and the feedback time it costs are measured, and what a red result turns into is written down. This escaped defect is the same concept measured in the Integration, Contract and End-to-End Testing course; this course keeps the short name.

The Plan’s Three Columns

A test plan writes down three things: scope says which defect classes are tested, criteria say when the plan starts and when it closes, responsibility says who owns each item. Until all three are tied to a number, a plan is a statement of intent.

Criteria have two faces. Entry criteria are the condition a stage must meet before it can start running; a test that runs without it satisfied produces no information, only spends time. Exit criteria say the plan has closed, and this lesson ties it to the share of defect classes covered.

Inventory: Four Courses’ Tests in One List

Across four courses, twelve test forms were written for the library loan system. Each one’s cost is recorded in its source lesson’s closing table as a quantity independent of the run: how many processes, how many requests or queries, how many manual review steps. Two assumptions convert these into minutes.

TP1 (assumption) — run minutes: 0.5 min per process, 0.0002 min per call. The process share comes from the six steps of the readiness poll in the Integration, Contract and End-to-End Testing course; the call share is 12 milliseconds per request. TP2 (assumption) — 6 minutes of human time per manual review step.

// library/inventory.mjs — the test suite written across four courses and the course's shared
// defect set. Cost quantities are taken from the M21/K03 and M21/K04 closing tables; the
// conversion to minutes uses the TP1 (run) and TP2 (human) assumptions.
const RAW = [
  // code, name, level, processes, calls, manual steps, defect class caught
  ["T01", "unit test suite",         "unit",        0,   266,  0, "rule-boundary"],
  ["T02", "integration test",        "integration", 1,   772,  0, "schema-mismatch"],
  ["T03", "database migration test", "integration", 1,    21,  0, "migration-data-loss"],
  ["T04", "contract test",           "contract",    2,   100,  0, "contract-field"],
  ["T05", "end-to-end browser test", "end-to-end",  2, 16477,  0, "interface-state"],
  ["T06", "visual verification",     "end-to-end",  1,   784,  1, "visual-deviation"],
  ["T07", "load test",               "performance", 9, 55000,  0, "data-growth"],
  ["T08", "static security scan",    "security",    0,    75,  7, "sql-concatenation"],
  ["T09", "dependency scan",         "security",    0,    10,  5, "known-vulnerability"],
  ["T10", "accessibility audit",     "security",    0,    14,  6, "accessibility"],
  ["T11", "fault injection",         "resilience",  1,    13,  5, "degraded-response"],
  ["T12", "recovery verification",   "resilience",  2,     7, 10, "recovery-gap"],
];

export const TESTS = RAW.map(([code, name, level, processes, calls, manual, cls]) => ({
  code, name, level, processes, calls, manual, cls,
  run: 0.5 * processes + 0.0002 * calls,          // TP1
  human: 6 * manual,                              // TP2
  min: 0.5 * processes + 0.0002 * calls + 6 * manual,
}));

// The two classes no member of the automated set catches.
export const UNCOVERED = ["off-scenario", "semantic-drift"];

const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4],
  ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4],
  ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3],
  ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]];

export function defects(seed, n) {
  let x = seed;
  const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; };
  const total = WEIGHT.reduce((a, [, w]) => a + w, 0);
  const list = [];
  for (let i = 1; i <= n; i += 1) {
    let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0];
    for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } }
    list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5),
      impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) });
  }
  return list;
}

The second structure is the defect set, and it is the shared input of every lesson in this course: fourteen classes, with likelihood, impact, and fix minutes drawn from a seed. Class names are taken from the defects each source lesson caught. Two of them — off-scenario and semantic-drift — are seen by no test in the inventory; the second is the semantic mismatch the API Testing lesson of the Integration, Contract and End-to-End Testing course recorded in its “missed” column.

Four Plans, One Defect Set

A plan is an ordered test list. A defect is caught in the minute the first test that sees its class finishes; if no test sees it, it escapes. Defect leakage rate is the escaped share of the set.

TP3 (assumption) — release test budget: 120 min. TP4 (assumption) — exit criterion: covered defect class share at least 70%. TP5 (assumption) — an escaped defect’s fix cost is 6 times a caught one’s. TP6 (assumption) — a horizon of 20 release candidates; the defect set appears across this horizon. TP7 (assumption) — a quarter of the candidates are red on the smoke test.

// library/plan.mjs — builds the test plan as a data structure and compares four plans over
// the same defect set. Input: library/inventory.mjs
import { TESTS, UNCOVERED, defects } from "./inventory.mjs";

const BUDGET = 120, CANDIDATES = 20, MULTIPLIER = 6, SEED = 20260731, N = 60;   // TP3, TP6, TP5
const OWNER = { T01: "development", T02: "development", T03: "development", T04: "development",
  T05: "testing", T06: "testing", T07: "testing", T08: "security", T09: "security",
  T10: "product", T11: "operations", T12: "operations" };
const DEFECTS = defects(SEED, N);
const CLASSES = [...new Set([...TESTS.map((t) => t.cls), ...UNCOVERED])];
const v = (x, n = 2) => x.toFixed(n);
const g = (x, n) => String(x).padStart(n);

// A plan is an ordered test list. Escaped defect = a defect no test in the plan sees.
// Feedback time = the average of the cumulative minute at which a caught defect is caught.
function measure(order) {
  const moment = new Map();
  let t = 0;
  for (const test of order) { t += test.min; moment.set(test.cls, t); }
  const caught = DEFECTS.filter((k) => moment.has(k.cls));
  const escaped = DEFECTS.filter((k) => moment.has(k.cls) === false);
  return { tests: order.length, cls: moment.size, time: t, escaped: escaped.length,
    rate: (100 * escaped.length) / DEFECTS.length,
    feedback: caught.reduce((a, k) => a + moment.get(k.cls), 0) / (caught.length || 1),
    fix: escaped.reduce((a, k) => a + k.fix, 0) };
}

const cheapest = [...TESTS].sort((a, b) => a.min - b.min);
const fitsBudget = [];
for (const t of cheapest) if (fitsBudget.reduce((a, x) => a + x.min, 0) + t.min <= BUDGET) fitsBudget.push(t);
const criterionSet = cheapest.slice(0, Math.ceil(0.7 * CLASSES.length));    // exit criterion: 70% of classes
const owned = TESTS.filter((t) => OWNER[t.code] !== "product" && OWNER[t.code] !== "security");
const PLANS = [["P1 full coverage", TESTS], ["P2 fits budget (no criterion)", fitsBudget],
  ["P3 exit criterion >= 70%", criterionSet], ["P4 full coverage, 3 items unowned", owned]];

console.log("test inventory — cost quantities from the M21/K03 and M21/K04 closing tables");
console.log("code  test                      level          run   human   total  class                owner");
for (const t of TESTS) console.log(t.code + "  " + t.name.padEnd(26) + t.level.padEnd(12) +
  g(v(t.run), 6) + g(v(t.human, 1), 8) + g(v(t.min), 8) + "  " + t.cls.padEnd(21) + OWNER[t.code]);
console.log("total".padEnd(44) + g(v(TESTS.reduce((a, t) => a + t.run, 0)), 6) +
  g(v(TESTS.reduce((a, t) => a + t.human, 0), 1), 8) +
  g(v(TESTS.reduce((a, t) => a + t.min, 0)), 8) + "  " + TESTS.length + "/" + CLASSES.length + " classes");

console.log(`\ndefect set: seed ${SEED}, ${N} defects, ${CLASSES.length} classes`);
console.log("class no test sees: " +
  UNCOVERED.map((s) => `${s} (${DEFECTS.filter((k) => k.cls === s).length})`).join(", "));

console.log(`\nplan comparison — budget TP3 = ${BUDGET} min/release, horizon TP6 = ${CANDIDATES} releases`);
console.log("plan                               test  class   time  budget  escaped    leak%  feedback");
for (const [name, order] of PLANS) {
  const o = measure(order);
  console.log(name.padEnd(35) + g(o.tests, 4) + g(o.cls, 7) + g(v(o.time, 1), 7) +
    g(o.time <= BUDGET ? "fits" : "over", 8) + g(o.escaped, 9) + g(v(o.rate, 1), 9) + g(v(o.feedback, 1), 10));
}

console.log("\nexit criterion scan: how many minutes a covered class share costs");
console.log("criterion  classes needed  cheapest time  budget  escaped");
for (const p of [50, 60, 70, 80, 90]) {
  const needed = Math.ceil((p / 100) * CLASSES.length);
  if (needed > TESTS.length) { console.log(g(p + "%", 9) + g(needed, 16) + g("unreachable", 15) +
    g("-", 8) + g("-", 9)); continue; }
  const o = measure(cheapest.slice(0, needed));
  console.log(g(p + "%", 9) + g(needed, 16) + g(v(o.time, 1), 15) +
    g(o.time <= BUDGET ? "fits" : "over", 8) + g(o.escaped, 9));
}

const smoke = TESTS[0].min + TESTS[1].min, ps = measure(fitsBudget).time;
let y = 4242;                                                       // TP7: candidate seed
const red = Array.from({ length: CANDIDATES }, () =>
  (y = (1103515245 * y + 12345) % 2147483648) / 2147483648 < 0.25).filter(Boolean).length;
console.log(`\nentry criterion: smoke test (T01+T02, ${v(smoke)} min) must be green or P2 does not run`);
console.log(`${CANDIDATES} release candidates, seed 4242: ${red} candidates are red on the smoke test`);
console.log(`without criterion  P2 runs ${CANDIDATES} times, ${v(CANDIDATES * ps, 1)} min; ${red} runs produce no information -> ${v(red * ps, 1)} min wasted`);
console.log(`with criterion     P2 runs ${CANDIDATES - red} times, ${v((CANDIDATES - red) * ps, 1)} min + smoke ${v(CANDIDATES * smoke, 1)} min = ${v((CANDIDATES - red) * ps + CANDIDATES * smoke, 1)} min`);

console.log(`\nescaped-defect multiplier TP5's sensitivity: total cost over ${CANDIDATES} releases (min)`);
console.log("mult   P1 full coverage   P2 fits budget  P3 criterion  cheapest plan");
for (const mult of [2, 6, 20]) {
  const b = PLANS.slice(0, 3).map(([, s]) => { const o = measure(s); return CANDIDATES * o.time + mult * o.fix; });
  const names = ["P1", "P2", "P3"];
  console.log(g(mult, 4) + g(v(b[0], 0), 19) + g(v(b[1], 0), 17) + g(v(b[2], 0), 14) +
    g(names[b.indexOf(Math.min(...b))], 15));
}

test inventory — cost quantities from the M21/K03 and M21/K04 closing tables
code  test                      level          run   human   total  class                owner
T01  unit test suite           unit          0.05     0.0    0.05  rule-boundary        development
T02  integration test          integration   0.65     0.0    0.65  schema-mismatch      development
T03  database migration test   integration   0.50     0.0    0.50  migration-data-loss  development
T04  contract test             contract      1.02     0.0    1.02  contract-field       development
T05  end-to-end browser test   end-to-end    4.30     0.0    4.30  interface-state      testing
T06  visual verification       end-to-end    0.66     6.0    6.66  visual-deviation     testing
T07  load test                 performance  15.50     0.0   15.50  data-growth          testing
T08  static security scan      security      0.02    42.0   42.02  sql-concatenation    security
T09  dependency scan           security      0.00    30.0   30.00  known-vulnerability  security
T10  accessibility audit       security      0.00    36.0   36.00  accessibility        product
T11  fault injection           resilience    0.50    30.0   30.50  degraded-response    operations
T12  recovery verification     resilience    1.00    60.0   61.00  recovery-gap         operations
total                                        24.21   204.0  228.21  12/14 classes

defect set: seed 20260731, 60 defects, 14 classes
class no test sees: off-scenario (4), semantic-drift (6)

plan comparison — budget TP3 = 120 min/release, horizon TP6 = 20 releases
plan                               test  class   time  budget  escaped    leak%  feedback
P1 full coverage                     12     12  228.2    over       10     16.7      40.8
P2 fits budget (no criterion)         9      9   89.2    fits       18     30.0      16.5
P3 exit criterion >= 70%             10     10  125.2    over       14     23.3      26.0
P4 full coverage, 3 items unowned     9      9  120.2    over       20     33.3      13.7

exit criterion scan: how many minutes a covered class share costs
criterion  classes needed  cheapest time  budget  escaped
      50%               7           28.7    fits       24
      60%               9           89.2    fits       18
      70%              10          125.2    over       14
      80%              12          228.2    over       10
      90%              13    unreachable       -        -

entry criterion: smoke test (T01+T02, 0.71 min) must be green or P2 does not run
20 release candidates, seed 4242: 3 candidates are red on the smoke test
without criterion  P2 runs 20 times, 1783.8 min; 3 runs produce no information -> 267.6 min wasted
with criterion     P2 runs 17 times, 1516.2 min + smoke 14.2 min = 1530.4 min

escaped-defect multiplier TP5's sensitivity: total cost over 20 releases (min)
mult   P1 full coverage   P2 fits budget  P3 criterion  cheapest plan
   2               5974             4424          4544             P2
   6               8794             9704          8624             P3
  20              18664            28184         22904             P1

Every number is deterministic: the same seed gives the same defect set. The inventory’s first reading already shows an imbalance — 204 of the plan’s 228.21 minutes are human time; the three most expensive items (T12, T08, T10) are almost entirely manual review steps.

Two Distributions, Two Outcomes

P1 covers every defect class and asks for 228.2 minutes per release — 108.2 minutes over budget. P2 takes the cheapest tests in order to fit the same budget: nine tests, 89.2 minutes. The two distributions’ outcomes on the same defect set differ: P1 lets 10 defects escape (16.7%), P2 lets 18 escape (30.0%). Against that, P2’s feedback time is 16.5 minutes, P1’s is 40.8 — because P1’s expensive tail reports what it catches late. A distribution that lowers the escaped defect count lengthens the feedback time. This is the trade-off that repeats in every lesson of this course.

The escaped defect’s cost depends on TP5, and this assumption is unmeasured; the sensitivity scan makes that visible. At multiplier 2 the cheapest plan is P2, at multiplier 6 it is P3, at multiplier 20 it is P1. So the plan’s justification depends on the value of a single assumption — yet only one of the three plans, P2, fits the budget. The budget is not a cost item; it is a limit.

The Exit Criterion’s Ceiling

The exit criterion scan prices the criterion itself. A 50% criterion costs 28.7 minutes, a 60% one costs 89.2, and both fit the budget. TP4’s chosen 70%, though, asks for 125.2 minutes — 5.2 minutes over budget. The criterion and the budget conflict, and the plan has to choose between them.

The 90% row says something different: unreachable. There is a test that sees twelve of the fourteen classes, none that sees the remaining two; no budget lifts this ceiling, because the ceiling comes from coverage, not time. The plan must either write these two classes explicitly out of scope, or add a test form that can see them. The escaped defect the out-of-scope choice produces in this set is 10 defects.

Entry Criterion and Responsibility

The entry criterion does not govern the run itself, but when it is allowed to start. Running P2 across twenty candidates while the smoke test (T01 and T02, 0.71 minutes) is not green spends 267.6 minutes on runs that produce no information. Once the criterion is in place, the smoke test running twenty times costs 14.2 minutes and the total drops to 1,530.4 minutes — a 253.4-minute gain, a cheap poll protecting an expensive stage.

The responsibility column is the plan’s quietest column. P4 writes the same scope as P1, but three items are owned outside the team and those items do not run. On paper 12 classes are covered, in practice 9; escaped defects rise from 10 to 20 and the leakage rate becomes 33.3% — a worse outcome than the budget-fitting, criterion-free plan, and this while the plan claims full coverage. An item with no owner written down is not in scope.

What a red result does is also written into the plan: when T01–T05 turn red the release candidate does not advance, T06 through T10 open a record without stopping the release, and T12’s red goes to a decision by operations. A test that changes no decision should not draw from the budget.

Summary

• A test plan is three columns: scope, criteria, and responsibility; until all three are tied to a number, a plan carries no decision.
• Four courses’ twelve tests total 228.21 minutes, and 204 of those are human time; since the budget (TP3) is 120 minutes, full coverage is a wish, not a plan.
• On the same defect set, full coverage lets 10 defects escape and reports in 40.8 minutes; the budget-fitting plan lets 18 escape and reports in 16.5 minutes.
• The exit criterion comes with a price: the 70% criterion asks for 125.2 minutes, over budget, and 90% is unreachable by any budget because two defect classes are seen by no test.
• The entry criterion puts a cheap poll ahead of an expensive stage: 267.6 minutes of wasted runs are prevented by a 14.2-minute smoke test.
• Three unowned items cut the full-coverage plan to nine tests and raise the leakage rate from 16.7% to 33.3%.

Next Step

The budget-fitting plan chose tests cheapest first; that ordering’s only justification was fitting as many items as possible into the budget. Yet every defect in the set has a likelihood and an impact, and cheapness has nothing to do with either. If the same 120 minutes were distributed in a different order, how many defects would be caught? The next lesson turns ordering into a decision: a subset chosen by the product of likelihood and impact is compared, at the same budget, against a random order, a cheapest-first order, and an order that touches the most code first.