--- title: 'Exploratory Testing' source: 'https://academia.sh/en/courses/testing-process/exploratory-testing' course: 'The Testing Process and Automation Infrastructure' language: en updated: '2026-08-23T14:25:19+00:00' license: 'CC BY-SA 4.0' --- # Exploratory Testing Turning testing whose script is not written in advance into a budget item: dividing the session's charter between width and depth, the defect class the scripted set cannot see becoming visible only through depth, the rate at which a found defect converts into a script, and measuring escaped defect and net cost across a twenty-release horizon. All five orders in the previous lesson chose from within the same twelve tests, and none of them touched the 93-point floor. One of the two classes producing that floor is `off-scenario`: defects that surface on paths no written scenario passes through. A class cannot be found by reordering as long as no test is written to see it. **Exploratory testing** is a test form whose script is not written in advance, whose direction is decided by what is seen during the test. Being free does not mean unstructured: the tester writes a **charter** — which area, for how long, looking for what. This lesson builds that charter as a budget distribution; the resource distributed is human attention, and it cannot be spent in two directions at once. ## Width and Depth Draw From the Same Budget The system's testable surface has two dimensions: how many **paths** are walked (width) and how far each path is followed (depth). A session's cost is the product of the two, so at a fixed budget one is traded for the other. **TP11 (assumption) — the per-release exploratory session budget is 120 minutes**; this is the same size as the first lesson's automated test budget (TP3), and the choice is tested with a sensitivity scan. **TP12 (assumption) — testing one path to one depth level takes 6 minutes.** The budget is thus 20 path-depth units. **TP13 (assumption) — every defect sits on one path at one depth; the two classes that fall outside scenarios are deep by definition (3–5), the rest are spread across 1–5.** This is a constraint that comes from the class's own definition: a defect sitting on a shallow path that written scenarios already pass through would already be visible in the scripted set. **TP14 (assumption) — a found class is converted into the scripted set starting the next release**; for an untested class this is 45 minutes to write plus 5 minutes of run time per release, for a class that already has a test in the inventory there is no writing, and the cost is that test's per-release minutes. ```js // library/inventory.mjs — the fields this lesson needs from the first lesson's inventory: each // test's defect class and cost quantities (processes, calls, manual steps; from the M21/K03 // and K04 closing tables). Conversion to minutes with the first lesson's TP1 and TP2, unchanged. const RAW = [["T01", "rule-boundary", 0, 266, 0], ["T02", "schema-mismatch", 1, 772, 0], ["T03", "migration-data-loss", 1, 21, 0], ["T04", "contract-field", 2, 100, 0], ["T05", "interface-state", 2, 16477, 0], ["T06", "visual-deviation", 1, 784, 1], ["T07", "data-growth", 9, 55000, 0], ["T08", "sql-concatenation", 0, 75, 7], ["T09", "known-vulnerability", 0, 10, 5], ["T10", "accessibility", 0, 14, 6], ["T11", "degraded-response", 1, 13, 5], ["T12", "recovery-gap", 2, 7, 10]]; export const TESTS = RAW.map(([code, cls, processes, calls, manual]) => ({ code, cls, min: 0.5 * processes + 0.0002 * calls + 6 * manual })); const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` ## Four Scopes, Twenty Releases Four distributions divide the same 120 minutes: twenty paths to depth one (E1), ten paths to depth two (E2), five paths to depth four (E3), four paths to depth five (E4). Which paths a scope covers is a matter of information — the plan does not know where the defect is, so path selection is seeded and random, and the average of two hundred draws is taken. The measurement runs across a twenty-release horizon: three defects surface each release, one session runs, and the found class enters the scripted set starting the next release. ```js // library/exploration.mjs — measures the exploratory session as a budget distribution. // Input: library/inventory.mjs (test set and defect set inherited from the first lesson). import { TESTS, defects } from "./inventory.mjs"; const SEED = 20260731, N = 60, PATHS = 20, UNIT = 6, SESSION = 120, DRAWS = 200; const MULTIPLIER = 6, CANDIDATES = 20, SHARE = N / CANDIDATES; // TP5, TP6: 3 defects/release // The second lesson's density order's nine budget-fitting items; the remaining classes are unscripted. const SCRIPTED = new Set(["T01", "T02", "T03", "T04", "T05", "T06", "T07", "T08", "T09"] .map((k) => TESTS.find((t) => t.code === k).cls)); // Unscripted classes split in two: ones with a test in the inventory that misses the budget (HAS_TEST) // and the two no test sees at all (DEEP). const HAS_TEST = new Map(TESTS.filter((t) => SCRIPTED.has(t.cls) === false).map((t) => [t.cls, t])); const DEEP = new Set(["off-scenario", "semantic-drift"]); const v = (x, n = 1) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); const makeRandom = (t) => { let x = t; return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; }; // TP13: every defect sits on one path at one depth. The two off-scenario classes are deep (3-5) // by definition, the rest are 1-5. Path and depth come from seed 31417. const r0 = makeRandom(31417); const DEFECTS = defects(SEED, N).map((k) => ({ ...k, path: 1 + Math.floor(r0() * PATHS), depth: DEEP.has(k.cls) ? 3 + Math.floor(r0() * 3) : 1 + Math.floor(r0() * 5) })); const UNSCRIPTED = DEFECTS.filter((k) => SCRIPTED.has(k.cls) === false); // A session walks p paths to depth d; its cost is p*d*UNIT minutes. A defect is found if its // path was walked and its depth does not exceed d; the find moment comes from that path's order. const walk = (r, p) => { const paths = Array.from({ length: PATHS }, (_, j) => j + 1); for (let j = PATHS - 1; j > 0; j -= 1) { const q = Math.floor(r() * (j + 1)); [paths[j], paths[q]] = [paths[q], paths[j]]; } return paths.slice(0, p); }; // Horizon: CANDIDATES releases, SHARE defects per release, and one session. A found class is // converted to the scripted set starting the next release (TP14), and no defect from it escapes after. function horizon(p, d, seed, budget = SESSION) { const r = makeRandom(seed), pathCount = Math.min(p, Math.floor(budget / (UNIT * d))), o = []; for (let c = 0; c < DRAWS; c += 1) { const converted = new Map(); let found = 0, moment = 0, escaped = []; for (let s = 1; s <= CANDIDATES; s += 1) { const place = new Map(walk(r, pathCount).map((y, i) => [y, (i + 1) * d * UNIT])); const newlyFound = []; for (const k of DEFECTS.slice((s - 1) * SHARE, s * SHARE)) { if (SCRIPTED.has(k.cls) || converted.has(k.cls)) continue; if (place.has(k.path) && k.depth <= d) { found += 1; moment += place.get(k.path); newlyFound.push(k.cls); } else escaped.push(k); } for (const x of newlyFound) if (converted.has(x) === false) converted.set(x, s); } const cost = [...converted].reduce((a, [x, s]) => a + (CANDIDATES - s) * (HAS_TEST.has(x) ? HAS_TEST.get(x).min : 5) + (HAS_TEST.has(x) ? 0 : 45), 0); o.push({ found, moment: moment / (found || 1), converted, cost, untested: [...converted.keys()].filter((x) => DEEP.has(x)).length, outOfBudget: [...converted.keys()].filter((x) => HAS_TEST.has(x)).length, escaped: escaped.length, risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), fix: escaped.reduce((a, k) => a + k.fix, 0) }); } const avg = (f) => o.reduce((a, x) => a + f(x), 0) / DRAWS; return { p: pathCount, d, budget, found: avg((x) => x.found), moment: avg((x) => x.moment), untested: avg((x) => x.untested), outOfBudget: avg((x) => x.outOfBudget), escaped: avg((x) => x.escaped), risk: avg((x) => x.risk), cost: avg((x) => x.cost), fix: avg((x) => x.fix) }; } const BASELINE_FIX = UNSCRIPTED.reduce((a, k) => a + k.fix, 0); const net = (o) => MULTIPLIER * (BASELINE_FIX - o.fix) - CANDIDATES * o.budget - o.cost; console.log(`defect set seed ${SEED}, ${N} defects; path and depth seed 31417, ${PATHS} paths`); console.log(`scripted set is the second lesson's nine items -> ${SCRIPTED.size} classes closed; ` + `${UNSCRIPTED.length} unscripted defects across the horizon, ${UNSCRIPTED.filter((k) => DEEP.has(k.cls)).length} in the two untested classes`); console.log("depth distribution (defect count): " + [1, 2, 3, 4, 5] .map((d) => d + ":" + DEFECTS.filter((k) => k.depth === d).length).join(" ")); console.log(`\nsession budget TP11 = ${SESSION} min = ${SESSION / UNIT} path-depth units; ` + `${CANDIDATES} releases, average of ${DRAWS} draws`); console.log("scope path depth found out-of-budget untested conversion find moment"); const D = [[20, 1], [10, 2], [5, 4], [4, 5]].map(([p, d], i) => horizon(p, d, 5501 + i)); D.forEach((o, i) => console.log(`E${i + 1} ${o.p} paths x depth ${o.d}`.padEnd(24) + g(o.p, 6) + g(o.d, 10) + g(v(o.found, 2), 10) + g(v(o.outOfBudget, 2), 15) + g(v(o.untested, 2), 10) + g(v((o.outOfBudget + o.untested) / o.found, 2), 12) + g(v(o.moment), 13))); console.log("\nend of horizon — the second lesson's density order alone was letting 18 defects escape"); console.log("scope escaped defect escaped risk session min conversion min gain from escaped defect net min"); D.forEach((o, i) => console.log(`E${i + 1}`.padEnd(9) + g(v(o.escaped, 2), 16) + g(v(o.risk), 15) + g(CANDIDATES * o.budget, 14) + g(v(o.cost, 0), 16) + g(v(MULTIPLIER * (BASELINE_FIX - o.fix), 0), 26) + g(v(net(o), 0), 9))); console.log(`without a session, baseline: 18 escaped, 155 risk, fix ${BASELINE_FIX} min (${MULTIPLIER * BASELINE_FIX} min with TP5)`); console.log("\nTP11's sensitivity — E4's depth (5) across three budgets; the 120 min row is E4 itself"); console.log("budget min paths walked found untested escaped defect net min"); for (const b of [60, 120, 240]) { const o = horizon(PATHS, 5, 5504, b); console.log(g(b, 10) + g(o.p, 15) + g(v(o.found, 2), 8) + g(v(o.untested, 2), 11) + g(v(o.escaped, 2), 17) + g(v(net(o), 0), 9)); } ``` ``` defect set seed 20260731, 60 defects; path and depth seed 31417, 20 paths scripted set is the second lesson's nine items -> 9 classes closed; 18 unscripted defects across the horizon, 10 in the two untested classes depth distribution (defect count): 1:18 2:10 3:13 4:8 5:11 session budget TP11 = 120 min = 20 path-depth units; 20 releases, average of 200 draws scope path depth found out-of-budget untested conversion find moment E1 20 paths x depth 1 20 1 3.00 3.00 0.00 1.00 64.4 E2 10 paths x depth 2 10 2 2.04 2.04 0.00 1.00 63.0 E3 5 paths x depth 4 5 4 2.53 1.35 1.12 0.98 73.3 E4 4 paths x depth 5 4 5 2.75 1.29 1.42 0.98 74.8 end of horizon — the second lesson's density order alone was letting 18 defects escape scope escaped defect escaped risk session min conversion min gain from escaped defect net min E1 12.00 114.0 2400 1281 2610 -1071 E2 13.85 126.5 2400 867 1881 -1386 E3 11.32 104.1 2400 729 2831 -298 E4 10.39 95.2 2400 735 3280 145 without a session, baseline: 18 escaped, 155 risk, fix 1260 min (7560 min with TP5) TP11's sensitivity — E4's depth (5) across three budgets; the 120 min row is E4 itself budget min paths walked found untested escaped defect net min 60 2 1.57 0.90 13.81 202 120 4 2.75 1.42 10.39 145 240 8 4.09 1.79 6.27 -923 ``` ## The Class Width Cannot See E1 walks all twenty paths and finds everything it can find: three unscripted defects sitting at depth 1. These three are a whole number, unchanged from draw to draw — at full width no randomness remains. Against that, E1's `untested` column is **0.00**, and stays 0.00 no matter how many draws are run. The reason is not a probability but a constraint: the two off-scenario classes' defects never sit at a depth shallower than 3, because if they did, written scenarios would already pass through there. E4 looks at only four paths with the same budget and finds fewer defects (2.75 against 3.00) — but 1.42 of what it finds come from the untested classes. **Width finds more defects, depth finds a different kind of defect.** The difference between the two distributions is not a difference in efficiency but a difference in reach; E1's shortfall is not that it works less, but that there is a place it never looks. The find moment shows the price of this reach: between 63.0 and 74.8 minutes. The previous lesson's density order was reporting a defect at an average of 18.2 minutes; the exploratory session reports with a delay close to four times that, and the delay grows with depth, because reaching the end of a deep path takes time. ## Conversion Rate and Two Kinds of Finding The rate at which a found defect converts into the scripted set falls between 0.98 and 1.00 across all four distributions: nearly every defect the session finds leads to a new scripted test. This is not as good news as it looks, because what converts is of two kinds. An **out-of-budget class** is one that has a test in the inventory but fell out of the plan because it does not fit 120 minutes — all three classes E1 finds are of this kind. A finding like this brings no new information; it re-asks the first lesson's budget decision, and its conversion brings back that decision's cost too: E1's conversion cost is 1,281 minutes, because the converted tests run every remaining release. An **untested class**, by contrast, is one with no counterpart in the inventory; its conversion is 45 minutes to write plus 5 minutes per release, and it is the small part of E4's 735-minute conversion cost. The distinction changes the decision. Finding an out-of-budget class is the plan's outcome, not the session's, and the answer to it is a budget discussion, not a session. Finding an untested class, though, is the session's own product, and it leaves a permanent test behind. ## The Number at the End of the Horizon At the end of twenty releases, the previous lesson's order alone was letting 18 defects escape, at 155 risk points. The exploratory session lowers that number in every distribution: to 13.85 for E2, 12.00 for E1, 11.32 for E3, 10.39 for E4. Escaped risk drops from 155 to 95.2 — meaning the session does not just let fewer defects escape, what escapes is also lighter, because the two untested classes sat at the heavy end of what escaped. The net column ties this to a decision. The twenty-release session cost is 2,400 minutes, and only E4 turns positive: 145 minutes. E1 lowers escaped defect to 12 but loses 1,071 minutes, because everything it finds requires an expensive test to be brought back into the budget. **A distribution lowering the escaped defect count does not justify it by itself; the cost of that reduction is counted too.** The TP11 scan gives this its limit. When the budget drops to 60 minutes, escaped defect rises to 13.81 but net rises to 202; raised to 240 minutes, escaped defect drops to 6.27 and net becomes −923. Escaped defect keeps falling with the budget, net does not. A plan that wants zero escaped defects exhausts the budget; this is this course's repeating trade-off, in this lesson's shape. What a red result means is also different here. An exploratory session does not give a pass or a fail, it gives a list of findings; every row in that list either opens a record or gives birth to a test. The authority to stop a release does not sit with the session — the session's product is a scripted test that can stop a release in later releases. ## Summary - A session's charter is a budget: the product of width and depth is fixed (TP11–TP12), one is traded for the other. - Full width (E1) is certain to find the three defects at depth 1 but never sees the two untested classes; this is not a probability but a constraint from the class's own definition (TP13). - Deep scope (E4) finds fewer defects (2.75 against 3.00) but 1.42 of what it finds come from untested classes; the difference is one of reach, not efficiency. - Conversion rate is 0.98–1.00, but most of what converts is out-of-budget classes: not new information, but the budget decision being re-asked. - Across twenty releases, escaped defect drops from 18 to 10.39, escaped risk from 155 to 95.2; at a session cost of 2,400 minutes, only deep scope turns positive (145 minutes). - When the budget is raised to 240 minutes, escaped defect drops to 6.27 and net becomes −923: escaped defect keeps dropping with the budget, net does not. ## Next Step So far, three lessons have distributed testing itself: which test, in what order, at what scope. All three share a silent assumption — that a defect is a defect, that a piece of work being done is clear. Yet the exploratory session's findings in the `semantic-drift` class open exactly this up to dispute: the tester counts it a defect, the developer counts it expected behavior, and the product side agrees with neither. In this case, escaped defect count is not a measurement, it is a disagreement. The next lesson takes up the definition of done and acceptance criteria: who writes the criterion, when it closes, and how many units of reopened work a criterion left ambiguous produces.