--- title: 'User Acceptance Testing' source: 'https://academia.sh/en/courses/testing-process/user-acceptance-testing' course: 'The Testing Process and Automation Infrastructure' language: en updated: '2026-08-23T14:25:20+00:00' license: 'CC BY-SA 4.0' --- # User Acceptance Testing Managing end-user verification as a distribution decision: splitting acceptance scope four ways across user tasks, separating the returning feedback into defect, request, and misunderstanding, each class's cost measured in minutes, and why feedback volume is not a quality measure. Every criterion in the previous lesson was written by parties inside the team. All three stand on the side designing the system and use the same vocabulary; yet the expectation the criterion must ultimately meet sits outside the team. How many steps a library clerk expects a loan transaction to take, or when a member finds a delay notification meaningful, was written into no slot. **User acceptance testing** has the end user answer this gap. This lesson builds it as a distribution decision: the resource distributed is the user's time, the distribution unit is the **user task**, and what comes back is not of one kind. Part of the feedback is a defect, part is a new **request**, and part is a **misunderstanding** that misreads the system's correct behavior — and the three cost differently. ## What Reaches the Acceptance Stage Acceptance testing is the last gate: a defect an earlier gate already saw does not reach here. The second lesson's nine budget-fitting items closed nine classes; the remaining five classes' 18 defects reach this stage. The fourth lesson's Y5 policy also left behind a state: twenty-nine numeric criteria have an empty limit. These two inheritances determine feedback's class. When a user observes a criterion, the outcome splits three ways: at **a criterion with an empty limit**, the user states their own limit, and this is a request; at **a complete criterion with a defect underneath**, they report a defect; at **a complete, flawless existence criterion whose measure is not visible to the user**, they compare what they see to their own expectation and produce a misunderstanding. **TP18 (assumption) — 20 work items are distributed across 12 user tasks, each task getting at least one item; usage share comes from seed 4712.** **TP19 (assumption) — the acceptance budget is 600 user minutes, observing one criterion takes 8 minutes**; the budget is thus 75 criterion observations, and does not cover all eighty criteria. **TP20 (assumption) — 30% of requests are accepted and turn into work**, the rest stay on record. **TP21 (assumption) — deciding on a request takes 30 minutes, doing an accepted request takes 120, resolving a misunderstanding takes 20 minutes; a defect's cost is the fix minutes in the defect set.** ```js // library/defects.mjs — the first lesson's defect set, unchanged (seed 20260731, 60 defects, // 14 classes). This lesson does not use the inventory's cost columns, only the defect set. const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4], ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4], ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3], ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]]; export function defects(seed, n) { let x = seed; const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; const total = WEIGHT.reduce((a, [, w]) => a + w, 0); const list = []; for (let i = 1; i <= n; i += 1) { let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0]; for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } } list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5), impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) }); } return list; } ``` ## Four Scope Distributions The same 600 minutes is split by four rules: equal (K1), by usage share (K2), toward the task with the most incomplete criteria (K3), and only toward the last four new tasks (K4). ```js // library/acceptance.mjs — distributes acceptance scope across user tasks and separates the // returning feedback into defect / request / misunderstanding. Input: library/defects.mjs import { defects } from "./defects.mjs"; const ITEMS = 20, PER_ITEM = 4, TASKS = 12, SEED = 20260731, N = 60, MULTIPLIER = 6; const BUDGET = 600, CRITERION_MIN = 8; // TP19: acceptance budget, user minutes per criterion let ACCEPT_RATE = 0.3; // TP20: rate at which a request converts into work const REQUEST_MIN = 30, MISUNDERSTANDING_MIN = 20, NEW_WORK_MIN = 120; // TP21: cost per class const v = (x, n = 1) => x.toFixed(n); const g = (x, n) => String(x).padStart(n); const makeRandom = (t) => { let x = t; return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; }; // The fourth lesson's criterion set, unchanged: 80 criteria, seed 6060, three types. const r = makeRandom(6060); const TYPE = ["existence", "numeric", "semantic"]; const CRITERIA = Array.from({ length: ITEMS * PER_ITEM }, (_, i) => { const p = r(); return { id: i, item: Math.floor(i / PER_ITEM), type: TYPE[p < 0.45 ? 0 : p < 0.8 ? 1 : 2] }; }); // Defects that reach the acceptance stage: the five classes the second lesson's nine // budget-fitting items miss, i.e. that lesson's 18 escaped defects. What the scripted set caught does not arrive here. const REACHED = new Set(["accessibility", "degraded-response", "recovery-gap", "off-scenario", "semantic-drift"]); const DEFECTS = defects(SEED, N).map((k, i) => ({ ...k, item: Math.floor(i / (N / ITEMS)) })) .filter((k) => REACHED.has(k.cls)); for (const k of DEFECTS.filter((k) => k.cls === "semantic-drift")) { const itemCriteria = CRITERIA.filter((o) => o.item === k.item); if (itemCriteria.some((o) => o.type === "semantic") === false) Object.assign(itemCriteria[Math.floor(r() * itemCriteria.length)], { type: "semantic" }); } // The state the fourth lesson's Y5 policy left behind: numeric criteria have an empty limit. for (const o of CRITERIA) o.complete = o.type !== "numeric"; // TP18: 20 work items are distributed across 12 user tasks (each task gets at least one item); // a task's usage share comes from the same seed. const r2 = makeRandom(4712); const SHARE = Array.from({ length: TASKS }, () => 1 + Math.floor(r2() * 9)); const SHARE_TOTAL = SHARE.reduce((a, x) => a + x, 0); const ITEM_TASK = Array.from({ length: ITEMS }, (_, i) => i % TASKS); for (let j = ITEMS - 1; j > 0; j -= 1) { const q = Math.floor(r2() * (j + 1)); [ITEM_TASK[j], ITEM_TASK[q]] = [ITEM_TASK[q], ITEM_TASK[j]]; } const taskOf = (o) => ITEM_TASK[o.item]; // Every defect sits under one criterion of its item; semantic drift under a semantic criterion. const r3 = makeRandom(9313); for (const k of DEFECTS) { const candidates = CRITERIA.filter((o) => o.item === k.item && (k.cls === "semantic-drift" ? o.type === "semantic" : true)); k.criterion = candidates[Math.floor(r3() * candidates.length)]; } const FLAWED = new Set(DEFECTS.map((k) => k.criterion.id)); // A task with m minutes: m/CRITERION_MIN criteria are observed. Feedback follows the criterion's // state: empty limit -> request (user states their own limit), flawed complete criterion -> defect, // flawless existence criterion -> misunderstanding (its measure is invisible to the user, who // compares it to their own expectation), flawless semantic criterion -> no feedback. function evaluate(distribution) { const findings = { defect: [], request: 0, misunderstanding: 0 }; let observed = 0, moment = 0, count = 0; for (let task = 0; task < TASKS; task += 1) { const list = CRITERIA.filter((o) => taskOf(o) === task); const seen = Math.min(list.length, Math.floor(distribution[task] / CRITERION_MIN)); observed += seen; list.slice(0, seen).forEach((o, i) => { const when = (i + 1) * CRITERION_MIN; if (o.complete === false) { findings.request += 1; moment += when; count += 1; } else if (FLAWED.has(o.id)) { const matches = DEFECTS.filter((k) => k.criterion.id === o.id); findings.defect.push(...matches); moment += when * matches.length; count += matches.length; } else if (o.type === "existence") { findings.misunderstanding += 1; moment += when; count += 1; } }); } const escaped = DEFECTS.filter((k) => findings.defect.includes(k) === false); // A found defect's fix is paid once, an escaped defect's fix is paid at the TP5 multiplier. const o = { observed, moment: moment / (count || 1), defect: findings.defect.length, request: findings.request, misunderstanding: findings.misunderstanding, escaped: escaped.length, escapedRisk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0), defectMin: findings.defect.reduce((a, k) => a + k.fix, 0), escapedMin: MULTIPLIER * escaped.reduce((a, k) => a + k.fix, 0), requestMin: REQUEST_MIN * findings.request + ACCEPT_RATE * findings.request * NEW_WORK_MIN, misunderstandingMin: MISUNDERSTANDING_MIN * findings.misunderstanding }; return { ...o, total: o.defectMin + o.escapedMin + o.requestMin + o.misunderstandingMin }; } const equal = () => Array.from({ length: TASKS }, () => BUDGET / TASKS); const byShare = () => SHARE.map((p) => (BUDGET * p) / SHARE_TOTAL); const byGap = () => { const e = Array.from({ length: TASKS }, (_, task) => CRITERIA.filter((o) => taskOf(o) === task && o.complete === false).length); const t = e.reduce((a, x) => a + x, 0); return e.map((x) => (BUDGET * x) / t); }; const newOnly = () => Array.from({ length: TASKS }, (_, task) => (task >= TASKS - 4 ? BUDGET / 4 : 0)); const D = [["K1 equal distribution", equal()], ["K2 by usage share", byShare()], ["K3 to most-incomplete criteria", byGap()], ["K4 only four new tasks", newOnly()]]; console.log(`${CRITERIA.length} criteria (seed 6060) distributed across ${TASKS} user tasks (seed 4712)`); console.log("criteria per task: " + Array.from({ length: TASKS }, (_, task) => CRITERIA.filter((o) => taskOf(o) === task).length).join(" ") + "; usage share: " + SHARE.map((p) => v((100 * p) / SHARE_TOTAL, 0) + "%").join(" ")); console.log(`the fourth lesson's Y5 state: ${CRITERIA.filter((o) => o.complete === false).length} criteria have an empty limit; ` + `${FLAWED.size} criteria have a defect underneath`); console.log(`\nacceptance budget TP19 = ${BUDGET} user min, ${CRITERION_MIN} min per criterion = ${BUDGET / CRITERION_MIN} criteria`); console.log("distribution observed defect request misunderstanding defect share find moment"); for (const [name, d] of D) { const o = evaluate(d); const t = o.defect + o.request + o.misunderstanding; console.log(name.padEnd(32) + g(o.observed, 8) + g(o.defect, 8) + g(o.request, 9) + g(o.misunderstanding, 19) + g(t ? v(o.defect / t, 2) : "-", 14) + g(v(o.moment), 13)); } console.log("\nclass costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21)"); console.log("distribution defect min request min misunderstanding min escaped escaped risk escaped min total min"); for (const [name, d] of D) { const o = evaluate(d); console.log(name.padEnd(32) + g(o.defectMin, 12) + g(v(o.requestMin, 0), 13) + g(o.misunderstandingMin, 23) + g(o.escaped, 9) + g(o.escapedRisk, 14) + g(o.escapedMin, 13) + g(v(o.total, 0), 11)); } console.log("\nTP20's sensitivity: what share of requests convert into work"); console.log("accept rate " + D.map(([name]) => g(name.slice(0, 2), 9)).join("") + " cheapest"); for (const p of [0, 0.3, 0.6]) { ACCEPT_RATE = p; const t = D.map(([, d]) => evaluate(d).total); console.log(g((100 * p) + "%", 11) + t.map((n) => g(v(n, 0), 9)).join("") + g(D[t.indexOf(Math.min(...t))][0].slice(0, 2), 10)); } ``` ``` 80 criteria (seed 6060) distributed across 12 user tasks (seed 4712) criteria per task: 8 8 8 8 8 8 8 8 4 4 4 4; usage share: 4% 8% 10% 12% 3% 10% 10% 3% 12% 12% 12% 6% the fourth lesson's Y5 state: 29 criteria have an empty limit; 15 criteria have a defect underneath acceptance budget TP19 = 600 user min, 8 min per criterion = 75 criteria distribution observed defect request misunderstanding defect share find moment K1 equal distribution 64 7 21 30 0.12 25.2 K2 by usage share 54 8 21 22 0.16 26.0 K3 to most-incomplete criteria 60 6 23 24 0.11 28.7 K4 only four new tasks 16 6 4 8 0.33 19.6 class costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21) distribution defect min request min misunderstanding min escaped escaped risk escaped min total min K1 equal distribution 675 1386 600 11 88 3510 6171 K2 by usage share 720 1386 440 10 82 3240 5786 K3 to most-incomplete criteria 555 1518 480 12 102 4230 6783 K4 only four new tasks 570 264 160 12 89 4140 5134 TP20's sensitivity: what share of requests convert into work accept rate K1 K2 K3 K4 cheapest 0% 5415 5030 5955 4990 K4 30% 6171 5786 6783 5134 K4 60% 6927 6542 7611 5278 K4 ``` ## Feedback Volume Is Not a Quality Measure The **defect share** of the feedback the four distributions produce is between 0.11 and 0.33. That is, in the best case one report in three, in the worst case one in nine, is an actual defect; the rest is request and misunderstanding. K1 produces fifty-eight reports, and only seven of these are defects. Summarizing an acceptance round as "fifty-eight findings" does not tell the team it is talking about seven defects. Without classification this distinction is invisible: all three classes fall into the same list and get written the same way. Tying the distinction to the criterion takes it out of being a judgment call: feedback arriving at a criterion with an empty limit is a request, because there is no unmet promise — the user's limit is being stated there for the first time. ## Which Distribution Buys What K3 is the distribution that brings the most requests (23). This is not surprising: targeting incomplete criteria has users observe, by definition, criteria whose limit has not been written. Its escaped defect is also highest (12), and its total is the most expensive (6,783 minutes). **Targeting uncertainty finds requests, not defects.** K2 distributes by usage share, finds the most defects (8), and lets the fewest escape (10, escaped risk 82). A defect sitting in a task the user uses heavily gets found when tested. Against that, K2 is lowest in the number of criteria it manages to observe (54), because the bulk of the budget goes to a small number of tasks. K4 looks cheapest (5,134 minutes) and has the highest defect share (0.33). The reason is not an advantage: because it observes only sixteen criteria, it also produces no noise. Request and misunderstanding cost stays at 424 minutes, but it lets twelve defects escape. **The total cost column alone is misleading, because producing no feedback is cheap.** The two columns must be read together: K2 is cheap on escaped defect, K4 is cheap on processed feedback, and the two do not measure the same thing. The TP20 scan confirms this. Even if none of the requests turn into work (acceptance rate 0%), the ranking does not change; raised to 60%, K4 pulls further ahead, because it already has few requests to process. What actually determines an acceptance round's cost is not the number of defects found but the **number of processed reports.** ## The Return on the Decision Acceptance testing's red is not a single decision, it is three separate queues. A defect stops the release and goes to a fix. A request goes to the product side's decision, and 30% of it turns into work; the rest stays on record. A misunderstanding goes not to the code but to interface copy or documentation, and feeds back into the fourth lesson's criterion slots — a misunderstanding is the mark of a criterion whose measure is not visible to the user. The finding's average moment is between 19.6 and 28.7 minutes, but this number is deceptively small: the acceptance window opens **after** development is finished. The second lesson's scripted set reported a defect at an average of 18.2 minutes, the third lesson's exploratory session at 74.8 minutes; acceptance testing's finding, though, only surfaces once the 600-minute window opens. Along the same defect chain, the reporting moment grows, and its cost grows with it. ## Summary - Only defects the earlier gates did not see reach the acceptance stage: 18 defects from the second lesson's five unclosed classes. - Feedback is three classes, and the criterion's state determines the class: an empty-limit criterion produces a request, a flawed complete criterion produces a defect, a flawless criterion whose measure is invisible to the user produces a misunderstanding. - Defect share is between 0.11 and 0.33; summarizing an acceptance round by finding count does not tell the team how many defects were found. - The distribution targeting uncertainty produces the most requests and the most escaped defect (23 requests, 12 escaped, 6,783 minutes); the distribution by usage share finds the most defects (8) and escapes the fewest. - The distribution with the lowest total cost is the one that produces the fewest reports (K4, 5,134 minutes, 12 escaped); producing no feedback is cheap, so total cost is not read alone. - Acceptance testing's red goes to three separate queues: fix, product decision, and interface copy; a misunderstanding feeds back into the criterion slots. ## Next Step This topic has left four decisions behind. The plan wrote down scope, criteria, and responsibility; the risk order split the same budget by likelihood and impact; the exploratory session reached, through depth instead of width, the class the scripted set cannot see; the acceptance criterion and user acceptance testing carried the definition of done outside the team. What the four share is this: all of them are a distribution **on paper.** A plan is written but does not run; a risk order is a list, but a change can still be shipped without anyone looking at it; a charter opens only if someone decides to sit down; an acceptance criterion closes only if someone reads it. For this distribution to happen on its own, on every change, without depending on anyone remembering — a mechanism is needed, and that mechanism was never built. The Automation Infrastructure topic starts from exactly this point.