User Acceptance Testing

Every criterion in the previous lesson was written by parties inside the team. All three stand on the side designing the system and use the same vocabulary; yet the expectation the criterion must ultimately meet sits outside the team. How many steps a library clerk expects a loan transaction to take, or when a member finds a delay notification meaningful, was written into no slot.

User acceptance testing has the end user answer this gap. This lesson builds it as a distribution decision: the resource distributed is the user’s time, the distribution unit is the user task, and what comes back is not of one kind. Part of the feedback is a defect, part is a new request, and part is a misunderstanding that misreads the system’s correct behavior — and the three cost differently.

What Reaches the Acceptance Stage

Acceptance testing is the last gate: a defect an earlier gate already saw does not reach here. The second lesson’s nine budget-fitting items closed nine classes; the remaining five classes’ 18 defects reach this stage. The fourth lesson’s Y5 policy also left behind a state: twenty-nine numeric criteria have an empty limit.

These two inheritances determine feedback’s class. When a user observes a criterion, the outcome splits three ways: at a criterion with an empty limit, the user states their own limit, and this is a request; at a complete criterion with a defect underneath, they report a defect; at a complete, flawless existence criterion whose measure is not visible to the user, they compare what they see to their own expectation and produce a misunderstanding.

TP18 (assumption) — 20 work items are distributed across 12 user tasks, each task getting at least one item; usage share comes from seed 4712. TP19 (assumption) — the acceptance budget is 600 user minutes, observing one criterion takes 8 minutes; the budget is thus 75 criterion observations, and does not cover all eighty criteria. TP20 (assumption) — 30% of requests are accepted and turn into work, the rest stay on record. TP21 (assumption) — deciding on a request takes 30 minutes, doing an accepted request takes 120, resolving a misunderstanding takes 20 minutes; a defect’s cost is the fix minutes in the defect set.

// library/defects.mjs — the first lesson's defect set, unchanged (seed 20260731, 60 defects,
// 14 classes). This lesson does not use the inventory's cost columns, only the defect set.
const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4],
  ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4],
  ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3],
  ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]];
export function defects(seed, n) {
  let x = seed;
  const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; };
  const total = WEIGHT.reduce((a, [, w]) => a + w, 0);
  const list = [];
  for (let i = 1; i <= n; i += 1) {
    let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0];
    for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } }
    list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5),
      impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) });
  }
  return list;
}

Four Scope Distributions

The same 600 minutes is split by four rules: equal (K1), by usage share (K2), toward the task with the most incomplete criteria (K3), and only toward the last four new tasks (K4).

// library/acceptance.mjs — distributes acceptance scope across user tasks and separates the
// returning feedback into defect / request / misunderstanding. Input: library/defects.mjs
import { defects } from "./defects.mjs";

const ITEMS = 20, PER_ITEM = 4, TASKS = 12, SEED = 20260731, N = 60, MULTIPLIER = 6;
const BUDGET = 600, CRITERION_MIN = 8;   // TP19: acceptance budget, user minutes per criterion
let ACCEPT_RATE = 0.3;                   // TP20: rate at which a request converts into work
const REQUEST_MIN = 30, MISUNDERSTANDING_MIN = 20, NEW_WORK_MIN = 120;   // TP21: cost per class
const v = (x, n = 1) => x.toFixed(n);
const g = (x, n) => String(x).padStart(n);
const makeRandom = (t) => { let x = t;
  return () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; }; };

// The fourth lesson's criterion set, unchanged: 80 criteria, seed 6060, three types.
const r = makeRandom(6060);
const TYPE = ["existence", "numeric", "semantic"];
const CRITERIA = Array.from({ length: ITEMS * PER_ITEM }, (_, i) => { const p = r();
  return { id: i, item: Math.floor(i / PER_ITEM), type: TYPE[p < 0.45 ? 0 : p < 0.8 ? 1 : 2] }; });
// Defects that reach the acceptance stage: the five classes the second lesson's nine
// budget-fitting items miss, i.e. that lesson's 18 escaped defects. What the scripted set caught does not arrive here.
const REACHED = new Set(["accessibility", "degraded-response", "recovery-gap",
  "off-scenario", "semantic-drift"]);
const DEFECTS = defects(SEED, N).map((k, i) => ({ ...k, item: Math.floor(i / (N / ITEMS)) }))
  .filter((k) => REACHED.has(k.cls));
for (const k of DEFECTS.filter((k) => k.cls === "semantic-drift")) {
  const itemCriteria = CRITERIA.filter((o) => o.item === k.item);
  if (itemCriteria.some((o) => o.type === "semantic") === false)
    Object.assign(itemCriteria[Math.floor(r() * itemCriteria.length)], { type: "semantic" });
}
// The state the fourth lesson's Y5 policy left behind: numeric criteria have an empty limit.
for (const o of CRITERIA) o.complete = o.type !== "numeric";

// TP18: 20 work items are distributed across 12 user tasks (each task gets at least one item);
// a task's usage share comes from the same seed.
const r2 = makeRandom(4712);
const SHARE = Array.from({ length: TASKS }, () => 1 + Math.floor(r2() * 9));
const SHARE_TOTAL = SHARE.reduce((a, x) => a + x, 0);
const ITEM_TASK = Array.from({ length: ITEMS }, (_, i) => i % TASKS);
for (let j = ITEMS - 1; j > 0; j -= 1) { const q = Math.floor(r2() * (j + 1));
  [ITEM_TASK[j], ITEM_TASK[q]] = [ITEM_TASK[q], ITEM_TASK[j]]; }
const taskOf = (o) => ITEM_TASK[o.item];
// Every defect sits under one criterion of its item; semantic drift under a semantic criterion.
const r3 = makeRandom(9313);
for (const k of DEFECTS) {
  const candidates = CRITERIA.filter((o) => o.item === k.item &&
    (k.cls === "semantic-drift" ? o.type === "semantic" : true));
  k.criterion = candidates[Math.floor(r3() * candidates.length)];
}
const FLAWED = new Set(DEFECTS.map((k) => k.criterion.id));

// A task with m minutes: m/CRITERION_MIN criteria are observed. Feedback follows the criterion's
// state: empty limit -> request (user states their own limit), flawed complete criterion -> defect,
// flawless existence criterion -> misunderstanding (its measure is invisible to the user, who
// compares it to their own expectation), flawless semantic criterion -> no feedback.
function evaluate(distribution) {
  const findings = { defect: [], request: 0, misunderstanding: 0 };
  let observed = 0, moment = 0, count = 0;
  for (let task = 0; task < TASKS; task += 1) {
    const list = CRITERIA.filter((o) => taskOf(o) === task);
    const seen = Math.min(list.length, Math.floor(distribution[task] / CRITERION_MIN));
    observed += seen;
    list.slice(0, seen).forEach((o, i) => {
      const when = (i + 1) * CRITERION_MIN;
      if (o.complete === false) { findings.request += 1; moment += when; count += 1; }
      else if (FLAWED.has(o.id)) { const matches = DEFECTS.filter((k) => k.criterion.id === o.id);
        findings.defect.push(...matches); moment += when * matches.length; count += matches.length; }
      else if (o.type === "existence") { findings.misunderstanding += 1; moment += when; count += 1; }
    });
  }
  const escaped = DEFECTS.filter((k) => findings.defect.includes(k) === false);
  // A found defect's fix is paid once, an escaped defect's fix is paid at the TP5 multiplier.
  const o = { observed, moment: moment / (count || 1), defect: findings.defect.length, request: findings.request,
    misunderstanding: findings.misunderstanding, escaped: escaped.length,
    escapedRisk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0),
    defectMin: findings.defect.reduce((a, k) => a + k.fix, 0),
    escapedMin: MULTIPLIER * escaped.reduce((a, k) => a + k.fix, 0),
    requestMin: REQUEST_MIN * findings.request + ACCEPT_RATE * findings.request * NEW_WORK_MIN,
    misunderstandingMin: MISUNDERSTANDING_MIN * findings.misunderstanding };
  return { ...o, total: o.defectMin + o.escapedMin + o.requestMin + o.misunderstandingMin };
}
const equal = () => Array.from({ length: TASKS }, () => BUDGET / TASKS);
const byShare = () => SHARE.map((p) => (BUDGET * p) / SHARE_TOTAL);
const byGap = () => { const e = Array.from({ length: TASKS }, (_, task) =>
  CRITERIA.filter((o) => taskOf(o) === task && o.complete === false).length);
  const t = e.reduce((a, x) => a + x, 0); return e.map((x) => (BUDGET * x) / t); };
const newOnly = () => Array.from({ length: TASKS }, (_, task) => (task >= TASKS - 4 ? BUDGET / 4 : 0));
const D = [["K1 equal distribution", equal()], ["K2 by usage share", byShare()],
  ["K3 to most-incomplete criteria", byGap()], ["K4 only four new tasks", newOnly()]];

console.log(`${CRITERIA.length} criteria (seed 6060) distributed across ${TASKS} user tasks (seed 4712)`);
console.log("criteria per task: " + Array.from({ length: TASKS }, (_, task) =>
  CRITERIA.filter((o) => taskOf(o) === task).length).join(" ") +
  "; usage share: " + SHARE.map((p) => v((100 * p) / SHARE_TOTAL, 0) + "%").join(" "));
console.log(`the fourth lesson's Y5 state: ${CRITERIA.filter((o) => o.complete === false).length} criteria have an empty limit; ` +
  `${FLAWED.size} criteria have a defect underneath`);

console.log(`\nacceptance budget TP19 = ${BUDGET} user min, ${CRITERION_MIN} min per criterion = ${BUDGET / CRITERION_MIN} criteria`);
console.log("distribution                    observed  defect  request   misunderstanding  defect share  find moment");
for (const [name, d] of D) { const o = evaluate(d);
  const t = o.defect + o.request + o.misunderstanding;
  console.log(name.padEnd(32) + g(o.observed, 8) + g(o.defect, 8) + g(o.request, 9) + g(o.misunderstanding, 19) +
    g(t ? v(o.defect / t, 2) : "-", 14) + g(v(o.moment), 13)); }

console.log("\nclass costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21)");
console.log("distribution                      defect min  request min   misunderstanding min  escaped  escaped risk  escaped min  total min");
for (const [name, d] of D) { const o = evaluate(d);
  console.log(name.padEnd(32) + g(o.defectMin, 12) + g(v(o.requestMin, 0), 13) + g(o.misunderstandingMin, 23) +
    g(o.escaped, 9) + g(o.escapedRisk, 14) + g(o.escapedMin, 13) + g(v(o.total, 0), 11)); }

console.log("\nTP20's sensitivity: what share of requests convert into work");
console.log("accept rate   " + D.map(([name]) => g(name.slice(0, 2), 9)).join("") + "   cheapest");
for (const p of [0, 0.3, 0.6]) {
  ACCEPT_RATE = p;
  const t = D.map(([, d]) => evaluate(d).total);
  console.log(g((100 * p) + "%", 11) + t.map((n) => g(v(n, 0), 9)).join("") +
    g(D[t.indexOf(Math.min(...t))][0].slice(0, 2), 10));
}

80 criteria (seed 6060) distributed across 12 user tasks (seed 4712)
criteria per task: 8 8 8 8 8 8 8 8 4 4 4 4; usage share: 4% 8% 10% 12% 3% 10% 10% 3% 12% 12% 12% 6%
the fourth lesson's Y5 state: 29 criteria have an empty limit; 15 criteria have a defect underneath

acceptance budget TP19 = 600 user min, 8 min per criterion = 75 criteria
distribution                    observed  defect  request   misunderstanding  defect share  find moment
K1 equal distribution                 64       7       21                 30          0.12         25.2
K2 by usage share                     54       8       21                 22          0.16         26.0
K3 to most-incomplete criteria        60       6       23                 24          0.11         28.7
K4 only four new tasks                16       6        4                  8          0.33         19.6

class costs — request 30 min decision + 30% at 120 min work, misunderstanding 20 min (TP21)
distribution                      defect min  request min   misunderstanding min  escaped  escaped risk  escaped min  total min
K1 equal distribution                    675         1386                    600       11            88         3510       6171
K2 by usage share                        720         1386                    440       10            82         3240       5786
K3 to most-incomplete criteria           555         1518                    480       12           102         4230       6783
K4 only four new tasks                   570          264                    160       12            89         4140       5134

TP20's sensitivity: what share of requests convert into work
accept rate          K1       K2       K3       K4   cheapest
         0%     5415     5030     5955     4990        K4
        30%     6171     5786     6783     5134        K4
        60%     6927     6542     7611     5278        K4

Feedback Volume Is Not a Quality Measure

The defect share of the feedback the four distributions produce is between 0.11 and 0.33. That is, in the best case one report in three, in the worst case one in nine, is an actual defect; the rest is request and misunderstanding. K1 produces fifty-eight reports, and only seven of these are defects. Summarizing an acceptance round as “fifty-eight findings” does not tell the team it is talking about seven defects.

Without classification this distinction is invisible: all three classes fall into the same list and get written the same way. Tying the distinction to the criterion takes it out of being a judgment call: feedback arriving at a criterion with an empty limit is a request, because there is no unmet promise — the user’s limit is being stated there for the first time.

Which Distribution Buys What

K3 is the distribution that brings the most requests (23). This is not surprising: targeting incomplete criteria has users observe, by definition, criteria whose limit has not been written. Its escaped defect is also highest (12), and its total is the most expensive (6,783 minutes). Targeting uncertainty finds requests, not defects.

K2 distributes by usage share, finds the most defects (8), and lets the fewest escape (10, escaped risk 82). A defect sitting in a task the user uses heavily gets found when tested. Against that, K2 is lowest in the number of criteria it manages to observe (54), because the bulk of the budget goes to a small number of tasks.

K4 looks cheapest (5,134 minutes) and has the highest defect share (0.33). The reason is not an advantage: because it observes only sixteen criteria, it also produces no noise. Request and misunderstanding cost stays at 424 minutes, but it lets twelve defects escape. The total cost column alone is misleading, because producing no feedback is cheap. The two columns must be read together: K2 is cheap on escaped defect, K4 is cheap on processed feedback, and the two do not measure the same thing.

The TP20 scan confirms this. Even if none of the requests turn into work (acceptance rate 0%), the ranking does not change; raised to 60%, K4 pulls further ahead, because it already has few requests to process. What actually determines an acceptance round’s cost is not the number of defects found but the number of processed reports.

The Return on the Decision

Acceptance testing’s red is not a single decision, it is three separate queues. A defect stops the release and goes to a fix. A request goes to the product side’s decision, and 30% of it turns into work; the rest stays on record. A misunderstanding goes not to the code but to interface copy or documentation, and feeds back into the fourth lesson’s criterion slots — a misunderstanding is the mark of a criterion whose measure is not visible to the user.

The finding’s average moment is between 19.6 and 28.7 minutes, but this number is deceptively small: the acceptance window opens after development is finished. The second lesson’s scripted set reported a defect at an average of 18.2 minutes, the third lesson’s exploratory session at 74.8 minutes; acceptance testing’s finding, though, only surfaces once the 600-minute window opens. Along the same defect chain, the reporting moment grows, and its cost grows with it.

Summary

• Only defects the earlier gates did not see reach the acceptance stage: 18 defects from the second lesson’s five unclosed classes.
• Feedback is three classes, and the criterion’s state determines the class: an empty-limit criterion produces a request, a flawed complete criterion produces a defect, a flawless criterion whose measure is invisible to the user produces a misunderstanding.
• Defect share is between 0.11 and 0.33; summarizing an acceptance round by finding count does not tell the team how many defects were found.
• The distribution targeting uncertainty produces the most requests and the most escaped defect (23 requests, 12 escaped, 6,783 minutes); the distribution by usage share finds the most defects (8) and escapes the fewest.
• The distribution with the lowest total cost is the one that produces the fewest reports (K4, 5,134 minutes, 12 escaped); producing no feedback is cheap, so total cost is not read alone.
• Acceptance testing’s red goes to three separate queues: fix, product decision, and interface copy; a misunderstanding feeds back into the criterion slots.

Next Step

This topic has left four decisions behind. The plan wrote down scope, criteria, and responsibility; the risk order split the same budget by likelihood and impact; the exploratory session reached, through depth instead of width, the class the scripted set cannot see; the acceptance criterion and user acceptance testing carried the definition of done outside the team. What the four share is this: all of them are a distribution on paper. A plan is written but does not run; a risk order is a list, but a change can still be shipped without anyone looking at it; a charter opens only if someone decides to sit down; an acceptance criterion closes only if someone reads it. For this distribution to happen on its own, on every change, without depending on anyone remembering — a mechanism is needed, and that mechanism was never built. The Automation Infrastructure topic starts from exactly this point.