Acceptance Criteria

The previous lesson’s exploratory session produced findings in the semantic-drift class, but whether these findings were defects was not a measurement question: the tester counts a defect, the developer counts expected behavior, and the product side agrees with neither. In a case like this, escaped defect count is not a measure, it is a disagreement.

What turns disagreement into a measure is the acceptance criterion: a condition written in advance that says when a work item is done. Turning an acceptance criterion into an automated test was built in the Unit Testing and Test-Driven Development course; the question here is different — who writes the criterion and when it closes. This lesson builds those two questions as a budget distribution; the resource distributed is the product side’s attention, and it is measured by rework, meaning work that is opened again. (In the Resilience and Reliability course, a failback is returning to the primary node after a failover; what is meant here by rework is a piece of work counted as closed being reopened.)

The Criterion’s Three Slots

An acceptance criterion fills three slots: subject says what is observed, measure says how it is observed, limit says what value is accepted. A definition of done is the state where all of these criteria are met. Not every criterion needs all three slots, and which are required comes from the criterion’s type: an existence criterion needs subject and measure (“a delay notification is sent”), a numeric criterion also needs a limit (“five days after the due date”), a semantic criterion needs a limit and requires that limit not be derived from observation (“the notification goes to the member who holds the loan”).

The last item is this lesson’s sharpest point. If a criterion is written at closeout by looking at the system’s observed behavior, the limit is set equal to that behavior; the criterion never turns red, because what it tests is its own source.

TP15 (assumption) — 20 work items, 4 acceptance criteria per item, 80 criteria in total; the type distribution comes from seed 6060. The defect set’s 60 defects are distributed to items three at a time. TP16 (assumption) — the time the product side can set aside for writing criteria across the horizon is 300 minutes; reviewing one criterion takes 12 minutes. This is the distributed resource, and its sensitivity is scanned. TP17 (assumption) — a piece of rework’s cost depends on the phase where it surfaces: 5 minutes at write time, 20 during development, 60 at acceptance review, 180 after release.

// library/defects.mjs — the first lesson's defect set, unchanged (seed 20260731, 60 defects,
// 14 classes). This lesson does not use the inventory's cost columns, only the defect set.
const WEIGHT = [["rule-boundary", 9], ["schema-mismatch", 6], ["migration-data-loss", 4],
  ["contract-field", 5], ["interface-state", 8], ["visual-deviation", 3], ["data-growth", 4],
  ["sql-concatenation", 3], ["known-vulnerability", 3], ["accessibility", 4], ["degraded-response", 3],
  ["recovery-gap", 2], ["off-scenario", 5], ["semantic-drift", 4]];
export function defects(seed, n) {
  let x = seed;
  const random = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; };
  const total = WEIGHT.reduce((a, [, w]) => a + w, 0);
  const list = [];
  for (let i = 1; i <= n; i += 1) {
    let p = random() * total, cls = WEIGHT[WEIGHT.length - 1][0];
    for (const [s, w] of WEIGHT) { p -= w; if (p < 0) { cls = s; break; } }
    list.push({ id: i, cls, likelihood: 1 + Math.floor(random() * 5),
      impact: 1 + Math.floor(random() * 5), fix: 15 + 15 * Math.floor(random() * 8) });
  }
  return list;
}

Five Policies

Five policies distribute the criterion to different hands and different moments. A slot left missing surfaces at the phase when the side that will fill it first looks at the criterion: the product side fills a missing limit, the tester fills a missing measure.

// library/criteria.mjs — builds the acceptance criterion as a three-slot data structure and
// measures the reopened-work return on who writes it and when. Input: library/defects.mjs
import { defects } from "./defects.mjs";

const ITEMS = 20, PER_ITEM = 4, SEED = 20260731, N = 60, PRODUCT_MIN = 12, PRODUCT_BUDGET = 300;
const COST = [5, 20, 60, 180];        // TP17: reopened work's minutes per phase
const PHASE = ["write time", "development", "acceptance review", "after release"];
let MULTIPLIER = 6;                   // TP5: escaped defect's fix multiplier
const g = (x, n) => String(x).padStart(n);

// TP15: 20 work items x 4 acceptance criteria = 80 criteria. Criterion type sets required slots:
// existence (subject+measure), numeric (subject+measure+limit), semantic (all three; limit not from observation).
let x = 6060;
const r = () => { x = (1103515245 * x + 12345) % 2147483648; return x / 2147483648; };
const TYPE = ["existence", "numeric", "semantic"];
const CRITERIA = Array.from({ length: ITEMS * PER_ITEM }, (_, i) => { const p = r();
  return { id: i, item: Math.floor(i / PER_ITEM), type: TYPE[p < 0.45 ? 0 : p < 0.8 ? 1 : 2] }; });
const REQUIRED = { existence: ["subject", "measure"], numeric: ["subject", "measure", "limit"],
  semantic: ["subject", "measure", "limit"] };

// Defects are distributed to items three at a time. A semantic-drift defect by definition sits
// under a semantic criterion — what makes it a defect is not a rule in the code but what the
// criterion says; if the item has no semantic criterion, one of its criteria is marked semantic.
const DEFECTS = defects(SEED, N).map((k, i) => ({ ...k, item: Math.floor(i / (N / ITEMS)) }));
const SEMANTIC = DEFECTS.filter((k) => k.cls === "semantic-drift");
for (const k of SEMANTIC) {
  const itemCriteria = CRITERIA.filter((o) => o.item === k.item);
  k.criterion = itemCriteria.find((o) => o.type === "semantic")
    || Object.assign(itemCriteria[Math.floor(r() * itemCriteria.length)], { type: "semantic" });
}

// Policy: which slots are filled for each criterion, whether product fills the limit slot, whether
// the limit is derived from observation, and the phase at which the side covering the gap first reads it.
function evaluate(name, slots, productReviews, fromObservation, productPhase, testerPhase, writeMin) {
  const status = CRITERIA.map((o) => {
    const filled = slots(o);
    const missing = REQUIRED[o.type].filter((y) => filled.includes(y) === false);
    const judges = missing.length === 0 && (o.type !== "semantic" || fromObservation === false);
    return { ...o, judges, hasLimit: filled.includes("limit"),
      phase: missing.length ? productPhase : judges ? -1 : testerPhase };
  });
  const open = status.filter((o) => o.judges === false);
  const escaped = SEMANTIC.filter((k) => status[k.criterion.id].judges === false);
  const productHour = productReviews ? PRODUCT_MIN * status.filter((o) => o.hasLimit).length : 0;
  return { name, productHour, writeMin: writeMin * CRITERIA.length + productHour,
    reopened: open.length, item: new Set(open.map((o) => o.item)).size,
    reworkMin: open.reduce((a, o) => a + COST[o.phase], 0),
    distribution: [0, 1, 2, 3].map((s) => open.filter((o) => o.phase === s).length),
    escaped: escaped.length, risk: escaped.reduce((a, k) => a + k.likelihood * k.impact, 0),
    fix: MULTIPLIER * escaped.reduce((a, k) => a + k.fix, 0) };
}
const TWO = ["subject", "measure"], THREE = ["subject", "measure", "limit"];
const POLICIES = () => [evaluate("Y1 developer writes, product does not review", () => TWO, false, false, 2, 2, 5),
  evaluate("Y2 product reviews every criterion, upfront", () => THREE, true, false, 0, 1, 5),
  evaluate("Y3 tester writes at closeout", () => THREE, false, true, 3, 2, 8),
  evaluate("Y4 no criteria, one-line definition", () => [], false, false, 3, 3, 1),
  evaluate("Y5 product reviews only semantic criteria", (o) => (o.type === "semantic" ? THREE : TWO),
    true, false, 2, 2, 5)];
const total = (o) => o.writeMin + o.reworkMin + o.fix;
const fitsBudget = (t) => { const u = POLICIES().map((o, i) => [t[i], o.productHour <= PRODUCT_BUDGET, i])
  .filter(([, s]) => s); return "Y" + (u.sort((a, b) => a[0] - b[0])[0][2] + 1); };

console.log(`${ITEMS} work items x ${PER_ITEM} acceptance criteria = ${CRITERIA.length} criteria (seed 6060); type: ` +
  TYPE.map((t) => `${t} ${CRITERIA.filter((o) => o.type === t).length}`).join(", "));
console.log(`defect set seed ${SEED}, ${N} defects; ${SEMANTIC.length} semantic drift, ` +
  `${SEMANTIC.reduce((a, k) => a + k.likelihood * k.impact, 0)} risk points; ` +
  `across ${new Set(SEMANTIC.map((k) => k.item)).size} separate work items`);
console.log(`TP16 product budget ${PRODUCT_BUDGET} min (${PRODUCT_MIN} min per criterion); ` +
  `TP17 rework: ` + PHASE.map((a, i) => `${a} ${COST[i]} min`).join(", "));

console.log("\npolicy                                           product min  budget  reopened  item  discovery phase");
for (const o of POLICIES()) console.log(o.name.padEnd(46) + g(o.productHour, 14) +
  g(o.productHour <= PRODUCT_BUDGET ? "fits" : "over", 8) + g(o.reopened, 10) + g(o.item, 6) +
  "  " + (o.reopened ? PHASE[o.distribution.indexOf(Math.max(...o.distribution))] : "-"));

console.log("\ncost on the same defect set");
console.log("policy                                          write min   rework min  escaped  escaped risk  fix min  total min");
for (const o of POLICIES()) console.log(o.name.padEnd(46) + g(o.writeMin, 11) + g(o.reworkMin, 13) +
  g(o.escaped, 9) + g(o.risk, 14) + g(o.fix, 9) + g(total(o), 11));

console.log("\nTP16's sensitivity to product budget — product reviews semantic criteria first, then numeric ones");
console.log("product min   criteria reviewed   reopened   escaped   total min");
const order = [...CRITERIA].sort((a, b) => TYPE.indexOf(b.type) - TYPE.indexOf(a.type));
for (const b of [0, 204, 300, 960]) {
  const covered = new Set(order.slice(0, Math.floor(b / PRODUCT_MIN)).map((o) => o.id));
  const o = evaluate("", (t) => (covered.has(t.id) ? THREE : TWO), true, false, 2, 2, 5);
  console.log(g(b, 7) + g(covered.size, 16) + g(o.reopened, 13) + g(o.escaped, 8) + g(total(o), 12));
}

console.log("\nTP5's sensitivity: which policy is cheapest as the escaped defect multiplier changes");
console.log("mult  " + ["Y1", "Y2", "Y3", "Y4", "Y5"].map((a) => g(a, 8)).join("") +
  "   cheapest   cheapest within budget");
for (const mult of [1, 6, 20]) {
  MULTIPLIER = mult;
  const t = POLICIES().map(total);
  console.log(g(mult, 3) + t.map((n) => g(n, 8)).join("") +
    g("Y" + (t.indexOf(Math.min(...t)) + 1), 10) + g(fitsBudget(t), 22));
}

20 work items x 4 acceptance criteria = 80 criteria (seed 6060); type: existence 34, numeric 29, semantic 17
defect set seed 20260731, 60 defects; 6 semantic drift, 60 risk points; across 5 separate work items
TP16 product budget 300 min (12 min per criterion); TP17 rework: write time 5 min, development 20 min, acceptance review 60 min, after release 180 min

policy                                           product min  budget  reopened  item  discovery phase
Y1 developer writes, product does not review               0    fits        46    20  acceptance review
Y2 product reviews every criterion, upfront              960    over         0     0  -
Y3 tester writes at closeout                               0    fits        17    14  acceptance review
Y4 no criteria, one-line definition                        0    fits        80    20  after release
Y5 product reviews only semantic criteria                204    fits        29    19  acceptance review

cost on the same defect set
policy                                          write min   rework min  escaped  escaped risk  fix min  total min
Y1 developer writes, product does not review          400         2760        6            60     2970       6130
Y2 product reviews every criterion, upfront          1360            0        0             0        0       1360
Y3 tester writes at closeout                          640         1020        6            60     2970       4630
Y4 no criteria, one-line definition                    80        14400        6            60     2970      17450
Y5 product reviews only semantic criteria             604         1740        0             0        0       2344

TP16's sensitivity to product budget — product reviews semantic criteria first, then numeric ones
product min   criteria reviewed   reopened   escaped   total min
      0               0           46       6        6130
    204              17           29       0        2344
    300              25           21       0        1960
    960              80            0       0        1360

TP5's sensitivity: which policy is cheapest as the escaped defect multiplier changes
mult        Y1      Y2      Y3      Y4      Y5   cheapest   cheapest within budget
  1    3655    1360    2155   14975    2344        Y2                    Y3
  6    6130    1360    4630   17450    2344        Y2                    Y5
 20   13060    1360   11560   24380    2344        Y2                    Y5

The Cheapest Policy on Paper Is Not in the Budget

Y2 writes every criterion together with the product side and produces no rework in the table: at 1,360 minutes it is the cheapest of the five policies. But the product column says 960 minutes, and TP16’s budget is 300 — the policy cannot be carried out. The best of the five rows is not an option; it is a ceiling.

Among those that fit the budget, Y5 stands out: the product side reviews only the seventeen semantic criteria, spends 204 minutes, and drops escaped defect to zero. Y1 uses none of that budget and lets six defects through to release; its total cost is 6,130 minutes, 2.6 times Y5’s. The difference is where 204 minutes of attention gets placed.

The product budget scan gives the shape of this distribution. The first 204 minutes remove all of the escaped defect (6 → 0), because those minutes go to semantic criteria. The remaining 756 minutes save no further defect; they only drop rework from 29 to 0 and pull the total from 2,344 to 1,360. The same resource’s first part and last part buy different things: the first part buys escaped defect, the last part buys rework.

The Criterion Written at Closeout

Y3 looks reasonable in the table: it spends no product budget, produces only seventeen rework items, and at 4,630 minutes is cheaper than Y1. Yet it still lets six defects through to release. The reason is not that the criterion is missing — all three of its slots are filled. The reason is that the limit is derived from observation: the tester writes the criterion at closeout by watching the running system, and records what the system does as acceptable. The criterion does not turn red, because it is derived from the very behavior it tests. The number of criteria written is not a quality indicator; the criterion’s source is the indicator.

Y4 writes no criteria at all and leaves all eighty of the eighty criteria to be disputed after release: 14,400 minutes of rework, all twenty of the twenty items reopened. The discovery phase column is the one thing that separates this policy from the others — not the count, the moment. The same disagreement costs 60 minutes at acceptance review, 180 minutes after release.

The Return on the Decision

In this lesson, the return on the decision is a directly measured quantity. Y1 reopens all twenty of the twenty items, Y5 nineteen of them, Y3 fourteen, Y2 none. The number of reopened items changes little across policies; what changes is which phase the reopening happens at, and how many defects reach release.

The TP5 scan ties the decision to a context. When an escaped defect’s fix cost equals a caught one’s (multiplier 1), the cheapest policy that fits the budget is Y3 — a criterion written at closeout is defensible while defects are cheap. At multipliers 6 and 20, Y5 wins and widens the gap. So the answer to who writes the criterion depends on what an escaped defect costs the team; the policy cannot be chosen without knowing that number.

Summary

• An acceptance criterion is three slots: subject, measure, limit. The criterion’s type determines which slots are required; a semantic criterion additionally requires the limit not be derived from observation.
• The product side’s attention is a limited resource (TP16, 300 minutes); the policy that reviews every criterion asks for 960 minutes and does not fit the budget — the cheapest row in the table is not an option, it is a ceiling.
• The same budget’s first 204 minutes remove all escaped defect, the remaining 756 minutes only lower rework; a resource’s first and last parts buy different things.
• A criterion written at closeout fills all three slots but never turns red because it takes its limit from observed behavior, and it lets six defects through to release.
• When no criterion is written, the disagreement does not disappear, only its phase shifts: the same rework is 60 minutes at acceptance review, 180 minutes after release.
• Policy choice depends on the escaped defect multiplier: at multiplier 1, writing at closeout is cheapest within budget; at multipliers 6 and 20, targeted product attention is.

Next Step

Every criterion here was written by parties inside the team: the developer, the tester, the product side. All three stand on the side designing the system, and all three use the same vocabulary. Yet the expectation that must ultimately be met by the criterion sits outside the team, and that expectation was written into no slot — how many steps a library clerk expects a loan transaction to take, when a member finds a delay notification meaningful. The next lesson manages that verification: how acceptance scope is distributed across user tasks, how the returning feedback is separated into defect, request, and misunderstanding, and what each class costs the team.