Accessibility Testing

Four lessons assumed a client that misuses the system. The other face of the same question is a reader who cannot use it: does the lending screen work for someone who cannot see the screen, cannot use a mouse, or reads with the text enlarged?

This question’s criteria, their interface-side counterparts, and the automatic checker’s rule classes were built in the Frontend Quality course; they are not repeated here. This lesson’s question is a testing question, and it measures the accessibility audit itself: the automatable share. How many of the criteria can be handed to a program, how many must be manually verified, and what is the automatic check’s own false positive and false negative rate? The measurement will be made not over browser output but over a small document model.

The Document Model and Known Defects

The lending screen’s element tree carries eight defects; the location of each has been manually verified.

// document.mjs — the element tree of the lending screen and the manually verified defect list
export const DOCUMENT = {
  type: "page", title: "Loan Operations", lang: null, children: [
    { type: "section", name: "navigation", children: [
      { type: "link", href: "/catalog", text: "Catalog" },
      { type: "link", href: "/loan", text: "Loan" },
    ] },
    { type: "section", name: "main", children: [
      { type: "heading", level: 1, text: "Loan Operations" },
      { type: "heading", level: 3, text: "Waiting on the Shelf" },
      { type: "list", children: [
        { type: "row", children: [
          { type: "text", text: "The Hourglass" },
          { type: "action", id: "get-1", text: "", icon: "cart", size: 40, focusable: true },
        ] },
        { type: "row", children: [
          { type: "text", text: "Still on the Shelf" },
          { type: "action", id: "get-2", text: "Get", size: 40, focusable: true },
        ] },
      ] },
      { type: "field", id: "member-no", kind: "text", label: null },
      { type: "text", text: "Penalty: 30", color: "#8a8a8a", background: "#ffffff" },
      { type: "action", id: "history", text: "History", size: 40, focusable: true,
        disabled: true, color: "#9b9b9b", background: "#ffffff" },
    ] },
  ],
};

// Manually verified defects: criterion code, location, and a short reason.
// The disabled "history" action is exempt from the contrast threshold; it is not on the list.
export const ACTUAL = [
  ["D1", "get-1", "the cart-icon action's name is empty"],
  ["D2", "member-no", "the member-number field has no label"],
  ["D3", "heading 3", "jumps from heading level one to level three"],
  ["D5", "page", "the document carries no language declaration"],
  ["D7", "Penalty: 30", "the penalty text's contrast is below the threshold"],
  ["D9", "-", "the delay is signaled only by a color change"],
  ["D10", "-", "an action that performs two different operations carries the same name"],
  ["D11", "-", "the focus order does not match the visual order"],
];

export function* walk(d = DOCUMENT) {
  yield d;
  for (const c of d.children ?? []) yield* walk(c);
}

The Criteria’s Decision Source

What decides whether a criterion can be automated is not the criterion’s importance but where the decision is read from. There are four sources: a declaration read directly from the tree, a number computed from declared values, an intent that depends on what the content means, and a sequence of interaction over time. The first two can be handed to a program, the last two cannot.

// auditor.mjs — the criterion set, decision sources, and the checks decidable from the tree
import { walk } from "./document.mjs";

// decision source: tree | calc (both automatic) | intent | sequence (both manual)
export const CRITERION = [
  ["D1", "action name is empty", "tree", 90], ["D2", "field has no label", "tree", 90],
  ["D3", "heading level is skipped", "tree", 70], ["D4", "id is duplicated", "tree", 90],
  ["D5", "no language declaration", "tree", 90], ["D6", "action does not take focus", "tree", 90],
  ["D7", "contrast is below the threshold", "calc", 60], ["D8", "touch target is small", "calc", 50],
  ["D9", "signaled by color alone", "intent", 0], ["D10", "action name is ambiguous", "intent", 0],
  ["D11", "focus order does not match the visual order", "intent", 0],
  ["D12", "the dialog cannot be exited with the keyboard", "sequence", 0],
  ["D13", "the timing of the change announcement", "sequence", 0],
  ["D14", "content is lost when text is enlarged", "sequence", 0],
];

const channel = (c) => (c <= 0.03928 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4);
const luminance = (h) => {
  const [r, g, b] = [1, 3, 5].map((i) => channel(parseInt(h.slice(i, i + 2), 16) / 255));
  return 0.2126 * r + 0.7152 * g + 0.0722 * b;
};
export const contrast = (fg, bg) => {
  const [a, b] = [luminance(fg), luminance(bg)].sort((x, y) => y - x);
  return (a + 0.05) / (b + 0.05);
};

export function audit(document) {
  const findings = [], ids = [];
  let previous = 0;
  for (const d of walk(document)) {
    if (d.type === "page" && !d.lang) findings.push({ criterion: "D5", location: "page" });
    if (d.id) ids.push(d.id);
    if (d.type === "heading") {
      if (previous && d.level > previous + 1) findings.push({ criterion: "D3", location: `heading ${d.level}` });
      previous = d.level;
    }
    if (d.type === "action") {
      if (!d.text) findings.push({ criterion: "D1", location: d.id });
      if (!d.focusable) findings.push({ criterion: "D6", location: d.id });
      if (d.size < 24) findings.push({ criterion: "D8", location: d.id });
    }
    if (d.type === "field" && !d.label) findings.push({ criterion: "D2", location: d.id });
    if (d.color && d.background && contrast(d.color, d.background) < 4.5)
      findings.push({ criterion: "D7", location: d.id ?? d.text, ratio: contrast(d.color, d.background).toFixed(2) });
  }
  const duplicated = ids.filter((k, i) => ids.indexOf(k) !== i);
  for (const k of new Set(duplicated)) findings.push({ criterion: "D4", location: k });
  return findings;
}

NF13 (assumption): the decision-source classification and the manually verified defect list are correct. The automatable share is relative to these two lists; the share changes when the criterion set is expanded.

Automatable Share and Findings

// measurement.mjs — automatable share, findings, and threshold scanning
import { DOCUMENT, ACTUAL } from "./document.mjs";
import { CRITERION, audit } from "./auditor.mjs";

const AUTOMATIC = ["tree", "calc"];
const automatic = CRITERION.filter(([, , k]) => AUTOMATIC.includes(k));
console.log(`${CRITERION.length} criteria; breakdown by decision source:`);
for (const k of ["tree", "calc", "intent", "sequence"]) {
  const n = CRITERION.filter(([, , x]) => x === k).length;
  console.log(`  ${k.padEnd(10)}${String(n).padStart(3)}  ${((100 * n) / CRITERION.length).toFixed(0)}%`);
}
console.log(`automatable share: ${automatic.length}/${CRITERION.length} = ${((100 * automatic.length) / CRITERION.length).toFixed(0)}%`);

const score = Object.fromEntries(CRITERION.map(([k, , , p]) => [k, p]));
const label = Object.fromEntries(CRITERION.map(([k, a]) => [k, a]));
const actualSet = new Set(ACTUAL.map(([k, y]) => `${k}|${y}`));
const key = (b) => `${b.criterion}|${b.location}`;
const findings = audit(DOCUMENT);
const correct = findings.filter((b) => actualSet.has(key(b))).length;
console.log(`\n${findings.length} findings, ${ACTUAL.length} manually verified defects, automatically found ${correct}` +
  ` = ${((100 * correct) / ACTUAL.length).toFixed(0)}%`);
for (const b of findings) {
  console.log(`  ${b.criterion}  ${label[b.criterion].padEnd(33)}${String(b.location).padEnd(13)}` +
    `${String(score[b.criterion]).padStart(3)}  actual: ${actualSet.has(key(b)) ? "yes" : "no"}` +
    `${b.ratio ? `  (ratio ${b.ratio})` : ""}`);
}

console.log(`\n${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=5".padStart(10)}`);
for (const e of [50, 60, 70, 90]) {
  const k = findings.filter((b) => score[b.criterion] >= e);
  const fp = k.filter((b) => !actualSet.has(key(b))).length;
  const fn = ACTUAL.length - k.filter((b) => actualSet.has(key(b))).length;
  console.log(`${String(e).padStart(9)}${String(k.length).padStart(11)}${String(fp).padStart(16)}${String(fn).padStart(16)}${String(fp + 5 * fn).padStart(10)}`);
}
const automaticCode = automatic.map(([k]) => k);
const manual = ACTUAL.filter(([k]) => !automaticCode.includes(k));
console.log(`\n${manual.length} defects that never appear at any threshold (the manual verification area):`);
for (const [k, , g] of manual) console.log(`  ${k}  ${g}`);

14 criteria; breakdown by decision source:
  tree        6  43%
  calc        2  14%
  intent      3  21%
  sequence    3  21%
automatable share: 8/14 = 57%

6 findings, 8 manually verified defects, automatically found 5 = 63%
  D5  no language declaration          page          90  actual: yes
  D3  heading level is skipped         heading 3     70  actual: yes
  D1  action name is empty             get-1         90  actual: yes
  D2  field has no label               member-no     90  actual: yes
  D7  contrast is below the threshold  Penalty: 30   60  actual: yes  (ratio 3.45)
  D7  contrast is below the threshold  history       60  actual: no  (ratio 2.78)

threshold  remaining  false positive  false negative  cost w=5
       50          6               1               3        16
       60          6               1               3        16
       70          4               0               4        20
       90          3               0               5        25

3 defects that never appear at any threshold (the manual verification area):
  D9  the delay is signaled only by a color change
  D10  an action that performs two different operations carries the same name
  D11  the focus order does not match the visual order

The two ratios say different things. The automatable share, 57%, is the portion of the criterion set that can be handed to a program, and it is independent of the document. The found-defect rate, 63%, says that five of the eight defects in this document were caught by the automatic audit, and it depends on the document: had the defects fallen into a different distribution, this rate would change. The second number cannot be derived from the first.

The one false positive came from the contrast calculation. A disabled action’s text contrast is below the threshold, but disabled controls are exempt from this threshold, and the auditor does not know that. This is the common problem with calculated criteria: the number is correct, the rule is correct, the context is missing.

Why Raising the Threshold Does Not Help

Threshold scanning gives a different result than the previous three lessons. There, raising the threshold dropped false positives while missing some real defects, and there was a sweet spot in between. Here, raising the threshold drops a single false positive and, in exchange, misses two real defects; the cost rises monotonically. The lowest cost sits at the bottom end of the scan.

The table’s false-positive column is the false-fail count, its false-negative column is the false-pass count; here false pass never drops below three at any threshold. The reason is in the table’s last three rows: the three missed defects never appear at any threshold, because the criteria they correspond to carry no score — they cannot be handed to a program. Here, missing is not the result of the threshold choice; it is the method’s structural limit. A delay being signaled only by a color change, two actions with the same name doing different things, and a focus order that does not match the visual order: none of the three leaves any missing declaration in the tree.

So this audit’s threshold is not a score. The threshold and its source are written as follows: release is tied to the condition that the automatic audit reports zero findings and all six criteria on the manual verification list have been checked off. The threshold’s source is this measurement — three of the eight defects never appearing in the automatic audit at all shows that a gate relying only on the automatic audit would turn green with three real defects still present.

The Cost of the Audit

The run-independent cost lives in three places. The automatic side is cheap: one tree traversal, fourteen elements, zero processes, zero requests; it can run on every change.

The manual side is not cheap, and its cost is measured by the criterion count per screen: six criteria, separately for every screen. As the lending system’s screen count grows this number grows linearly, and unlike the automatic audit it is paid again with every release.

The third cost is maintaining the classification. A criterion’s decision source can change: a check that requires intent today can become readable from the tree once a new declaration is added to the document model. If the classification is not updated, the automatable share appears lower than it is, and the manual verification list stays longer than it needs to.

Who owns the decision: an automatic finding stops the release; the manual verification list is a checklist, and an unchecked criterion does not stop the release, it only records which criteria the release went out without testing.

Summary

• What determines a criterion’s automatability is the decision’s source: a declaration read from the tree and a computed value are automatic, intent and an interaction sequence are not.
• Eight of fourteen criteria can be checked automatically (automatable share 57%); five of the eight defects in this document were found automatically (63%). The two ratios measure different things, and one cannot be derived from the other.
• The one false positive came from a calculated criterion: a disabled control’s contrast was below the threshold, but that control was exempt from the threshold — the number was correct, the context was missing.
• The three missed defects never appeared at any threshold; here, missing is the result of the method, not the threshold choice, and raising the threshold only increases the cost in one direction.
• The threshold is not a score but a condition: zero automatic findings and all six manual verification criteria checked off. Its source is the measurement that a gate relying only on the automatic audit would turn green with three real defects present.
• The cost: the automatic side is as cheap as one tree traversal, the manual side is as expensive as six criteria per screen, and it is paid again with every release.

Next Step

This audit ran on a single document model and gave a single result. Yet the same lending screen does not produce the same tree for every reader: different rendering engines interpret the same declaration differently, different screen widths build different layouts, different input methods impose different interaction orders. Every measurement so far was made in a single environment, and the user share that environment represents was never asked. The next lesson opens the environment axes into a matrix, selects which subset of the matrix to run with two separate criteria, and counts the share the selection covers.