--- title: 'Accessibility Testing' source: 'https://academia.sh/en/courses/non-functional-testing/accessibility-testing' course: 'Non-Functional Testing' language: en updated: '2026-08-23T14:25:17+00:00' license: 'CC BY-SA 4.0' --- # Accessibility Testing Measuring an automatic audit's automatable share: separating criteria by decision source, the false positive and false negative counts an auditor that actually runs on a small element tree produces, and the area where manual verification is mandatory in a check where raising the threshold brings no gain. Four lessons assumed a client that misuses the system. The other face of the same question is a reader who **cannot use** it: does the lending screen work for someone who cannot see the screen, cannot use a mouse, or reads with the text enlarged? This question's criteria, their interface-side counterparts, and the automatic checker's rule classes were built in the Frontend Quality course; they are not repeated here. This lesson's question is a testing question, and it measures **the accessibility audit itself**: the **automatable share**. How many of the criteria can be handed to a program, how many must be manually verified, and what is the automatic check's own false positive and false negative rate? The measurement will be made not over browser output but over a small document model. ## The Document Model and Known Defects The lending screen's element tree carries eight defects; the location of each has been manually verified. ```js // document.mjs — the element tree of the lending screen and the manually verified defect list export const DOCUMENT = { type: "page", title: "Loan Operations", lang: null, children: [ { type: "section", name: "navigation", children: [ { type: "link", href: "/catalog", text: "Catalog" }, { type: "link", href: "/loan", text: "Loan" }, ] }, { type: "section", name: "main", children: [ { type: "heading", level: 1, text: "Loan Operations" }, { type: "heading", level: 3, text: "Waiting on the Shelf" }, { type: "list", children: [ { type: "row", children: [ { type: "text", text: "The Hourglass" }, { type: "action", id: "get-1", text: "", icon: "cart", size: 40, focusable: true }, ] }, { type: "row", children: [ { type: "text", text: "Still on the Shelf" }, { type: "action", id: "get-2", text: "Get", size: 40, focusable: true }, ] }, ] }, { type: "field", id: "member-no", kind: "text", label: null }, { type: "text", text: "Penalty: 30", color: "#8a8a8a", background: "#ffffff" }, { type: "action", id: "history", text: "History", size: 40, focusable: true, disabled: true, color: "#9b9b9b", background: "#ffffff" }, ] }, ], }; // Manually verified defects: criterion code, location, and a short reason. // The disabled "history" action is exempt from the contrast threshold; it is not on the list. export const ACTUAL = [ ["D1", "get-1", "the cart-icon action's name is empty"], ["D2", "member-no", "the member-number field has no label"], ["D3", "heading 3", "jumps from heading level one to level three"], ["D5", "page", "the document carries no language declaration"], ["D7", "Penalty: 30", "the penalty text's contrast is below the threshold"], ["D9", "-", "the delay is signaled only by a color change"], ["D10", "-", "an action that performs two different operations carries the same name"], ["D11", "-", "the focus order does not match the visual order"], ]; export function* walk(d = DOCUMENT) { yield d; for (const c of d.children ?? []) yield* walk(c); } ``` ## The Criteria's Decision Source What decides whether a criterion can be automated is not the criterion's importance but **where the decision is read from**. There are four sources: a declaration read directly from the tree, a number computed from declared values, an intent that depends on what the content means, and a sequence of interaction over time. The first two can be handed to a program, the last two cannot. ```js // auditor.mjs — the criterion set, decision sources, and the checks decidable from the tree import { walk } from "./document.mjs"; // decision source: tree | calc (both automatic) | intent | sequence (both manual) export const CRITERION = [ ["D1", "action name is empty", "tree", 90], ["D2", "field has no label", "tree", 90], ["D3", "heading level is skipped", "tree", 70], ["D4", "id is duplicated", "tree", 90], ["D5", "no language declaration", "tree", 90], ["D6", "action does not take focus", "tree", 90], ["D7", "contrast is below the threshold", "calc", 60], ["D8", "touch target is small", "calc", 50], ["D9", "signaled by color alone", "intent", 0], ["D10", "action name is ambiguous", "intent", 0], ["D11", "focus order does not match the visual order", "intent", 0], ["D12", "the dialog cannot be exited with the keyboard", "sequence", 0], ["D13", "the timing of the change announcement", "sequence", 0], ["D14", "content is lost when text is enlarged", "sequence", 0], ]; const channel = (c) => (c <= 0.03928 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4); const luminance = (h) => { const [r, g, b] = [1, 3, 5].map((i) => channel(parseInt(h.slice(i, i + 2), 16) / 255)); return 0.2126 * r + 0.7152 * g + 0.0722 * b; }; export const contrast = (fg, bg) => { const [a, b] = [luminance(fg), luminance(bg)].sort((x, y) => y - x); return (a + 0.05) / (b + 0.05); }; export function audit(document) { const findings = [], ids = []; let previous = 0; for (const d of walk(document)) { if (d.type === "page" && !d.lang) findings.push({ criterion: "D5", location: "page" }); if (d.id) ids.push(d.id); if (d.type === "heading") { if (previous && d.level > previous + 1) findings.push({ criterion: "D3", location: `heading ${d.level}` }); previous = d.level; } if (d.type === "action") { if (!d.text) findings.push({ criterion: "D1", location: d.id }); if (!d.focusable) findings.push({ criterion: "D6", location: d.id }); if (d.size < 24) findings.push({ criterion: "D8", location: d.id }); } if (d.type === "field" && !d.label) findings.push({ criterion: "D2", location: d.id }); if (d.color && d.background && contrast(d.color, d.background) < 4.5) findings.push({ criterion: "D7", location: d.id ?? d.text, ratio: contrast(d.color, d.background).toFixed(2) }); } const duplicated = ids.filter((k, i) => ids.indexOf(k) !== i); for (const k of new Set(duplicated)) findings.push({ criterion: "D4", location: k }); return findings; } ``` **NF13 (assumption):** the decision-source classification and the manually verified defect list are correct. The automatable share is relative to these two lists; the share changes when the criterion set is expanded. ## Automatable Share and Findings ```js // measurement.mjs — automatable share, findings, and threshold scanning import { DOCUMENT, ACTUAL } from "./document.mjs"; import { CRITERION, audit } from "./auditor.mjs"; const AUTOMATIC = ["tree", "calc"]; const automatic = CRITERION.filter(([, , k]) => AUTOMATIC.includes(k)); console.log(`${CRITERION.length} criteria; breakdown by decision source:`); for (const k of ["tree", "calc", "intent", "sequence"]) { const n = CRITERION.filter(([, , x]) => x === k).length; console.log(` ${k.padEnd(10)}${String(n).padStart(3)} ${((100 * n) / CRITERION.length).toFixed(0)}%`); } console.log(`automatable share: ${automatic.length}/${CRITERION.length} = ${((100 * automatic.length) / CRITERION.length).toFixed(0)}%`); const score = Object.fromEntries(CRITERION.map(([k, , , p]) => [k, p])); const label = Object.fromEntries(CRITERION.map(([k, a]) => [k, a])); const actualSet = new Set(ACTUAL.map(([k, y]) => `${k}|${y}`)); const key = (b) => `${b.criterion}|${b.location}`; const findings = audit(DOCUMENT); const correct = findings.filter((b) => actualSet.has(key(b))).length; console.log(`\n${findings.length} findings, ${ACTUAL.length} manually verified defects, automatically found ${correct}` + ` = ${((100 * correct) / ACTUAL.length).toFixed(0)}%`); for (const b of findings) { console.log(` ${b.criterion} ${label[b.criterion].padEnd(33)}${String(b.location).padEnd(13)}` + `${String(score[b.criterion]).padStart(3)} actual: ${actualSet.has(key(b)) ? "yes" : "no"}` + `${b.ratio ? ` (ratio ${b.ratio})` : ""}`); } console.log(`\n${"threshold".padStart(9)}${"remaining".padStart(11)}${"false positive".padStart(16)}${"false negative".padStart(16)}${"cost w=5".padStart(10)}`); for (const e of [50, 60, 70, 90]) { const k = findings.filter((b) => score[b.criterion] >= e); const fp = k.filter((b) => !actualSet.has(key(b))).length; const fn = ACTUAL.length - k.filter((b) => actualSet.has(key(b))).length; console.log(`${String(e).padStart(9)}${String(k.length).padStart(11)}${String(fp).padStart(16)}${String(fn).padStart(16)}${String(fp + 5 * fn).padStart(10)}`); } const automaticCode = automatic.map(([k]) => k); const manual = ACTUAL.filter(([k]) => !automaticCode.includes(k)); console.log(`\n${manual.length} defects that never appear at any threshold (the manual verification area):`); for (const [k, , g] of manual) console.log(` ${k} ${g}`); ``` ``` 14 criteria; breakdown by decision source: tree 6 43% calc 2 14% intent 3 21% sequence 3 21% automatable share: 8/14 = 57% 6 findings, 8 manually verified defects, automatically found 5 = 63% D5 no language declaration page 90 actual: yes D3 heading level is skipped heading 3 70 actual: yes D1 action name is empty get-1 90 actual: yes D2 field has no label member-no 90 actual: yes D7 contrast is below the threshold Penalty: 30 60 actual: yes (ratio 3.45) D7 contrast is below the threshold history 60 actual: no (ratio 2.78) threshold remaining false positive false negative cost w=5 50 6 1 3 16 60 6 1 3 16 70 4 0 4 20 90 3 0 5 25 3 defects that never appear at any threshold (the manual verification area): D9 the delay is signaled only by a color change D10 an action that performs two different operations carries the same name D11 the focus order does not match the visual order ``` The two ratios say different things. **The automatable share, 57%**, is the portion of the criterion set that can be handed to a program, and it is independent of the document. **The found-defect rate, 63%**, says that five of the eight defects in this document were caught by the automatic audit, and it depends on the document: had the defects fallen into a different distribution, this rate would change. The second number cannot be derived from the first. The one false positive came from the contrast calculation. A disabled action's text contrast is below the threshold, but disabled controls are exempt from this threshold, and the auditor does not know that. This is the common problem with calculated criteria: the number is correct, the rule is correct, **the context is missing**. ## Why Raising the Threshold Does Not Help Threshold scanning gives a different result than the previous three lessons. There, raising the threshold dropped false positives while missing some real defects, and there was a sweet spot in between. Here, raising the threshold drops a single false positive and, in exchange, misses two real defects; the cost rises monotonically. The lowest cost sits at the **bottom** end of the scan. The table's false-positive column is the **false-fail** count, its false-negative column is the **false-pass** count; here false pass never drops below three at any threshold. The reason is in the table's last three rows: the three missed defects never appear at any threshold, because the criteria they correspond to carry no score — they cannot be handed to a program. **Here, missing is not the result of the threshold choice; it is the method's structural limit.** A delay being signaled only by a color change, two actions with the same name doing different things, and a focus order that does not match the visual order: none of the three leaves any missing declaration in the tree. So this audit's threshold is not a score. **The threshold and its source** are written as follows: release is tied to the condition that the automatic audit reports zero findings **and** all six criteria on the manual verification list have been checked off. The threshold's source is this measurement — three of the eight defects never appearing in the automatic audit at all shows that a gate relying only on the automatic audit would turn green with three real defects still present. ## The Cost of the Audit The run-independent cost lives in three places. The automatic side is cheap: one tree traversal, fourteen elements, zero processes, zero requests; it can run on every change. The manual side is not cheap, and its cost is measured by **the criterion count per screen**: six criteria, separately for every screen. As the lending system's screen count grows this number grows linearly, and unlike the automatic audit it is paid again with every release. The third cost is maintaining the classification. A criterion's decision source can change: a check that requires intent today can become readable from the tree once a new declaration is added to the document model. If the classification is not updated, the automatable share appears lower than it is, and the manual verification list stays longer than it needs to. **Who owns the decision:** an automatic finding stops the release; the manual verification list is a checklist, and an unchecked criterion does not stop the release, it only records which criteria the release went out without testing. ## Summary - What determines a criterion's automatability is the decision's source: a declaration read from the tree and a computed value are automatic, intent and an interaction sequence are not. - Eight of fourteen criteria can be checked automatically (automatable share 57%); five of the eight defects in this document were found automatically (63%). The two ratios measure different things, and one cannot be derived from the other. - The one false positive came from a calculated criterion: a disabled control's contrast was below the threshold, but that control was exempt from the threshold — the number was correct, the context was missing. - The three missed defects never appeared at any threshold; here, missing is the result of the method, not the threshold choice, and raising the threshold only increases the cost in one direction. - The threshold is not a score but a condition: zero automatic findings **and** all six manual verification criteria checked off. Its source is the measurement that a gate relying only on the automatic audit would turn green with three real defects present. - The cost: the automatic side is as cheap as one tree traversal, the manual side is as expensive as six criteria per screen, and it is paid again with every release. ## Next Step This audit ran on a single document model and gave a single result. Yet the same lending screen does not produce the same tree for every reader: different rendering engines interpret the same declaration differently, different screen widths build different layouts, different input methods impose different interaction orders. Every measurement so far was made in a single environment, and the user share that environment represents was never asked. The next lesson opens the environment axes into a matrix, selects which subset of the matrix to run with two separate criteria, and counts the share the selection covers.