Accessibility Testing

Most of the audits in the previous lessons ran through a script: the accessible name computation, tab order, plane defects, and contrast ratios were all computations that took input and produced output. This raises an unavoidable question. How much of accessibility can be delegated to a program, and what is left over?

The answer is two numbers, and both are computed in this lesson: how much of the criteria can be decided by a machine, and how much of an interface’s real defects are found by automated auditing. The two numbers are different, and the second one matters more.

What an Automated Checker Does

An automated accessibility checker reads the rendered document and the accessibility tree derived from it, looks at computed style values, and applies a set of rules. The rules it can apply fall into four classes.

Presence rules. They check whether a declaration exists at all: does the image have an alternative text attribute, is the interactive element’s accessible name empty, does the document declare a language.

Validity rules. They check whether a written declaration is defined: is the role name a defined role, is the state used valid for that role, does aria-labelledby reference an id that exists, does the same id appear twice in the document.

Computable-value rules. They compute a numeric threshold directly: the contrast ratio between two flat-colored layers, the tab order derived from tabindex values, the size of a touch target.

Uniqueness and consistency rules. They count at the scale of the document: is there more than one main landmark, are heading levels skipped, is there more than one unlabeled landmark of the same kind.

The shared property of these four classes is that the decision is readable from the document. Where the decision depends on intent, meaning, or a sequence of interaction over time, no rule can be written.

Criteria and Findings

The script below performs two computations together: the automatability distribution of criteria, and the detection rate of automated auditing on the real defects found in the North Slope interface across previous lessons.

// coverage.mjs — automatability classes of criteria and coverage on real findings

// class: "automatic" (machine-decidable) | "partial" (part of it) | "manual"
const CRITERION = [
  ["1.1.1", "Non-text Content", "partial"],
  ["1.3.1", "Info and Relationships", "partial"],
  ["1.3.5", "Identify Input Purpose", "partial"],
  ["1.4.1", "Use of Color", "manual"],
  ["1.4.3", "Contrast (Minimum)", "partial"],
  ["1.4.4", "Resize Text", "manual"],
  ["1.4.11", "Non-text Contrast", "partial"],
  ["1.4.12", "Text Spacing", "partial"],
  ["2.1.1", "Keyboard", "manual"],
  ["2.1.2", "No Keyboard Trap", "manual"],
  ["2.4.1", "Bypass Blocks", "partial"],
  ["2.4.2", "Page Titled", "automatic"],
  ["2.4.3", "Focus Order", "manual"],
  ["2.4.4", "Link Purpose", "partial"],
  ["2.4.6", "Headings and Labels", "manual"],
  ["2.4.7", "Focus Visible", "partial"],
  ["2.5.3", "Label in Name", "partial"],
  ["3.1.1", "Language of Page", "automatic"],
  ["3.2.2", "On Input", "manual"],
  ["3.3.1", "Error Identification", "partial"],
  ["3.3.2", "Labels or Instructions", "partial"],
  ["4.1.2", "Name, Role, Value", "partial"],
  ["4.1.3", "Status Messages", "manual"],
];

const counts = {};
for (const [, , c] of CRITERION) counts[c] = (counts[c] ?? 0) + 1;
console.log("criterion class  count  share");
for (const c of ["automatic", "partial", "manual"]) {
  console.log(`${c.padEnd(13)} ${String(counts[c]).padStart(4)}  ${((100 * counts[c]) / CRITERION.length).toFixed(1)}%`);
}
console.log(`total         ${String(CRITERION.length).padStart(4)}`);

// Real defects found in the North Slope interface across previous lessons
// [finding, criterion, does the automated checker catch it]
const FINDING = [
  ["empty alternative text on an icon button", "4.1.2", true],
  ["aria-labelledby references an invalid id", "4.1.2", true],
  ["aria-label suppresses the visible label", "2.5.3", true],
  ["role=button not taking focus", "2.1.1", true],
  ["focusable element carries aria-hidden", "4.1.2", true],
  ["role=presentation on a data table", "1.3.1", false],
  ["positive tabindex value", "2.4.3", true],
  ["tab order does not match visual order", "2.4.3", false],
  ["cannot exit the dialog by keyboard", "2.1.2", false],
  ["focus does not return when the dialog closes", "2.4.3", false],
  ["unlabeled second navigation landmark", "1.3.1", true],
  ["heading level skipped", "1.3.1", true],
  ["link text disconnected from context", "2.4.4", true],
  ["same text goes to two different targets", "2.4.4", false],
  ["assertive announcement drops what is pending", "4.1.3", false],
  ["field border below the 3:1 threshold", "1.4.11", false],
  ["text contrast below the 4.5:1 threshold", "1.4.3", true],
  ["attribute reported by color alone", "1.4.1", false],
];

const caught = FINDING.filter(([, , y]) => y).length;
console.log(`\ntotal findings: ${FINDING.length}, caught automatically: ${caught}, missed: ${FINDING.length - caught}`);
console.log(`automated audit detection rate: ${((100 * caught) / FINDING.length).toFixed(1)}%`);

console.log("\nwhat automated auditing misses:");
for (const [b, c, y] of FINDING) {
  if (!y) console.log(`  ${c.padEnd(7)} ${b}`);
}

// Which manual test finds each miss
const TEST = {
  "role=presentation on a data table": "markup reading",
  "tab order does not match visual order": "keyboard pass",
  "cannot exit the dialog by keyboard": "keyboard pass",
  "focus does not return when the dialog closes": "keyboard pass",
  "same text goes to two different targets": "navigation plane review",
  "assertive announcement drops what is pending": "task walkthrough",
  "field border below the 3:1 threshold": "state matrix check",
  "attribute reported by color alone": "grayscale image test",
};
const testCounts = {};
for (const [b, , y] of FINDING) {
  if (y) continue;
  const t = TEST[b];
  testCounts[t] = (testCounts[t] ?? 0) + 1;
}
console.log("\nmanual tests that find the misses:");
for (const [t, n] of Object.entries(testCounts).sort((a, b) => b[1] - a[1])) {
  console.log(`  ${t.padEnd(26)} ${n} finding${n > 1 ? "s" : ""}`);
}

criterion class  count  share
automatic        2  8.7%
partial         13  56.5%
manual           8  34.8%
total           23

total findings: 18, caught automatically: 10, missed: 8
automated audit detection rate: 55.6%

what automated auditing misses:
  1.3.1   role=presentation on a data table
  2.4.3   tab order does not match visual order
  2.1.2   cannot exit the dialog by keyboard
  2.4.3   focus does not return when the dialog closes
  2.4.4   same text goes to two different targets
  4.1.3   assertive announcement drops what is pending
  1.4.11  field border below the 3:1 threshold
  1.4.1   attribute reported by color alone

manual tests that find the misses:
  keyboard pass              3 findings
  markup reading             1 finding
  navigation plane review    1 finding
  task walkthrough           1 finding
  state matrix check         1 finding
  grayscale image test       1 finding

The first table gives the criteria side: only two of the twenty-three criteria can be decided by a machine from beginning to end. The majority are in the partial class, and this class’s meaning is easy to misread. For a partial criterion, automated auditing can only report failure; it cannot report success. The presence of an alternative text attribute does not show that criterion 1.1.1 is met, only that the attribute exists.

The second table is the more concrete one: of the eighteen real defects found across the six lessons in this course, ten are caught by automated auditing and eight are missed. The rate varies by interface; what stays constant is which kind the misses belong to.

What the Misses Have in Common

The eight missed findings split into three groups, and all three are structurally impossible to automate.

Those that require a judgment of meaning. Whether a table carries data or exists for layout cannot be read from the document; on a table with the presentational role written, the checker cannot know the author’s intent. The same text going to two different destinations is the same kind of case: it is a defect if the two links go to separate things, and not a defect if they go to two copies of the same thing.

Those that require a sequence over time. A keyboard trap, focus not returning when the dialog closes, and an assertive announcement dropping what is pending cannot be seen by looking at a single document state; they do not surface without running a sequence of interaction. These are auditable — the state machine in the Keyboard Access lesson did exactly that — but auditing them requires modeling the component’s behavior or running an actual flow.

Those that depend on the rendered result. A border’s contrast, measured under layers of shadow and transparency, stops being the ratio of two flat colors; text over a gradient fill or an image has no single contrast value. Whether color is a single channel is a question of which channels the information is given through, and it does not exist as a declaration in the document.

The Order of Manual Verification

Manual verification is not an unbounded review; it is five steps ordered by cost. The ordering rests on the cheap steps producing the most findings first.

Keyboard pass. The tab key is used to go from the beginning of the page to the end. Four things are checked: is focus visible at every stop, does the order match the order on screen, can every opened component be exited by keyboard, and where does focus return on close. This step alone finds three of the eight missed findings and requires no tooling.

Resize check. Text is enlarged to double size and the interface is used to complete the same task. Clipped text, overlapping boxes, and controls that disappear show up at this step.

Markup reading. The markup a component produces is read: is the role correct, is the state updated, which source does the name come from. This step finds defects that require a judgment of meaning.

Navigation plane review. The landmark list, heading outline, and link list are each extracted separately and read. Does the list make sense stripped of its surrounding context?

Task walkthrough. A task is completed end to end without seeing the interface and without using a mouse — filtering the measurement list, reporting a correction, confirming the result. This is the most expensive step, and it finds what the other four cannot: whether the information is sufficient.

Tying the Audit to the Process

A one-time audit becomes stale at the next change. The way the audit ties into the process has three layers.

At the component scale, the role and accessible name a component produces are locked in with a test: when the component renders, the button role and the expected name must be present in the tree. These tests look at the counterpart in the tree, not at the markup structure; they stay valid even when the component’s internals are rewritten. The same approach is the foundation of the user-centric queries in the Testing and Monitoring topic.

At the page scale, the automated checker runs before release, and every failure it finds blocks the release. The budget form from the Accessibility Standards lesson applies here: the count of unmet A and AA criteria must be zero, and known exceptions sit in an open list with the criterion number and a date.

At the release scale, the five steps of manual verification are applied not to every release but to the parts of the interface that change. When a new component is added, the keyboard pass and markup reading are mandatory; the task walkthrough is done less often.

What Zero Findings Means

Automated auditing producing no findings does not show that the interface is accessible. This sentence is this lesson’s conclusion, and it has two sides.

First, the computed rate: on this interface, 55.6 percent of findings were caught, and the rest were not. Second, the checker can also produce a false positive. Text over a gradient fill can be reported below the threshold by assuming a single background color; a correctly built component can be flagged because of a pattern the checker does not recognize. Every finding needs to be confirmed by a person.

The value of automated auditing is in catching regressions. It cheaply prevents a once-fixed defect from coming back, and it reserves manual verification time for the defects that can only be found by hand.

Summary

• An automated checker applies four classes of rule: presence, validity, computable value, and uniqueness-consistency; in all of them, the decision must be readable from the document.
• Most criteria are in the partial class, and a partial criterion’s automated audit can only report failure, never success.
• Of the eighteen real defects found in this course, ten are caught automatically and eight are missed; the misses require a judgment of meaning, a sequence over time, or the rendered result.
• Manual verification is ordered by cost: keyboard pass, resize check, markup reading, navigation plane review, and task walkthrough.
• The audit ties into the process at three scales: role and name testing at the component scale, an automated check that blocks release at the page scale, and manual verification applied to changed parts at the release scale.
• Zero findings from automated auditing is not proof of conformance; the checker also produces false positives, and its real value is catching regressions.

Next Step

Two axes of quality are now measurable. Performance is tied to a budget, accessibility to a set of criteria; both are countable, and both carry a definition of regression. A third axis remains, and it asks a different question than the previous two.

Accessibility makes sure that the information an interface carries reaches the user. The next topic takes up the flow in the opposite direction: text coming from the user entering the interface. Text written into the description field on the measurement entry form is shown to other people in the measurement list. When that text is written onto the page, is it processed as data or as code? Where that distinction is lost, the user has to be protected from the interface itself. The first lesson of the Client-Side Security topic, Cross-Site Scripting, takes up how that distinction gets lost and how output escaping restores it.