---
title: 'Accessibility Testing'
source: 'https://academia.sh/en/courses/frontend-quality/accessibility-testing'
course: 'Frontend Quality'
language: en
updated: '2026-08-17T18:11:05+00:00'
license: 'CC BY-SA 4.0'
---

# Accessibility Testing

The rule classes an automated checker can decide, the automatability distribution of criteria, the rate that is missed on real findings, and the cost-ordered sequence of manual verification.

Most of the audits in the previous lessons ran through a script: the accessible name
computation, tab order, plane defects, and contrast ratios were all computations that
took input and produced output. This raises an unavoidable question. How much of
accessibility can be delegated to a program, and what is left over?

The answer is two numbers, and both are computed in this lesson: how much of the criteria
can be decided by a machine, and how much of an interface's real defects are found by
automated auditing. The two numbers are different, and the second one matters more.

## What an Automated Checker Does

An **automated accessibility checker** reads the rendered document and the accessibility
tree derived from it, looks at computed style values, and applies a set of rules. The
rules it can apply fall into four classes.

**Presence rules.** They check whether a declaration exists at all: does the image have
an alternative text attribute, is the interactive element's accessible name empty, does
the document declare a language.

**Validity rules.** They check whether a written declaration is defined: is the role name
a defined role, is the state used valid for that role, does `aria-labelledby` reference an
id that exists, does the same id appear twice in the document.

**Computable-value rules.** They compute a numeric threshold directly: the contrast ratio
between two flat-colored layers, the tab order derived from `tabindex` values, the size of
a touch target.

**Uniqueness and consistency rules.** They count at the scale of the document: is there
more than one `main` landmark, are heading levels skipped, is there more than one
unlabeled landmark of the same kind.

The shared property of these four classes is that the decision is **readable from the
document**. Where the decision depends on intent, meaning, or a sequence of interaction
over time, no rule can be written.

## Criteria and Findings

The script below performs two computations together: the automatability distribution of
criteria, and the detection rate of automated auditing on the real defects found in the
North Slope interface across previous lessons.

```js
// coverage.mjs — automatability classes of criteria and coverage on real findings

// class: "automatic" (machine-decidable) | "partial" (part of it) | "manual"
const CRITERION = [
  ["1.1.1", "Non-text Content", "partial"],
  ["1.3.1", "Info and Relationships", "partial"],
  ["1.3.5", "Identify Input Purpose", "partial"],
  ["1.4.1", "Use of Color", "manual"],
  ["1.4.3", "Contrast (Minimum)", "partial"],
  ["1.4.4", "Resize Text", "manual"],
  ["1.4.11", "Non-text Contrast", "partial"],
  ["1.4.12", "Text Spacing", "partial"],
  ["2.1.1", "Keyboard", "manual"],
  ["2.1.2", "No Keyboard Trap", "manual"],
  ["2.4.1", "Bypass Blocks", "partial"],
  ["2.4.2", "Page Titled", "automatic"],
  ["2.4.3", "Focus Order", "manual"],
  ["2.4.4", "Link Purpose", "partial"],
  ["2.4.6", "Headings and Labels", "manual"],
  ["2.4.7", "Focus Visible", "partial"],
  ["2.5.3", "Label in Name", "partial"],
  ["3.1.1", "Language of Page", "automatic"],
  ["3.2.2", "On Input", "manual"],
  ["3.3.1", "Error Identification", "partial"],
  ["3.3.2", "Labels or Instructions", "partial"],
  ["4.1.2", "Name, Role, Value", "partial"],
  ["4.1.3", "Status Messages", "manual"],
];

const counts = {};
for (const [, , c] of CRITERION) counts[c] = (counts[c] ?? 0) + 1;
console.log("criterion class  count  share");
for (const c of ["automatic", "partial", "manual"]) {
  console.log(`${c.padEnd(13)} ${String(counts[c]).padStart(4)}  ${((100 * counts[c]) / CRITERION.length).toFixed(1)}%`);
}
console.log(`total         ${String(CRITERION.length).padStart(4)}`);

// Real defects found in the North Slope interface across previous lessons
// [finding, criterion, does the automated checker catch it]
const FINDING = [
  ["empty alternative text on an icon button", "4.1.2", true],
  ["aria-labelledby references an invalid id", "4.1.2", true],
  ["aria-label suppresses the visible label", "2.5.3", true],
  ["role=button not taking focus", "2.1.1", true],
  ["focusable element carries aria-hidden", "4.1.2", true],
  ["role=presentation on a data table", "1.3.1", false],
  ["positive tabindex value", "2.4.3", true],
  ["tab order does not match visual order", "2.4.3", false],
  ["cannot exit the dialog by keyboard", "2.1.2", false],
  ["focus does not return when the dialog closes", "2.4.3", false],
  ["unlabeled second navigation landmark", "1.3.1", true],
  ["heading level skipped", "1.3.1", true],
  ["link text disconnected from context", "2.4.4", true],
  ["same text goes to two different targets", "2.4.4", false],
  ["assertive announcement drops what is pending", "4.1.3", false],
  ["field border below the 3:1 threshold", "1.4.11", false],
  ["text contrast below the 4.5:1 threshold", "1.4.3", true],
  ["attribute reported by color alone", "1.4.1", false],
];

const caught = FINDING.filter(([, , y]) => y).length;
console.log(`\ntotal findings: ${FINDING.length}, caught automatically: ${caught}, missed: ${FINDING.length - caught}`);
console.log(`automated audit detection rate: ${((100 * caught) / FINDING.length).toFixed(1)}%`);

console.log("\nwhat automated auditing misses:");
for (const [b, c, y] of FINDING) {
  if (!y) console.log(`  ${c.padEnd(7)} ${b}`);
}

// Which manual test finds each miss
const TEST = {
  "role=presentation on a data table": "markup reading",
  "tab order does not match visual order": "keyboard pass",
  "cannot exit the dialog by keyboard": "keyboard pass",
  "focus does not return when the dialog closes": "keyboard pass",
  "same text goes to two different targets": "navigation plane review",
  "assertive announcement drops what is pending": "task walkthrough",
  "field border below the 3:1 threshold": "state matrix check",
  "attribute reported by color alone": "grayscale image test",
};
const testCounts = {};
for (const [b, , y] of FINDING) {
  if (y) continue;
  const t = TEST[b];
  testCounts[t] = (testCounts[t] ?? 0) + 1;
}
console.log("\nmanual tests that find the misses:");
for (const [t, n] of Object.entries(testCounts).sort((a, b) => b[1] - a[1])) {
  console.log(`  ${t.padEnd(26)} ${n} finding${n > 1 ? "s" : ""}`);
}
```

```
criterion class  count  share
automatic        2  8.7%
partial         13  56.5%
manual           8  34.8%
total           23

total findings: 18, caught automatically: 10, missed: 8
automated audit detection rate: 55.6%

what automated auditing misses:
  1.3.1   role=presentation on a data table
  2.4.3   tab order does not match visual order
  2.1.2   cannot exit the dialog by keyboard
  2.4.3   focus does not return when the dialog closes
  2.4.4   same text goes to two different targets
  4.1.3   assertive announcement drops what is pending
  1.4.11  field border below the 3:1 threshold
  1.4.1   attribute reported by color alone

manual tests that find the misses:
  keyboard pass              3 findings
  markup reading             1 finding
  navigation plane review    1 finding
  task walkthrough           1 finding
  state matrix check         1 finding
  grayscale image test       1 finding
```

The first table gives the criteria side: only two of the twenty-three criteria can be
decided by a machine from beginning to end. The majority are in the **partial** class,
and this class's meaning is easy to misread. For a partial criterion, automated auditing
can only report **failure**; it cannot report success. The presence of an alternative
text attribute does not show that criterion 1.1.1 is met, only that the attribute exists.

The second table is the more concrete one: of the eighteen real defects found across the
six lessons in this course, ten are caught by automated auditing and eight are missed.
The rate varies by interface; what stays constant is **which kind** the misses belong to.

## What the Misses Have in Common

The eight missed findings split into three groups, and all three are structurally
impossible to automate.

**Those that require a judgment of meaning.** Whether a table carries data or exists for
layout cannot be read from the document; on a table with the presentational role written,
the checker cannot know the author's intent. The same text going to two different
destinations is the same kind of case: it is a defect if the two links go to separate
things, and not a defect if they go to two copies of the same thing.

**Those that require a sequence over time.** A keyboard trap, focus not returning when the
dialog closes, and an assertive announcement dropping what is pending cannot be seen by
looking at a single document state; they do not surface without running a sequence of
interaction. These are auditable — the state machine in the Keyboard Access lesson did
exactly that — but auditing them requires modeling the component's behavior or running an
actual flow.

**Those that depend on the rendered result.** A border's contrast, measured under layers
of shadow and transparency, stops being the ratio of two flat colors; text over a
gradient fill or an image has no single contrast value. Whether color is a single channel
is a question of which channels the information is given through, and it does not exist
as a declaration in the document.

## The Order of Manual Verification

Manual verification is not an unbounded review; it is five steps ordered by cost. The
ordering rests on the cheap steps producing the most findings first.

**Keyboard pass.** The tab key is used to go from the beginning of the page to the end.
Four things are checked: is focus visible at every stop, does the order match the order
on screen, can every opened component be exited by keyboard, and where does focus return
on close. This step alone finds three of the eight missed findings and requires no
tooling.

**Resize check.** Text is enlarged to double size and the interface is used to complete
the same task. Clipped text, overlapping boxes, and controls that disappear show up at
this step.

**Markup reading.** The markup a component produces is read: is the role correct, is the
state updated, which source does the name come from. This step finds defects that require
a judgment of meaning.

**Navigation plane review.** The landmark list, heading outline, and link list are each
extracted separately and read. Does the list make sense stripped of its surrounding
context?

**Task walkthrough.** A task is completed end to end without seeing the interface and
without using a mouse — filtering the measurement list, reporting a correction, confirming
the result. This is the most expensive step, and it finds what the other four cannot:
whether the information is **sufficient**.

## Tying the Audit to the Process

A one-time audit becomes stale at the next change. The way the audit ties into the
process has three layers.

At the component scale, the role and accessible name a component produces are locked in
with a test: when the component renders, the button role and the expected name must be
present in the tree. These tests look at the counterpart in the tree, not at the markup
structure; they stay valid even when the component's internals are rewritten. The same
approach is the foundation of the user-centric queries in the Testing and Monitoring
topic.

At the page scale, the automated checker runs before release, and every failure it finds
blocks the release. The budget form from the Accessibility Standards lesson applies here:
the count of unmet A and AA criteria must be zero, and known exceptions sit in an open
list with the criterion number and a date.

At the release scale, the five steps of manual verification are applied not to every
release but to the parts of the interface that change. When a new component is added, the
keyboard pass and markup reading are mandatory; the task walkthrough is done less often.

## What Zero Findings Means

Automated auditing producing no findings does not show that the interface is accessible.
This sentence is this lesson's conclusion, and it has two sides.

First, the computed rate: on this interface, 55.6 percent of findings were caught, and
the rest were not. Second, the checker can also produce a **false positive**. Text over a
gradient fill can be reported below the threshold by assuming a single background color;
a correctly built component can be flagged because of a pattern the checker does not
recognize. Every finding needs to be confirmed by a person.

The value of automated auditing is in catching regressions. It cheaply prevents a
once-fixed defect from coming back, and it reserves manual verification time for the
defects that can only be found by hand.

## Summary

- An automated checker applies four classes of rule: presence, validity, computable
  value, and uniqueness-consistency; in all of them, the decision must be readable from
  the document.
- Most criteria are in the partial class, and a partial criterion's automated audit can
  only report failure, never success.
- Of the eighteen real defects found in this course, ten are caught automatically and
  eight are missed; the misses require a judgment of meaning, a sequence over time, or
  the rendered result.
- Manual verification is ordered by cost: keyboard pass, resize check, markup reading,
  navigation plane review, and task walkthrough.
- The audit ties into the process at three scales: role and name testing at the component
  scale, an automated check that blocks release at the page scale, and manual
  verification applied to changed parts at the release scale.
- Zero findings from automated auditing is not proof of conformance; the checker also
  produces false positives, and its real value is catching regressions.

## Next Step

Two axes of quality are now measurable. Performance is tied to a budget, accessibility to
a set of criteria; both are countable, and both carry a definition of regression. A third
axis remains, and it asks a different question than the previous two.

Accessibility makes sure that the information an interface carries reaches the user. The
next topic takes up the flow in the opposite direction: text coming from the user
entering the interface. Text written into the description field on the measurement entry
form is shown to other people in the measurement list. When that text is written onto the
page, is it processed **as data** or **as code**? Where that distinction is lost, the
user has to be protected from the interface itself. The first lesson of the Client-Side
Security topic, Cross-Site Scripting, takes up how that distinction gets lost and how
output escaping restores it.