Usability Heuristics

The prototype is ready, but a cheap check can be run before calling in participants. A portion of usability problems are found without ever showing the interface to a user, by reviewing it with a known set of criteria in hand. This method does not replace the participant session; it is done first so the session is not spent needlessly.

A usability heuristic is a general criterion used when evaluating an interface. The name “heuristic” comes from its not being a strict rule: when a criterion is not met, the result is not a definite flaw but a candidate that needs examining. This lesson applies an established set of criteria principle by principle and computes how many people’s evaluation is needed.

The Established Set of Criteria

Ten principles long used in the field are read here, matched against the catalog’s decisions.

• Visibility of system status. The user always knows what is happening. A loading state while search runs, a progress indicator during borrowing.
• Match between the system and the real world. The interface speaks the user’s vocabulary, not the institution’s. “Circulation operations” is the institution’s vocabulary; “borrowing” is the user’s.
• User control and freedom. A wrong step can be undone, a transaction can be abandoned partway through, an earlier state can be returned to.
• Consistency and standards. The same thing has the same name and the same form everywhere. This is the structural-level counterpart of the rule set for visuals in the Repetition and Consistency lesson.
• Error prevention. Better than a good error message is an error that never occurs. If a late fee is going to be charged, it is confirmed first.
• Recognition rather than recall. The user does not have to memorize what they saw on a previous screen. Showing the shelf code again at the borrowing step instead of making the user hold it in memory.
• Flexibility and efficiency of use. A shortcut for frequent users, an explicit path for new users; both exist in the same interface.
• Aesthetic and minimalist design. No unnecessary information sits on the screen; every added element reduces the visibility of the information that matters.
• Help users recognize, diagnose, and recover from errors. The message states what happened, why, and what to do about it.
• Help and documentation. If help is needed, it is searchable, short, and task-focused; not needing help at all is better still.

The names of the criteria are not used as templates. The output of an evaluation is not “the consistency principle was violated” but is written as “the borrow action is called Take in the list, Borrow in the record detail, and On Hold on the confirmation screen.” The principle gives the finding’s class; the finding does not stand in for the principle.

Running the Session

The evaluation follows three rules.

Independent. Each evaluator walks through the interface alone and writes findings to their own list. If they walk through it together, the first person to speak steers the others’ attention, and the number of independent findings drops.

Two-pass. The first pass follows the flow as a whole and grasps the overall structure; the second pass scans each screen against the criteria list one by one. An evaluation done in a single pass judges a screen without knowing the flow’s context.

Severity-scored. Every finding gets a severity. The computation below uses a four-level scale: 1 a cosmetic flaw, 2 a minor obstacle, 3 a problem that noticeably hinders the task, 4 a problem that makes the task impossible to complete or produces irreversible damage. The score does the ranking after findings are merged.

One Evaluator Is Not Enough

Five evaluators reviewed the prototype independently, and the findings were merged.

// heuristic-evaluation.mjs — overlap and severity agreement of five evaluators' findings

// Each finding: which principle it belongs to and which evaluator gave it what severity (1 minor, 4 major)
const FINDINGS = [
  { name: "loading state not shown", principle: "status visibility", score: { D1: 2, D2: 3, D3: 2, D4: 2, D5: 3 } },
  { name: "same action, different names", principle: "consistency", score: { D1: 3, D2: 3, D4: 2 } },
  { name: "borrowing cannot be undone", principle: "control and freedom", score: { D2: 4, D3: 3, D5: 4 } },
  { name: "error message does not say why", principle: "error recovery", score: { D1: 3, D3: 4, D4: 3, D5: 3 } },
  { name: "institutional jargon used", principle: "real-world match", score: { D2: 3, D5: 2 } },
  { name: "search lost on return", principle: "control and freedom", score: { D1: 4, D2: 4, D3: 4, D4: 3, D5: 4 } },
  { name: "fee not confirmed beforehand", principle: "error prevention", score: { D3: 4 } },
  { name: "placeholder text used as label", principle: "recognition over recall", score: { D1: 3, D4: 3 } },
  { name: "shelf code is not explained", principle: "real-world match", score: { D2: 3, D3: 3, D4: 4 } },
  { name: "no shortcut to repeat a search", principle: "flexibility and efficiency", score: { D5: 2 } },
  { name: "three actions carry equal weight", principle: "minimalist design", score: { D1: 3, D2: 2 } },
  { name: "help is buried", principle: "help and documentation", score: { D4: 1 } },
];
const EVALUATORS = ["D1", "D2", "D3", "D4", "D5"];

const median = (d) => {
  const s = [...d].sort((a, b) => a - b), o = s.length >> 1;
  return s.length % 2 ? s[o] : (s[o - 1] + s[o]) / 2;
};

console.log("finding                            principle                   found  median  range");
for (const b of FINDINGS) {
  const p = Object.values(b.score);
  console.log(
    `${b.name.padEnd(34)} ${b.principle.padEnd(27)} ${String(p.length).padStart(5)} ${median(p).toFixed(1).padStart(8)}  ${Math.min(...p)}-${Math.max(...p)}`
  );
}

console.log("\nevaluator  findings  coverage");
for (const d of EVALUATORS) {
  const n = FINDINGS.filter((b) => d in b.score).length;
  console.log(`${d.padEnd(11)} ${String(n).padStart(9)}  ${((n / FINDINGS.length) * 100).toFixed(1)}%`);
}

// Expected union over all subsets of k evaluators
const subsets = (arr, k) =>
  k === 0 ? [[]] : arr.flatMap((x, i) => subsets(arr.slice(i + 1), k - 1).map((r) => [x, ...r]));
console.log("\nevaluators  subsets  expected found  expected coverage");
for (let k = 1; k <= EVALUATORS.length; k++) {
  const groups = subsets(EVALUATORS, k);
  const total = groups.reduce(
    (t, group) => t + FINDINGS.filter((b) => group.some((d) => d in b.score)).length, 0);
  const avg = total / groups.length;
  console.log(
    `${String(k).padStart(10)}  ${String(groups.length).padStart(7)}  ${avg.toFixed(2).padStart(14)}  ${((avg / FINDINGS.length) * 100).toFixed(1).padStart(17)}%`
  );
}

// Findings caught by only one person, and severity agreement
const soloFindings = FINDINGS.filter((b) => Object.keys(b.score).length === 1);
console.log(`\nfindings caught by only one evaluator: ${soloFindings.length} / ${FINDINGS.length}  (${((soloFindings.length / FINDINGS.length) * 100).toFixed(1)}%)`);
console.log(`  ${soloFindings.map((b) => b.name).join("; ")}`);

const widelyFound = FINDINGS.filter((b) => Object.keys(b.score).length >= 3);
const spread = widelyFound.map((b) => Math.max(...Object.values(b.score)) - Math.min(...Object.values(b.score)));
console.log(`\nseverity spread on the ${widelyFound.length} findings caught by three or more people: average ${(spread.reduce((a, c) => a + c, 0) / spread.length).toFixed(2)} points, max ${Math.max(...spread)} points`);

// Findings per principle
const principles = new Map();
for (const b of FINDINGS) principles.set(b.principle, (principles.get(b.principle) ?? 0) + 1);
console.log("\nprinciple                    findings");
for (const [i, c] of [...principles].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])))
  console.log(`${i.padEnd(27)} ${c}`);

finding                            principle                   found  median  range
loading state not shown            status visibility               5      2.0  2-3
same action, different names       consistency                     3      3.0  2-3
borrowing cannot be undone         control and freedom             3      4.0  3-4
error message does not say why     error recovery                  4      3.0  3-4
institutional jargon used          real-world match                2      2.5  2-3
search lost on return              control and freedom             5      4.0  3-4
fee not confirmed beforehand       error prevention                1      4.0  4-4
placeholder text used as label     recognition over recall         2      3.0  3-3
shelf code is not explained        real-world match                3      3.0  3-4
no shortcut to repeat a search     flexibility and efficiency      1      2.0  2-2
three actions carry equal weight   minimalist design               2      2.5  2-3
help is buried                     help and documentation          1      1.0  1-1

evaluator  findings  coverage
D1                  6  50.0%
D2                  7  58.3%
D3                  6  50.0%
D4                  7  58.3%
D5                  6  50.0%

evaluators  subsets  expected found  expected coverage
         1        5            6.40               53.3%
         2       10            9.00               75.0%
         3       10           10.50               87.5%
         4        5           11.40               95.0%
         5        1           12.00              100.0%

findings caught by only one evaluator: 3 / 12  (25.0%)
  fee not confirmed beforehand; no shortcut to repeat a search; help is buried

severity spread on the 6 findings caught by three or more people: average 1.00 points, max 1 points

principle                    findings
control and freedom         2
real-world match            2
consistency                 1
error prevention            1
error recovery              1
flexibility and efficiency  1
help and documentation      1
minimalist design           1
recognition over recall     1
status visibility           1

What the Numbers Say

A single evaluator finds about half. Each of the five people found roughly half of the twelve findings; expected coverage over all single-person subsets is 53.3%. Having one person do the evaluation is agreeing in advance not to see about half the problems.

Gains grow with diminishing returns. Expected coverage is 75% with two evaluators, 87.5% with three, 95% with four. Going from one to two gains 21.7 points, going from three to four gains 7.5 points. The evaluator count is chosen between three and four; a fifth person adds only 5 points in this example, and the same effort pays off more in a participant session.

Only one person found the most severe finding. The “fee not confirmed beforehand” finding has a severity of 4 — the highest value on the scale — and only the third evaluator found it. A single evaluator chosen at random from the pool of five has a one-in-five chance of finding it. This is the real risk of working with a single evaluator: a missed finding does not have to be as unimportant as the average of missed findings.

Agreement is good on severity, poor on coverage. For the six findings caught by three or more people, the severity spread averages one point; no one gave the same finding scores ranging from 1 to 4. When evaluators see a problem, they think similarly about its weight; where they diverge is whether they see the problem at all. So what gets discussed at the merge stage is not the scores but the finding missing from the list.

Two principles stand out. Control and freedom, and real-world match, each have two findings; both point to the same place as the problems measured in user research — search lost on return and the unexplained shelf code. When a heuristic evaluation and field data converge on the same finding, evidence strength grows; the multi-source criterion set in the sixth lesson applies here too.

The Limit of Heuristic Evaluation

The method is cheap, and this cheapness is also the source of its limit.

Findings are hypotheses. The evaluator sees the interface not as a user but as an expert walking through it with a criteria list. The “help is buried” finding was given a severity of 1; if real users never look for help, its severity is zero, and if they try to look for it and cannot find it, it is far higher. Only measurement tells us which, not the heuristic evaluation.

It can produce false findings. Walking through with a criteria list breeds an attitude oriented toward hunting violations; every deviation looks like a flaw. A decision that deliberately departs from an established criterion — the differentiator decisions discussed in the fifth lesson — gets flagged as a violation in a heuristic evaluation. This is why findings are reviewed together with their design rationale.

It does not say what the user actually wanted to do. A heuristic evaluation checks the interface’s existing state against criteria. It cannot discover that the user actually wanted to do something else, that the information they were looking for is not in the interface at all, or that the task is entirely unnecessary. These belong to the domain of research and user testing.

Summary

• A usability heuristic is a criterion, not a rule; when it is not met, the result is a candidate to examine, not a definite flaw, and a finding is written as a concrete fact, not the principle’s name.
• The evaluation is run independently, in two passes, and with severity scoring; walking through together lowers the count of independent findings.
• In the sample data, a single evaluator’s expected coverage is 53.3%; it is 87.5% with three evaluators, 95% with four, and the gains diminish.
• Only one of the five evaluators found the highest-severity finding; a missed finding cannot be assumed to be as unimportant as the average.
• Evaluators agree on severity and diverge on coverage; what gets discussed at the merge is not the scores but what is missing from the list.
• A heuristic evaluation produces hypotheses, can yield false findings, and cannot say what the user wanted to do.

Next Step

The heuristic evaluation produced twelve candidates, and each is a hypothesis. Which ones actually stop a real user cannot be known without measuring it. The next lesson puts the prototype in a participant’s hands: it shows how the session is set up, how many participants are enough, and under which conditions the “five users are enough” claim is true and under which it is misleading, by computing the discovery curve.