Usability Testing

The heuristic evaluation produced twelve candidates, and each is a hypothesis. Which ones actually stop a real user, and which go entirely unnoticed, cannot be known without measuring it. The way to measure this is to put the prototype in a participant’s hands and give them a task.

A usability test is observing a participant as they try to complete a predetermined task. Its purpose is not collecting opinions; what is measured is where the user pauses, where they click, and where they give up. This lesson takes on how the session is set up, how many participants it requires, and what a small sample can and cannot say, by computation.

Setting Up the Session

The session has three parts, and each part has its own rule.

Task scenario. The task is given to the participant with a context, not an instruction. “Press the Borrow button” is an instruction, and it measures nothing. “A friend recommended a book to you; find out whether the library has it, and if so, how to get it” is a scenario; it requires the user to find their own way. The scenario does not use the interface’s own words — if the word “narrowing” appears in the scenario, whether that word is understood cannot be measured.

Think-aloud. The participant is asked to say out loud what they are doing and what they expect. This surfaces what observation alone cannot: we learn that the user saw an item but did not understand it, or was looking for something not on the screen, only from what they say. Think-aloud lengthens the duration; so if duration is to be measured, a separate task is run in silence.

What the moderator does not do. The moderator does not point the way, does not give hints, does not say “over there.” When the participant gets stuck, the moderator waits; when they truly stop, the task is ended, and this is a finding. The moderator also does not defend the design: the sentence “actually, we did put it there” silences the participant and devalues the rest of the session.

The Test’s Ethical Framework

A usability test is a session where the participant’s failure counts as data; this makes it, ethically, one of the forms of research that demands the most care.

What is being tested is the interface, not the participant. This sentence is said explicitly at the start of the session, and it is not left as a formality: when a task cannot be completed, the participant is reminded that this is a finding about the interface. The participant feeling that they are being tested is both an ethical problem and something that corrupts the data.

Consent is detailed and reversible. The participant gives consent knowing what is being recorded — screen, audio, click trail — and who will see the recording. They can stop the session at any moment and do not have to justify it. If they ask for their recording to be deleted at the end of the session, it is deleted.

Anonymization and the retention limit. Findings are written with codes like P01. If the participant’s account appears in the screen recording, a test account is used instead of a real membership; if that is not possible, identifying fields in the recording are scrubbed before analysis. Once the raw recording has been turned into a coded note, it is deleted within the written retention period.

Using a quote is a separate consent. Putting the participant’s words into a report is a different use from analyzing their recording. If a direct quote is going to be used, this is asked for separately.

The Five-Participant Claim

There is a well-known computation defending small samples. If a problem’s probability of appearing in a single participant is p, its probability of showing up at least once across n participants is 1 - (1 - p)^n. This curve saturates quickly and leads to the conclusion that five participants are enough to find most problems. The computation itself is correct; the assumption it carries is not.

// discovery-curve.mjs — the 1 - (1 - p)^n curve and non-uniform finding probabilities

// Uniform assumption: every problem surfaces with the same probability for every participant
console.log("share of problems found at uniform p");
console.log("participants  p=0.10  p=0.20  p=0.31  p=0.50");
for (const n of [1, 2, 3, 5, 8, 10, 15, 20]) {
  const s = [0.1, 0.2, 0.31, 0.5].map((p) => ((1 - (1 - p) ** n) * 100).toFixed(1).padStart(6));
  console.log(`${String(n).padStart(12)}  ${s.join("  ")}`);
}

// The real problem set is not uniform: some problems show up for everyone, some rarely
const PROBLEMS = [
  { name: "search lost on return", p: 0.90, severity: 4 },
  { name: "shelf code not understood", p: 0.75, severity: 3 },
  { name: "same-titled records confused", p: 0.60, severity: 3 },
  { name: "no loading state", p: 0.50, severity: 2 },
  { name: "same action, different names", p: 0.40, severity: 3 },
  { name: "placeholder instead of label", p: 0.30, severity: 3 },
  { name: "three actions carry equal weight", p: 0.25, severity: 2 },
  { name: "error message has no reason", p: 0.20, severity: 3 },
  { name: "renewal path hidden", p: 0.15, severity: 3 },
  { name: "help is buried", p: 0.10, severity: 1 },
  { name: "no shortcut", p: 0.08, severity: 2 },
  { name: "fee not confirmed beforehand", p: 0.05, severity: 4 },
];
const avgP = PROBLEMS.reduce((t, s) => t + s.p, 0) / PROBLEMS.length;
console.log(`\naverage p of the problem set: ${avgP.toFixed(4)}`);

console.log("\nparticipants  at uniform p  actual set  severity-weighted  lowest problem's probability");
const severityTotal = PROBLEMS.reduce((t, s) => t + s.severity, 0);
for (const n of [1, 3, 5, 8, 12, 20, 32]) {
  const uniform = (1 - (1 - avgP) ** n) * 100;
  const actual = (PROBLEMS.reduce((t, s) => t + (1 - (1 - s.p) ** n), 0) / PROBLEMS.length) * 100;
  const weighted = (PROBLEMS.reduce((t, s) => t + s.severity * (1 - (1 - s.p) ** n), 0) / severityTotal) * 100;
  const lowest = Math.min(...PROBLEMS.map((s) => 1 - (1 - s.p) ** n)) * 100;
  console.log(
    `${String(n).padStart(12)}  ${uniform.toFixed(1).padStart(11)}%  ${actual.toFixed(1).padStart(10)}%  ${weighted.toFixed(1).padStart(17)}%  ${lowest.toFixed(1).padStart(37)}%`
  );
}

// Each problem's probability of being found with 5 participants
console.log("\nproblem                           p     severity  found with 5 participants");
for (const s of PROBLEMS) {
  const b = (1 - (1 - s.p) ** 5) * 100;
  console.log(`${s.name.padEnd(33)} ${s.p.toFixed(2)}  ${String(s.severity).padStart(8)}  ${b.toFixed(1).padStart(26)}%`);
}

// How many participants are needed to find each problem with at least 80% probability
const rarest = PROBLEMS.reduce((a, b) => (a.p < b.p ? a : b));
const needed = Math.ceil(Math.log(0.2) / Math.log(1 - rarest.p));
console.log(`\nrarest problem "${rarest.name}" (p = ${rarest.p}): ${needed} participants needed for an 80% chance of finding it`);

share of problems found at uniform p
participants  p=0.10  p=0.20  p=0.31  p=0.50
           1    10.0    20.0    31.0    50.0
           2    19.0    36.0    52.4    75.0
           3    27.1    48.8    67.1    87.5
           5    41.0    67.2    84.4    96.9
           8    57.0    83.2    94.9    99.6
          10    65.1    89.3    97.6    99.9
          15    79.4    96.5    99.6   100.0
          20    87.8    98.8    99.9   100.0

average p of the problem set: 0.3567

participants  at uniform p  actual set  severity-weighted  lowest problem's probability
           1         35.7%        35.7%               38.7%                                    5.0%
           3         73.4%        61.0%               63.3%                                   14.3%
           5         89.0%        72.3%               73.9%                                   22.6%
           8         97.1%        81.4%               82.2%                                   33.7%
          12         99.5%        87.9%               88.1%                                   46.0%
          20        100.0%        94.0%               93.7%                                   64.2%
          32        100.0%        97.5%               97.1%                                   80.6%

problem                           p     severity  found with 5 participants
search lost on return             0.90         4                       100.0%
shelf code not understood         0.75         3                        99.9%
same-titled records confused      0.60         3                        99.0%
no loading state                  0.50         2                        96.9%
same action, different names      0.40         3                        92.2%
placeholder instead of label      0.30         3                        83.2%
three actions carry equal weight  0.25         2                        76.3%
error message has no reason       0.20         3                        67.2%
renewal path hidden               0.15         3                        55.6%
help is buried                    0.10         1                        41.0%
no shortcut                       0.08         2                        34.1%
fee not confirmed beforehand      0.05         4                        22.6%

rarest problem "fee not confirmed beforehand" (p = 0.05): 32 participants needed for an 80% chance of finding it

Reading the Curve

The uniform assumption is what keeps the claim standing. If every problem is assumed to have the same p value, five participants find 84.4% of the problems at p = 0.31. This computation is done correctly, and the result is internally consistent.

The real problem set is not uniform. The sample set’s average p value is 0.3567; if treated as uniform, expected coverage at five participants is 89.0%. In the real set, which carries the same average but a wide spread, expected coverage is 72.3%. The 16.7-point gap comes entirely from the spread: high-p problems already show up with the first participant and pull the curve upward, while low-p problems approach very slowly as participants are added. Computing from the average overcounts the easy-to-find ones.

The problem that gets missed is not chosen at random. The last table ranks problems by their probability of being found, and at the very bottom sits a severity-4 problem: the fee not being confirmed beforehand. Its probability of being found with five participants is 22.6%. This was also the problem only one person found in the heuristic evaluation in the previous lesson. Both methods miss the same problem for the same reason: rare, high-consequence problems cannot be reliably found by any frequency-based method. These are searched for with structural methods, such as flow-diagram auditing and criteria review.

Weighting by severity does not save the table. Weighted coverage at five participants is 73.9%, very close to unweighted coverage. Because some severe problems are common and some are rare, the weighting balances out. In an interface where severe problems happen to be especially rare, weighted coverage would fall below unweighted.

Finding every problem is expensive. Finding the rarest problem with 80% probability requires thirty-two participants. This number cannot be met in most conditions, and it does not need to be. The correct conclusion is this: small-sample testing is a method that finds common problems cheaply and is good for fixing what it finds; it does not prove the absence of what it does not find. The sentence “we tested with five users and no problems came up” is not evidence.

In practice, this translates into running the test as successive small rounds rather than one large session: five participants, fixes, five new participants. Because each round clears out the most common problems, rarer problems become visible in the next round.

The Limit of Measuring Duration in a Small Sample

The test’s second output is duration measurement, and here the limit of a small sample is harsher.

// duration-measurement.mjs — median duration and its uncertainty in a small sample

// Five participants' screen time on the "find the known record and borrow it" task (seconds)
const DURATIONS = [42, 55, 61, 78, 154];

const sorted = [...DURATIONS].sort((a, b) => a - b);
const mean = DURATIONS.reduce((a, b) => a + b, 0) / DURATIONS.length;
const median = sorted[(sorted.length - 1) / 2];
const geometric = Math.exp(DURATIONS.reduce((t, x) => t + Math.log(x), 0) / DURATIONS.length);

console.log(`measured durations : ${sorted.join(", ")} s`);
console.log(`arithmetic mean    : ${mean.toFixed(1)} s`);
console.log(`geometric mean     : ${geometric.toFixed(1)} s`);
console.log(`median             : ${median.toFixed(1)} s`);
console.log(`mean / median      : ${(mean / median).toFixed(2)}`);
const largest = Math.max(...DURATIONS);
console.log(`mean with the largest value (${largest} s) removed: ${(DURATIONS.filter((x) => x !== largest).reduce((a, b) => a + b, 0) / 4).toFixed(1)} s`);

// Distribution-free confidence interval for the median, from order statistics
const binom = (n, k) => {
  let r = 1;
  for (let i = 0; i < k; i++) r = (r * (n - i)) / (i + 1);
  return r;
};
const coverage = (n, i, j) => {
  let t = 0;
  for (let k = i; k < j; k++) t += binom(n, k);
  return t / 2 ** n;
};

console.log("\nfor n = 5, probability that an order-statistic interval covers the median");
for (const [i, j] of [[1, 5], [2, 4]]) {
  console.log(
    `  [order ${i}, order ${j}] = [${sorted[i - 1]}, ${sorted[j - 1]}] s  ->  ${(coverage(5, i, j) * 100).toFixed(2)}%`
  );
}

// The narrowest order-statistic interval giving at least 90% coverage
const narrowestInterval = (n) => {
  for (let g = 1; g < n; g++)
    for (let i = 1; i + g <= n; i++)
      if (coverage(n, i, i + g) >= 0.9) return [i, i + g, g, coverage(n, i, i + g)];
  return null;
};
console.log("\nparticipants  narrowest interval at 90% coverage  orders spanned  share of sample");
for (const n of [5, 8, 12, 20, 30, 50]) {
  const a = narrowestInterval(n);
  console.log(
    `${String(n).padStart(12)}  ${`[order ${a[0]}, order ${a[1]}]`.padStart(33)}  ${String(a[2] + 1).padStart(15)}  ${(((a[2] + 1) / n) * 100).toFixed(1).padStart(16)}%`
  );
}

measured durations : 42, 55, 61, 78, 154 s
arithmetic mean    : 78.0 s
geometric mean     : 70.1 s
median             : 61.0 s
mean / median      : 1.28
mean with the largest value (154 s) removed: 59.0 s

for n = 5, probability that an order-statistic interval covers the median
  [order 1, order 5] = [42, 154] s  ->  93.75%
  [order 2, order 4] = [55, 78] s  ->  62.50%

participants  narrowest interval at 90% coverage  orders spanned  share of sample
           5                 [order 1, order 5]                5             100.0%
           8                 [order 2, order 7]                6              75.0%
          12                 [order 3, order 9]                7              58.3%
          20                [order 6, order 14]                9              45.0%
          30               [order 11, order 20]               10              33.3%
          50               [order 19, order 31]               13              26.0%

Task durations are right-skewed: the lower bound is the time it physically takes to do the task, and there is no upper bound. When one participant gets stuck and spends 154 seconds, the arithmetic mean rises to 78 seconds; this value is larger than four of the five participants and represents none of them. Removing that single value brings the mean down to 59 seconds. The median is 61 seconds and is not affected by a single outlier. The median is used as the duration metric.

The second table gives the real warning. With five participants, the interval that covers the true median with 93.75% probability is the entire sample: between 42 and 154 seconds. A narrower interval — 55 to 78 seconds — covers only 62.5%. The 61-second median that comes out of five participants is not a measurement but the midpoint of a very wide interval.

The last table shows how this narrows. At thirty participants, the interval covering 90% spans the middle ten of the sorted values — a third of the sample; at fifty participants, a quarter. If a duration comparison is going to be made, the sample it requires is an order of magnitude larger than what is needed to find problems.

The practical consequence of this is a hard distinction: small-sample testing finds problems, it does not produce numbers. A five-participant session does not yield the sentence “task duration is 61 seconds”; it yields “four of the five participants finished in under a minute, one got stuck searching for the shelf code.” The second sentence is both true and produces a decision.

Summary

• A usability test does not collect opinions; a task scenario is given, think-aloud is requested, and the moderator does not point the way or defend the design.
• What is tested is the interface; consent is detailed and reversible, findings are written with codes, the raw recording is deleted within the retention limit, and using a quote is asked for separately.
• The discovery curve assumes a uniform finding probability; in the sample set, the uniform computation with the same average gave 89.0% at five participants, the real distribution gave 72.3%.
• Missed problems are not random: rare and severe problems cannot be found by frequency-based methods and are searched for with structural audits; one of the sample set’s most severe problems shows up with only 22.6% probability at five participants.
• Small-sample testing finds common problems cheaply and is run in successive rounds; no findings turning up does not prove there is no problem.
• The duration metric is the median; at five participants, the interval that covers the true median with 93.75% probability is the entire sample, so small-sample testing finds problems, it does not produce numbers.

Next Step

The test shows where the user gets stuck. The participant who got stuck searching for the shelf code is seen looking at that information on the screen but not using it; the participant who picked the wrong one of two same-titled volumes is seen not reading the full title. Observation records both but explains neither. Why did the user not use the information they looked at, why did they choose without reading? These are not carelessness; they are questions about how the decision was made. The next topic’s first lesson, Dual-Process Thinking, distinguishes whether decisions are made through a fast, intuitive process or a slow, deliberate one, and asks which the interface speaks to.