User Research Methods

The Fundamentals of Interface Design course measured the inside of a screen: hierarchy, grid, spacing, contrast thresholds, component states. In that course, the catalog interface’s flow — search field, result list, record detail, borrowing — was given from the start, and why that flow existed was never asked. This course opens with that question. What gets measured is no longer the inside of a screen but the path between screens.

The start of that path is an assumption: knowing the tasks the interface must support. This knowledge cannot be derived from the designer’s own use; because the designer knows how the interface works, they hold a position no user occupies. The source of this knowledge is research. This lesson separates three basic methods — observation, interviews, and surveys — by the questions they answer, and shows with numbers what each can and cannot say.

Three Methods, Three Separate Questions

Choosing a method is not a budget decision; it is a decision about the type of question. The three methods answer three separate questions, and one method’s answer does not substitute for another’s.

Observation records how a person actually performs a task. In the catalog interface, which row of the result list a user looks at after typing a search term, what they do when they go back, where they go when they cannot find something on the shelf — these are seen through observation. The power of observation is that it captures behavior the person does not state and is often not even aware of. Its limit is that it gives no reason: why the user went back cannot be observed, only asked.

Interviews collect the reason and the context. An interview uncovers why a person performs a task, which alternatives they tried, and which constraints they carry independent of the interface. Its limit has two layers. First, people misremember their own behavior; someone who says “I usually search by subject” may search by author’s name in observation. Second, interviews are conducted with a small number of people and do not produce a numerical prevalence estimate.

Surveys measure how widespread a phenomenon is. A survey can only measure among options already known: a problem you did not put in the survey does not come out of the survey. That is why a survey belongs after discovery, not at the start of research.

The order follows from this: observation and interviews find what exists, and a survey measures how widespread what was found is. Research conducted in reverse order only confirms the options the researcher already had in mind.

The Ethical Framework of Research

Research means collecting data from people, and these three rules are not decoration on the method — they are part of it.

Informed consent. At the start of the session, the participant learns what the data is being collected for, how long it will take, in what form the recording will be kept, and that they can leave at any time, and gives explicit consent to this. The scope of consent cannot be expanded afterward: consent obtained for an interview does not cover using the recording in promotional text.

Anonymization. When notes are coded, a code such as P01 is used instead of the participant’s name, and the mapping between name and code is kept separate from the analysis data. The measure of anonymization is that identity cannot be traced back from the data; “I deleted the name” is not enough. Details in free-text quotes that could identify a person on their own are also removed. In the catalog example, “the only night-shift attendant on the third floor” is a name.

Retention limit. A retention period is written for each dataset before the research begins, and the data is deleted once the period expires. A raw recording is converted into a coded note as soon as possible; the coded note is kept, the raw recording is deleted. Retention with no limit produces a pile of data whose future use is unknown and effectively extends the scope of consent indefinitely.

These rules do not slow research down; they define its scope. Data collected without consent already does not exist, since it cannot be used in analysis.

How Many Interviews Are Enough

The measure of interview count is not representativeness but saturation: the point at which new interviews stop producing new codes. A code is a short name for a phenomenon within a note; the same phenomenon receives the same code even when different participants describe it in different words.

// saturation.mjs — coding interview notes, saturation, and code frequency

// Coded notes from 12 interviews (data constructed for this lesson)
const INTERVIEWS = [
  ["P01", ["same-name-record", "shelf-code-unclear", "borrow-status-delayed", "subject-browsing-hard"]],
  ["P02", ["shelf-code-unclear", "search-lost-on-return", "same-name-record"]],
  ["P03", ["borrow-status-delayed", "filter-not-visible", "same-name-record", "author-spelling-mismatch"]],
  ["P04", ["subject-browsing-hard", "search-lost-on-return", "shelf-code-unclear"]],
  ["P05", ["renewal-path-unknown", "borrow-status-delayed", "same-name-record"]],
  ["P06", ["filter-not-visible", "narrow-screen-list", "shelf-code-unclear", "subject-browsing-hard"]],
  ["P07", ["same-name-record", "author-spelling-mismatch", "search-lost-on-return"]],
  ["P08", ["reservation-order-unclear", "borrow-status-delayed", "shelf-code-unclear"]],
  ["P09", ["subject-browsing-hard", "same-name-record", "narrow-screen-list"]],
  ["P10", ["shelf-code-unclear", "borrow-status-delayed", "search-lost-on-return", "filter-not-visible"]],
  ["P11", ["same-name-record", "subject-browsing-hard", "renewal-path-unknown"]],
  ["P12", ["borrow-status-delayed", "shelf-code-unclear", "author-spelling-mismatch"]],
];

// Saturation: how many new codes each interview brings
const seen = new Set();
console.log("interview  codes  new codes  total unique codes");
for (const [id, codes] of INTERVIEWS) {
  const fresh = codes.filter((c) => !seen.has(c));
  fresh.forEach((c) => seen.add(c));
  console.log(
    `${id.padEnd(8)} ${String(codes.length).padStart(3)} ${String(fresh.length).padStart(9)} ${String(seen.size).padStart(21)}`
  );
}

// Code frequency: how many participants' notes a code appeared in
const frequency = new Map();
for (const [, codes] of INTERVIEWS) {
  for (const c of new Set(codes)) frequency.set(c, (frequency.get(c) ?? 0) + 1);
}
const N = INTERVIEWS.length;
console.log("\ncode                          participants  share");
for (const [c, n] of [...frequency].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))) {
  console.log(`${c.padEnd(28)} ${String(n).padStart(12)}  ${((n / N) * 100).toFixed(1)}%`);
}
console.log(`\ntotal unique codes: ${frequency.size}, last new code: in interview ${INTERVIEWS.length - 4}`);

interview  codes  new codes  total unique codes
P01        4         4                     4
P02        3         1                     5
P03        4         2                     7
P04        3         0                     7
P05        3         1                     8
P06        4         1                     9
P07        3         0                     9
P08        3         1                    10
P09        3         0                    10
P10        4         0                    10
P11        3         0                    10
P12        3         0                    10

code                          participants  share
same-name-record                        7  58.3%
shelf-code-unclear                      7  58.3%
borrow-status-delayed                   6  50.0%
subject-browsing-hard                   5  41.7%
search-lost-on-return                   4  33.3%
author-spelling-mismatch                3  25.0%
filter-not-visible                      3  25.0%
narrow-screen-list                      2  16.7%
renewal-path-unknown                    2  16.7%
reservation-order-unclear               1  8.3%

total unique codes: 10, last new code: in interview 8

The first table shows saturation. The first four interviews brought seven of the ten codes; after the eighth interview, no new code appeared. The last four interviews added no new phenomenon to the analysis.

This cannot be turned into a rule of the form “eight interviews are enough.” The saturation point depends on the diversity of participants: if all twelve people had been drawn from the same department, the curve would flatten earlier, and that early flattening would not be sufficiency — it would be the narrowness of the sample. Saturation is a stopping criterion, not a statement about sample size. In practice, this means: stop as long as new interviews bring no new code, but try once more with a changed participant profile.

Interview Proportions Are Not Prevalence

The second table contains a trap. The sentence “58.3% of participants mentioned the same-name-record problem” is true but misleading; this proportion reads like an estimate about a whole user population. It is necessary to calculate how much uncertainty the proportion carries.

// confidence.mjs — the limits of interview counts as a prevalence estimate

const N = 12; // number of interviews
const Z = 1.96; // 95% confidence level
const OBSERVED = [
  ["same-name-record", 7],
  ["borrow-status-delayed", 6],
  ["reservation-order-unclear", 1],
];

const wald = (count, n) => {
  const p = count / n;
  const margin = Z * Math.sqrt((p * (1 - p)) / n);
  return [p - margin, p + margin];
};
const wilson = (count, n) => {
  const p = count / n;
  const denom = 1 + (Z * Z) / n;
  const center = (p + (Z * Z) / (2 * n)) / denom;
  const margin = (Z / denom) * Math.sqrt((p * (1 - p)) / n + (Z * Z) / (4 * n * n));
  return [center - margin, center + margin];
};

console.log("code                          observed  Wald interval    Wilson interval  width");
for (const [name, c] of OBSERVED) {
  const [wa, wb] = wald(c, N);
  const [sa, sb] = wilson(c, N);
  console.log(
    `${name.padEnd(28)} ${`${c}/${N}`.padStart(8)}  ` +
      `${`${(wa * 100).toFixed(1)} - ${(wb * 100).toFixed(1)}`.padEnd(16)} ` +
      `${`${(sa * 100).toFixed(1)} - ${(sb * 100).toFixed(1)}`.padEnd(16)} ` +
      `${((sb - sa) * 100).toFixed(1)} points`
  );
}

console.log("\nsample size  95% margin of error (p = 0.5)  45% vs 55% distinguishable");
for (const n of [8, 12, 30, 100, 400, 1000]) {
  const margin = Z * Math.sqrt(0.25 / n);
  console.log(
    `${String(n).padStart(11)}  ${`+/- ${(margin * 100).toFixed(1)} points`.padStart(27)}  ${margin * 100 < 5 ? "yes" : "no"}`
  );
}
console.log(`\nsmallest sample needed for +/- 5 points: ${Math.ceil((Z / 0.05) ** 2 * 0.25)}`);

code                          observed  Wald interval    Wilson interval  width
same-name-record                 7/12  30.4 - 86.2      32.0 - 80.7      48.7 points
borrow-status-delayed            6/12  21.7 - 78.3      25.4 - 74.6      49.2 points
reservation-order-unclear        1/12  -7.3 - 24.0      1.5 - 35.4       33.9 points

sample size  95% margin of error (p = 0.5)  45% vs 55% distinguishable
          8              +/- 34.6 points  no
         12              +/- 28.3 points  no
         30              +/- 17.9 points  no
        100               +/- 9.8 points  no
        400               +/- 4.9 points  yes
       1000               +/- 3.1 points  yes

smallest sample needed for +/- 5 points: 385

The interval for the problem seven people mentioned runs from 32% to 81%; its width is 48.7 points. That interval contains both the claim “more than half the users” and the claim “a quarter of the users.” A proportion drawn from twelve interviews cannot be used as an estimate.

The Wald interval producing a negative lower bound on the third row is also instructive: the normal approximation loses validity at small sample sizes and extreme proportions, giving an impossible value for a probability. The Wilson interval places the same data between 1.5% and 35.4%. The only thing that can be said about the prevalence of a phenomenon observed once is that it is not zero.

The second table gives the price of a survey. In a hundred-person survey, the margin of error is ±9.8 points, and it cannot distinguish a gap between 45% and 55%; reaching ±5 points of precision requires at least 385 responses. Quadrupling the sample size halves the margin of error — precision improves at the rate of the square root of the sample.

The Method Selection Rule

The calculations above reduce to a single selection rule.

• If the question is “how do people do this,” observation is chosen; what is done is measured, not what is said.
• If the question is “why do they do it this way” or “under which constraints,” an interview is chosen; its output is coded phenomena, not a proportion.
• If the question is “how widespread is this,” a survey is chosen and the required sample size is calculated up front; if the calculation cannot be met, the survey is not run, because a small survey does not answer.

The common rule across all three is that the question is written before the method. A study that starts with the method fits itself to whatever question that method can answer.

Summary

• Observation measures what is done, an interview measures the reason, a survey measures prevalence; discovery is done with observation and interviews, measurement with a survey, and the order cannot be reversed.
• Informed consent, anonymization, and a retention limit are part of the method; the scope of consent cannot be expanded afterward, and a raw recording is deleted once it is coded.
• The measure of interview count is saturation; in the sample data, no new code appeared after the eighth interview, but saturation depends on participant diversity and does not yield a fixed number.
• The 95% interval for a 58.3% proportion drawn from twelve interviews runs from 32% to 81%; a qualitative sample does not produce a prevalence estimate.
• Survey precision improves with the square root of the sample: at least 385 responses are needed for ±5 points, and a hundred-person survey cannot distinguish gaps of ten points.

Next Step

This lesson pulled ten codes out of twelve interviews but still left the participants as nothing more than a list of P01–P12 codes. Making a design decision requires seeing the structure of that list: do some participants resemble each other, and if so, along which axis? The next lesson clusters participants by their characteristics, shows with calculation that grouping by behavior does not give the same result as grouping by demographics, and sets out how a persona grounded in real data is built.