--- title: 'User Research Methods' source: 'https://academia.sh/en/courses/user-experience/user-research-methods' course: 'User Experience and Behavior Design' language: en updated: '2026-08-19T05:19:59+00:00' license: 'CC BY-SA 4.0' --- # User Research Methods What question observation, interviews, and surveys each answer; measuring interview saturation, the limit of inferring prevalence from a small sample, and the ethical framework of research. The Fundamentals of Interface Design course measured the inside of a screen: hierarchy, grid, spacing, contrast thresholds, component states. In that course, the catalog interface's flow — search field, result list, record detail, borrowing — was given from the start, and why that flow existed was never asked. This course opens with that question. What gets measured is no longer the inside of a screen but the path between screens. The start of that path is an assumption: knowing the tasks the interface must support. This knowledge cannot be derived from the designer's own use; because the designer knows how the interface works, they hold a position no user occupies. The source of this knowledge is research. This lesson separates three basic methods — observation, interviews, and surveys — by the questions they answer, and shows with numbers what each can and cannot say. ## Three Methods, Three Separate Questions Choosing a method is not a budget decision; it is a decision about the type of question. The three methods answer three separate questions, and one method's answer does not substitute for another's. **Observation** records how a person actually performs a task. In the catalog interface, which row of the result list a user looks at after typing a search term, what they do when they go back, where they go when they cannot find something on the shelf — these are seen through observation. The power of observation is that it captures behavior the person **does not state** and is often not even aware of. Its limit is that it gives no reason: why the user went back cannot be observed, only asked. **Interviews** collect the reason and the context. An interview uncovers why a person performs a task, which alternatives they tried, and which constraints they carry independent of the interface. Its limit has two layers. First, people misremember their own behavior; someone who says "I usually search by subject" may search by author's name in observation. Second, interviews are conducted with a small number of people and **do not produce a numerical prevalence estimate**. **Surveys** measure how widespread a phenomenon is. A survey can only measure among options already known: a problem you did not put in the survey does not come out of the survey. That is why a survey belongs after discovery, not at the start of research. The order follows from this: observation and interviews find what exists, and a survey measures how widespread what was found is. Research conducted in reverse order only confirms the options the researcher already had in mind. ## The Ethical Framework of Research Research means collecting data from people, and these three rules are not decoration on the method — they are part of it. **Informed consent.** At the start of the session, the participant learns what the data is being collected for, how long it will take, in what form the recording will be kept, and that they can leave at any time, and gives explicit consent to this. The scope of consent cannot be expanded afterward: consent obtained for an interview does not cover using the recording in promotional text. **Anonymization.** When notes are coded, a code such as `P01` is used instead of the participant's name, and the mapping between name and code is kept separate from the analysis data. The measure of anonymization is that identity cannot be traced back from the data; "I deleted the name" is not enough. Details in free-text quotes that could identify a person on their own are also removed. In the catalog example, "the only night-shift attendant on the third floor" is a name. **Retention limit.** A retention period is written for each dataset before the research begins, and the data is deleted once the period expires. A raw recording is converted into a coded note as soon as possible; the coded note is kept, the raw recording is deleted. Retention with no limit produces a pile of data whose future use is unknown and effectively extends the scope of consent indefinitely. These rules do not slow research down; they define its scope. Data collected without consent already does not exist, since it cannot be used in analysis. ## How Many Interviews Are Enough The measure of interview count is not representativeness but **saturation**: the point at which new interviews stop producing new codes. A code is a short name for a phenomenon within a note; the same phenomenon receives the same code even when different participants describe it in different words. ```js // saturation.mjs — coding interview notes, saturation, and code frequency // Coded notes from 12 interviews (data constructed for this lesson) const INTERVIEWS = [ ["P01", ["same-name-record", "shelf-code-unclear", "borrow-status-delayed", "subject-browsing-hard"]], ["P02", ["shelf-code-unclear", "search-lost-on-return", "same-name-record"]], ["P03", ["borrow-status-delayed", "filter-not-visible", "same-name-record", "author-spelling-mismatch"]], ["P04", ["subject-browsing-hard", "search-lost-on-return", "shelf-code-unclear"]], ["P05", ["renewal-path-unknown", "borrow-status-delayed", "same-name-record"]], ["P06", ["filter-not-visible", "narrow-screen-list", "shelf-code-unclear", "subject-browsing-hard"]], ["P07", ["same-name-record", "author-spelling-mismatch", "search-lost-on-return"]], ["P08", ["reservation-order-unclear", "borrow-status-delayed", "shelf-code-unclear"]], ["P09", ["subject-browsing-hard", "same-name-record", "narrow-screen-list"]], ["P10", ["shelf-code-unclear", "borrow-status-delayed", "search-lost-on-return", "filter-not-visible"]], ["P11", ["same-name-record", "subject-browsing-hard", "renewal-path-unknown"]], ["P12", ["borrow-status-delayed", "shelf-code-unclear", "author-spelling-mismatch"]], ]; // Saturation: how many new codes each interview brings const seen = new Set(); console.log("interview codes new codes total unique codes"); for (const [id, codes] of INTERVIEWS) { const fresh = codes.filter((c) => !seen.has(c)); fresh.forEach((c) => seen.add(c)); console.log( `${id.padEnd(8)} ${String(codes.length).padStart(3)} ${String(fresh.length).padStart(9)} ${String(seen.size).padStart(21)}` ); } // Code frequency: how many participants' notes a code appeared in const frequency = new Map(); for (const [, codes] of INTERVIEWS) { for (const c of new Set(codes)) frequency.set(c, (frequency.get(c) ?? 0) + 1); } const N = INTERVIEWS.length; console.log("\ncode participants share"); for (const [c, n] of [...frequency].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))) { console.log(`${c.padEnd(28)} ${String(n).padStart(12)} ${((n / N) * 100).toFixed(1)}%`); } console.log(`\ntotal unique codes: ${frequency.size}, last new code: in interview ${INTERVIEWS.length - 4}`); ``` ``` interview codes new codes total unique codes P01 4 4 4 P02 3 1 5 P03 4 2 7 P04 3 0 7 P05 3 1 8 P06 4 1 9 P07 3 0 9 P08 3 1 10 P09 3 0 10 P10 4 0 10 P11 3 0 10 P12 3 0 10 code participants share same-name-record 7 58.3% shelf-code-unclear 7 58.3% borrow-status-delayed 6 50.0% subject-browsing-hard 5 41.7% search-lost-on-return 4 33.3% author-spelling-mismatch 3 25.0% filter-not-visible 3 25.0% narrow-screen-list 2 16.7% renewal-path-unknown 2 16.7% reservation-order-unclear 1 8.3% total unique codes: 10, last new code: in interview 8 ``` The first table shows saturation. The first four interviews brought seven of the ten codes; after the eighth interview, no new code appeared. The last four interviews added no new phenomenon to the analysis. This cannot be turned into a rule of the form "eight interviews are enough." The saturation point depends on the diversity of participants: if all twelve people had been drawn from the same department, the curve would flatten earlier, and that early flattening would not be sufficiency — it would be the narrowness of the sample. Saturation is a stopping criterion, not a statement about sample size. In practice, this means: stop as long as new interviews bring no new code, but try once more with a changed participant profile. ## Interview Proportions Are Not Prevalence The second table contains a trap. The sentence "58.3% of participants mentioned the same-name-record problem" is true but misleading; this proportion reads like an estimate about a whole user population. It is necessary to calculate how much uncertainty the proportion carries. ```js // confidence.mjs — the limits of interview counts as a prevalence estimate const N = 12; // number of interviews const Z = 1.96; // 95% confidence level const OBSERVED = [ ["same-name-record", 7], ["borrow-status-delayed", 6], ["reservation-order-unclear", 1], ]; const wald = (count, n) => { const p = count / n; const margin = Z * Math.sqrt((p * (1 - p)) / n); return [p - margin, p + margin]; }; const wilson = (count, n) => { const p = count / n; const denom = 1 + (Z * Z) / n; const center = (p + (Z * Z) / (2 * n)) / denom; const margin = (Z / denom) * Math.sqrt((p * (1 - p)) / n + (Z * Z) / (4 * n * n)); return [center - margin, center + margin]; }; console.log("code observed Wald interval Wilson interval width"); for (const [name, c] of OBSERVED) { const [wa, wb] = wald(c, N); const [sa, sb] = wilson(c, N); console.log( `${name.padEnd(28)} ${`${c}/${N}`.padStart(8)} ` + `${`${(wa * 100).toFixed(1)} - ${(wb * 100).toFixed(1)}`.padEnd(16)} ` + `${`${(sa * 100).toFixed(1)} - ${(sb * 100).toFixed(1)}`.padEnd(16)} ` + `${((sb - sa) * 100).toFixed(1)} points` ); } console.log("\nsample size 95% margin of error (p = 0.5) 45% vs 55% distinguishable"); for (const n of [8, 12, 30, 100, 400, 1000]) { const margin = Z * Math.sqrt(0.25 / n); console.log( `${String(n).padStart(11)} ${`+/- ${(margin * 100).toFixed(1)} points`.padStart(27)} ${margin * 100 < 5 ? "yes" : "no"}` ); } console.log(`\nsmallest sample needed for +/- 5 points: ${Math.ceil((Z / 0.05) ** 2 * 0.25)}`); ``` ``` code observed Wald interval Wilson interval width same-name-record 7/12 30.4 - 86.2 32.0 - 80.7 48.7 points borrow-status-delayed 6/12 21.7 - 78.3 25.4 - 74.6 49.2 points reservation-order-unclear 1/12 -7.3 - 24.0 1.5 - 35.4 33.9 points sample size 95% margin of error (p = 0.5) 45% vs 55% distinguishable 8 +/- 34.6 points no 12 +/- 28.3 points no 30 +/- 17.9 points no 100 +/- 9.8 points no 400 +/- 4.9 points yes 1000 +/- 3.1 points yes smallest sample needed for +/- 5 points: 385 ``` The interval for the problem seven people mentioned runs from 32% to 81%; its width is 48.7 points. That interval contains both the claim "more than half the users" and the claim "a quarter of the users." A proportion drawn from twelve interviews cannot be used as an estimate. The Wald interval producing a **negative** lower bound on the third row is also instructive: the normal approximation loses validity at small sample sizes and extreme proportions, giving an impossible value for a probability. The Wilson interval places the same data between 1.5% and 35.4%. The only thing that can be said about the prevalence of a phenomenon observed once is that it is not zero. The second table gives the price of a survey. In a hundred-person survey, the margin of error is ±9.8 points, and it cannot distinguish a gap between 45% and 55%; reaching ±5 points of precision requires at least 385 responses. Quadrupling the sample size halves the margin of error — precision improves at the rate of the square root of the sample. ## The Method Selection Rule The calculations above reduce to a single selection rule. - If the question is "how do people do this," **observation** is chosen; what is done is measured, not what is said. - If the question is "why do they do it this way" or "under which constraints," an **interview** is chosen; its output is coded phenomena, not a proportion. - If the question is "how widespread is this," a **survey** is chosen and the required sample size is calculated up front; if the calculation cannot be met, the survey is not run, because a small survey does not answer. The common rule across all three is that the question is written before the method. A study that starts with the method fits itself to whatever question that method can answer. ## Summary - Observation measures what is done, an interview measures the reason, a survey measures prevalence; discovery is done with observation and interviews, measurement with a survey, and the order cannot be reversed. - Informed consent, anonymization, and a retention limit are part of the method; the scope of consent cannot be expanded afterward, and a raw recording is deleted once it is coded. - The measure of interview count is saturation; in the sample data, no new code appeared after the eighth interview, but saturation depends on participant diversity and does not yield a fixed number. - The 95% interval for a 58.3% proportion drawn from twelve interviews runs from 32% to 81%; a qualitative sample does not produce a prevalence estimate. - Survey precision improves with the square root of the sample: at least 385 responses are needed for ±5 points, and a hundred-person survey cannot distinguish gaps of ten points. ## Next Step This lesson pulled ten codes out of twelve interviews but still left the participants as nothing more than a list of `P01`–`P12` codes. Making a design decision requires seeing the structure of that list: do some participants resemble each other, and if so, along which axis? The next lesson clusters participants by their characteristics, shows with calculation that grouping by behavior does not give the same result as grouping by demographics, and sets out how a persona grounded in real data is built.