--- title: 'User Experience and Behavior Design' source: 'https://academia.sh/en/courses/user-experience' language: en updated: '2026-08-19T05:19:56+00:00' license: 'CC BY-SA 4.0' lessons: 24 outcomes: - 'Builds the chain from user research to design decision' - 'Eases task completion through information architecture and flow design' - 'Applies behavior models within ethical limits' --- # User Experience and Behavior Design A course that builds the chain from user research to design decision, turning information architecture, flow design, and behavior models into measurable decisions within ethical limits. ## Research and Definition When observation, interviews, and surveys are each chosen; deriving personas, task lists, and journey maps from data; reading existing solutions and writing the problem before the solution. ### User Research Methods Source: https://academia.sh/en/courses/user-experience/user-research-methods What question observation, interviews, and surveys each answer; measuring interview saturation, the limit of inferring prevalence from a small sample, and the ethical framework of research. The Fundamentals of Interface Design course measured the inside of a screen: hierarchy, grid, spacing, contrast thresholds, component states. In that course, the catalog interface's flow — search field, result list, record detail, borrowing — was given from the start, and why that flow existed was never asked. This course opens with that question. What gets measured is no longer the inside of a screen but the path between screens. The start of that path is an assumption: knowing the tasks the interface must support. This knowledge cannot be derived from the designer's own use; because the designer knows how the interface works, they hold a position no user occupies. The source of this knowledge is research. This lesson separates three basic methods — observation, interviews, and surveys — by the questions they answer, and shows with numbers what each can and cannot say. #### Three Methods, Three Separate Questions Choosing a method is not a budget decision; it is a decision about the type of question. The three methods answer three separate questions, and one method's answer does not substitute for another's. **Observation** records how a person actually performs a task. In the catalog interface, which row of the result list a user looks at after typing a search term, what they do when they go back, where they go when they cannot find something on the shelf — these are seen through observation. The power of observation is that it captures behavior the person **does not state** and is often not even aware of. Its limit is that it gives no reason: why the user went back cannot be observed, only asked. **Interviews** collect the reason and the context. An interview uncovers why a person performs a task, which alternatives they tried, and which constraints they carry independent of the interface. Its limit has two layers. First, people misremember their own behavior; someone who says "I usually search by subject" may search by author's name in observation. Second, interviews are conducted with a small number of people and **do not produce a numerical prevalence estimate**. **Surveys** measure how widespread a phenomenon is. A survey can only measure among options already known: a problem you did not put in the survey does not come out of the survey. That is why a survey belongs after discovery, not at the start of research. The order follows from this: observation and interviews find what exists, and a survey measures how widespread what was found is. Research conducted in reverse order only confirms the options the researcher already had in mind. #### The Ethical Framework of Research Research means collecting data from people, and these three rules are not decoration on the method — they are part of it. **Informed consent.** At the start of the session, the participant learns what the data is being collected for, how long it will take, in what form the recording will be kept, and that they can leave at any time, and gives explicit consent to this. The scope of consent cannot be expanded afterward: consent obtained for an interview does not cover using the recording in promotional text. **Anonymization.** When notes are coded, a code such as `P01` is used instead of the participant's name, and the mapping between name and code is kept separate from the analysis data. The measure of anonymization is that identity cannot be traced back from the data; "I deleted the name" is not enough. Details in free-text quotes that could identify a person on their own are also removed. In the catalog example, "the only night-shift attendant on the third floor" is a name. **Retention limit.** A retention period is written for each dataset before the research begins, and the data is deleted once the period expires. A raw recording is converted into a coded note as soon as possible; the coded note is kept, the raw recording is deleted. Retention with no limit produces a pile of data whose future use is unknown and effectively extends the scope of consent indefinitely. These rules do not slow research down; they define its scope. Data collected without consent already does not exist, since it cannot be used in analysis. #### How Many Interviews Are Enough The measure of interview count is not representativeness but **saturation**: the point at which new interviews stop producing new codes. A code is a short name for a phenomenon within a note; the same phenomenon receives the same code even when different participants describe it in different words. ```js // saturation.mjs — coding interview notes, saturation, and code frequency // Coded notes from 12 interviews (data constructed for this lesson) const INTERVIEWS = [ ["P01", ["same-name-record", "shelf-code-unclear", "borrow-status-delayed", "subject-browsing-hard"]], ["P02", ["shelf-code-unclear", "search-lost-on-return", "same-name-record"]], ["P03", ["borrow-status-delayed", "filter-not-visible", "same-name-record", "author-spelling-mismatch"]], ["P04", ["subject-browsing-hard", "search-lost-on-return", "shelf-code-unclear"]], ["P05", ["renewal-path-unknown", "borrow-status-delayed", "same-name-record"]], ["P06", ["filter-not-visible", "narrow-screen-list", "shelf-code-unclear", "subject-browsing-hard"]], ["P07", ["same-name-record", "author-spelling-mismatch", "search-lost-on-return"]], ["P08", ["reservation-order-unclear", "borrow-status-delayed", "shelf-code-unclear"]], ["P09", ["subject-browsing-hard", "same-name-record", "narrow-screen-list"]], ["P10", ["shelf-code-unclear", "borrow-status-delayed", "search-lost-on-return", "filter-not-visible"]], ["P11", ["same-name-record", "subject-browsing-hard", "renewal-path-unknown"]], ["P12", ["borrow-status-delayed", "shelf-code-unclear", "author-spelling-mismatch"]], ]; // Saturation: how many new codes each interview brings const seen = new Set(); console.log("interview codes new codes total unique codes"); for (const [id, codes] of INTERVIEWS) { const fresh = codes.filter((c) => !seen.has(c)); fresh.forEach((c) => seen.add(c)); console.log( `${id.padEnd(8)} ${String(codes.length).padStart(3)} ${String(fresh.length).padStart(9)} ${String(seen.size).padStart(21)}` ); } // Code frequency: how many participants' notes a code appeared in const frequency = new Map(); for (const [, codes] of INTERVIEWS) { for (const c of new Set(codes)) frequency.set(c, (frequency.get(c) ?? 0) + 1); } const N = INTERVIEWS.length; console.log("\ncode participants share"); for (const [c, n] of [...frequency].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))) { console.log(`${c.padEnd(28)} ${String(n).padStart(12)} ${((n / N) * 100).toFixed(1)}%`); } console.log(`\ntotal unique codes: ${frequency.size}, last new code: in interview ${INTERVIEWS.length - 4}`); ``` ``` interview codes new codes total unique codes P01 4 4 4 P02 3 1 5 P03 4 2 7 P04 3 0 7 P05 3 1 8 P06 4 1 9 P07 3 0 9 P08 3 1 10 P09 3 0 10 P10 4 0 10 P11 3 0 10 P12 3 0 10 code participants share same-name-record 7 58.3% shelf-code-unclear 7 58.3% borrow-status-delayed 6 50.0% subject-browsing-hard 5 41.7% search-lost-on-return 4 33.3% author-spelling-mismatch 3 25.0% filter-not-visible 3 25.0% narrow-screen-list 2 16.7% renewal-path-unknown 2 16.7% reservation-order-unclear 1 8.3% total unique codes: 10, last new code: in interview 8 ``` The first table shows saturation. The first four interviews brought seven of the ten codes; after the eighth interview, no new code appeared. The last four interviews added no new phenomenon to the analysis. This cannot be turned into a rule of the form "eight interviews are enough." The saturation point depends on the diversity of participants: if all twelve people had been drawn from the same department, the curve would flatten earlier, and that early flattening would not be sufficiency — it would be the narrowness of the sample. Saturation is a stopping criterion, not a statement about sample size. In practice, this means: stop as long as new interviews bring no new code, but try once more with a changed participant profile. #### Interview Proportions Are Not Prevalence The second table contains a trap. The sentence "58.3% of participants mentioned the same-name-record problem" is true but misleading; this proportion reads like an estimate about a whole user population. It is necessary to calculate how much uncertainty the proportion carries. ```js // confidence.mjs — the limits of interview counts as a prevalence estimate const N = 12; // number of interviews const Z = 1.96; // 95% confidence level const OBSERVED = [ ["same-name-record", 7], ["borrow-status-delayed", 6], ["reservation-order-unclear", 1], ]; const wald = (count, n) => { const p = count / n; const margin = Z * Math.sqrt((p * (1 - p)) / n); return [p - margin, p + margin]; }; const wilson = (count, n) => { const p = count / n; const denom = 1 + (Z * Z) / n; const center = (p + (Z * Z) / (2 * n)) / denom; const margin = (Z / denom) * Math.sqrt((p * (1 - p)) / n + (Z * Z) / (4 * n * n)); return [center - margin, center + margin]; }; console.log("code observed Wald interval Wilson interval width"); for (const [name, c] of OBSERVED) { const [wa, wb] = wald(c, N); const [sa, sb] = wilson(c, N); console.log( `${name.padEnd(28)} ${`${c}/${N}`.padStart(8)} ` + `${`${(wa * 100).toFixed(1)} - ${(wb * 100).toFixed(1)}`.padEnd(16)} ` + `${`${(sa * 100).toFixed(1)} - ${(sb * 100).toFixed(1)}`.padEnd(16)} ` + `${((sb - sa) * 100).toFixed(1)} points` ); } console.log("\nsample size 95% margin of error (p = 0.5) 45% vs 55% distinguishable"); for (const n of [8, 12, 30, 100, 400, 1000]) { const margin = Z * Math.sqrt(0.25 / n); console.log( `${String(n).padStart(11)} ${`+/- ${(margin * 100).toFixed(1)} points`.padStart(27)} ${margin * 100 < 5 ? "yes" : "no"}` ); } console.log(`\nsmallest sample needed for +/- 5 points: ${Math.ceil((Z / 0.05) ** 2 * 0.25)}`); ``` ``` code observed Wald interval Wilson interval width same-name-record 7/12 30.4 - 86.2 32.0 - 80.7 48.7 points borrow-status-delayed 6/12 21.7 - 78.3 25.4 - 74.6 49.2 points reservation-order-unclear 1/12 -7.3 - 24.0 1.5 - 35.4 33.9 points sample size 95% margin of error (p = 0.5) 45% vs 55% distinguishable 8 +/- 34.6 points no 12 +/- 28.3 points no 30 +/- 17.9 points no 100 +/- 9.8 points no 400 +/- 4.9 points yes 1000 +/- 3.1 points yes smallest sample needed for +/- 5 points: 385 ``` The interval for the problem seven people mentioned runs from 32% to 81%; its width is 48.7 points. That interval contains both the claim "more than half the users" and the claim "a third of the users." A proportion drawn from twelve interviews cannot be used as an estimate. The Wald interval producing a **negative** lower bound on the third row is also instructive: the normal approximation loses validity at small sample sizes and extreme proportions, giving an impossible value for a probability. The Wilson interval places the same data between 1.5% and 35.4%. The only thing that can be said about the prevalence of a phenomenon observed once is that it is not zero. The second table gives the price of a survey. In a hundred-person survey, the margin of error is ±9.8 points, and it cannot distinguish a gap between 45% and 55%; reaching ±5 points of precision requires at least 385 responses. Quadrupling the sample size halves the margin of error — precision improves at the rate of the square root of the sample. #### The Method Selection Rule The calculations above reduce to a single selection rule. - If the question is "how do people do this," **observation** is chosen; what is done is measured, not what is said. - If the question is "why do they do it this way" or "under which constraints," an **interview** is chosen; its output is coded phenomena, not a proportion. - If the question is "how widespread is this," a **survey** is chosen and the required sample size is calculated up front; if the calculation cannot be met, the survey is not run, because a small survey does not answer. The common rule across all three is that the question is written before the method. A study that starts with the method fits itself to whatever question that method can answer. #### Summary - Observation measures what is done, an interview measures the reason, a survey measures prevalence; discovery is done with observation and interviews, measurement with a survey, and the order cannot be reversed. - Informed consent, anonymization, and a retention limit are part of the method; the scope of consent cannot be expanded afterward, and a raw recording is deleted once it is coded. - The measure of interview count is saturation; in the sample data, no new code appeared after the eighth interview, but saturation depends on participant diversity and does not yield a fixed number. - The 95% interval for a 58.3% proportion drawn from twelve interviews runs from 32% to 81%; a qualitative sample does not produce a prevalence estimate. - Survey precision improves with the square root of the sample: at least 385 responses are needed for ±5 points, and a hundred-person survey cannot distinguish gaps of ten points. #### Next Step This lesson pulled ten codes out of twelve interviews but still left the participants as nothing more than a list of `P01`–`P12` codes. Making a design decision requires seeing the structure of that list: do some participants resemble each other, and if so, along which axis? The next lesson clusters participants by their characteristics, shows with calculation that grouping by behavior does not give the same result as grouping by demographics, and sets out how a persona grounded in real data is built. ### User Profiles Source: https://academia.sh/en/courses/user-experience/user-profiles That a persona is not a fabricated person but a behavior set derived from data; clustering participants, calculating the explanatory power of role-based grouping, and the ethical limit of a persona. The previous lesson pulled ten codes out of twelve interviews, but still left the participants as a list in the form `P01`–`P12`. No design decision can be made with this list: twelve separate people do not get twelve separate interfaces. Seeing the list's structure is necessary to make a decision — who resembles whom, along which axis, and does that axis change the design? The common answer to this question is wrong. Participants are usually grouped by role: undergraduate student, graduate student, faculty member, library staff member, external member. This grouping looks manageable because roles are already on record. This lesson compares two groupings on the same data and calculates how much of the behavior role actually explains. #### What a Persona Is and Is Not A **persona** is a definition derived from research data that represents a set of behaviors. It has three parts: - **Goal:** the reason the person uses the interface. "Browsing sources" and "finding a known book" are separate goals, and they produce separate interface decisions. - **Behavior:** the observed form of the goal. Session duration, search type, borrowing frequency, which screen they entered from. - **Constraint:** conditions the person cannot change. Time, screen, language, accessibility requirement. When a persona contains anything outside these three, it stops producing design decisions. A made-up name, an age, a photograph, and a life story try to make the persona look "alive," but no decision can be grounded in these details. In the catalog interface, the decision of how many rows the result list shows depends on whether the user's session lasts four minutes or twenty-two, not on whether the user is thirty-two or forty-five years old. The criterion is this: a design decision must be demonstrable for every line written in the persona. A line with no corresponding decision is removed from the persona. #### Clustering Participants by What Is Measured Four measures were extracted from the observation sessions: the share of a session spent on subject browsing, the average session duration, the monthly borrow count, and the share of sessions entered from a narrow screen. Because these measures are on different scales, they are first moved into a common range, then the distance between participants is calculated, and the ones close to each other are merged. ```js // profile.mjs — comparing grouping by behavior with grouping by role // Measures derived from observation sessions (data constructed for this lesson). // browse: share of session spent on subject browsing, duration: average session minutes, // borrows: monthly borrow count, narrowScreen: share of sessions on a narrow screen. const PARTICIPANTS = [ { id: "P01", role: "undergraduate", browse: 0.15, duration: 4, borrows: 6, narrowScreen: 0.10 }, { id: "P02", role: "graduate", browse: 0.85, duration: 22, borrows: 2, narrowScreen: 0.20 }, { id: "P03", role: "undergraduate", browse: 0.45, duration: 6, borrows: 2, narrowScreen: 0.85 }, { id: "P04", role: "undergraduate", browse: 0.80, duration: 19, borrows: 3, narrowScreen: 0.15 }, { id: "P05", role: "faculty", browse: 0.20, duration: 5, borrows: 8, narrowScreen: 0.05 }, { id: "P06", role: "external", browse: 0.50, duration: 7, borrows: 1, narrowScreen: 0.90 }, { id: "P07", role: "staff", browse: 0.40, duration: 5, borrows: 3, narrowScreen: 0.80 }, { id: "P08", role: "staff", browse: 0.10, duration: 3, borrows: 7, narrowScreen: 0.15 }, { id: "P09", role: "faculty", browse: 0.90, duration: 25, borrows: 2, narrowScreen: 0.10 }, { id: "P10", role: "graduate", browse: 0.48, duration: 8, borrows: 2, narrowScreen: 0.88 }, { id: "P11", role: "external", browse: 0.78, duration: 20, borrows: 1, narrowScreen: 0.25 }, { id: "P12", role: "graduate", browse: 0.18, duration: 5, borrows: 6, narrowScreen: 0.12 }, ]; const MEASURES = ["browse", "duration", "borrows", "narrowScreen"]; // Measures are on different scales; move each into the 0-1 range. const min = {}, max = {}; for (const m of MEASURES) { min[m] = Math.min(...PARTICIPANTS.map((p) => p[m])); max[m] = Math.max(...PARTICIPANTS.map((p) => p[m])); } const vector = (p) => MEASURES.map((m) => (p[m] - min[m]) / (max[m] - min[m])); const V = PARTICIPANTS.map(vector); const distance = (a, b) => Math.hypot(...a.map((v, i) => v - b[i])); // Agglomerative clustering with complete linkage: merge the two nearest clusters, stop at k clusters. function cluster(target) { let clusters = V.map((_, i) => [i]); while (clusters.length > target) { let best = [0, 1], smallest = Infinity; for (let i = 0; i < clusters.length; i++) for (let j = i + 1; j < clusters.length; j++) { let d = 0; for (const a of clusters[i]) for (const b of clusters[j]) d = Math.max(d, distance(V[a], V[b])); if (d < smallest) { smallest = d; best = [i, j]; } } const [i, j] = best; clusters = clusters.filter((_, x) => x !== i && x !== j).concat([clusters[i].concat(clusters[j])]); } return clusters.map((c) => c.sort((a, b) => a - b)).sort((a, b) => a[0] - b[0]); } // Within-cluster sum of squares: each member's squared distance to the cluster mean const sumOfSquares = (groups) => groups.reduce((t, g) => { const avg = MEASURES.map((_, d) => g.reduce((s, i) => s + V[i][d], 0) / g.length); return t + g.reduce((s, i) => s + distance(V[i], avg) ** 2, 0); }, 0); const TOTAL = sumOfSquares([PARTICIPANTS.map((_, i) => i)]); const roleGroups = [...new Set(PARTICIPANTS.map((p) => p.role))].map((r) => PARTICIPANTS.map((p, i) => [p, i]).filter(([p]) => p.role === r).map(([, i]) => i) ); const behaviorGroups = cluster(3); console.log("clusters by behavior (k = 3)"); for (const g of behaviorGroups) { const avg = MEASURES.map((_, d) => g.reduce((s, i) => s + PARTICIPANTS[i][MEASURES[d]], 0) / g.length); console.log( ` ${g.map((i) => PARTICIPANTS[i].id).join(" ").padEnd(20)} n=${g.length} ` + `browse ${avg[0].toFixed(2)} duration ${avg[1].toFixed(1)} min borrows ${avg[2].toFixed(1)} narrow screen ${avg[3].toFixed(2)}` ); console.log(` roles: ${[...new Set(g.map((i) => PARTICIPANTS[i].role))].sort().join(", ")}`); } console.log("\ngroups by role"); for (const g of roleGroups) { console.log(` ${PARTICIPANTS[g[0]].role.padEnd(14)} ${g.map((i) => PARTICIPANTS[i].id).join(" ").padEnd(16)} n=${g.length}`); } const roleSS = sumOfSquares(roleGroups), behaviorSS = sumOfSquares(behaviorGroups); console.log("\ngrouping groups within-cluster sum of squares explained variance"); console.log(`by role ${String(roleGroups.length).padStart(6)} ${roleSS.toFixed(3).padStart(28)} ${((1 - roleSS / TOTAL) * 100).toFixed(1)}%`); console.log(`by behavior ${String(behaviorGroups.length).padStart(6)} ${behaviorSS.toFixed(3).padStart(28)} ${((1 - behaviorSS / TOTAL) * 100).toFixed(1)}%`); console.log(`total variance (single group): ${TOTAL.toFixed(3)}`); // Each cluster's medoid: the member whose distance sum within the cluster is smallest console.log("\ncluster medoid within-cluster distance sum"); for (const g of behaviorGroups) { const scores = g.map((i) => [i, g.reduce((s, j) => s + distance(V[i], V[j]), 0)]); scores.sort((a, b) => a[1] - b[1]); console.log(`${g.map((i) => PARTICIPANTS[i].id).join("+").padEnd(20)} ${PARTICIPANTS[scores[0][0]].id} ${scores[0][1].toFixed(3)}`); } ``` ``` clusters by behavior (k = 3) P01 P05 P08 P12 n=4 browse 0.16 duration 4.3 min borrows 6.8 narrow screen 0.11 roles: faculty, graduate, staff, undergraduate P02 P04 P09 P11 n=4 browse 0.83 duration 21.5 min borrows 2.0 narrow screen 0.17 roles: external, faculty, graduate, undergraduate P03 P06 P07 P10 n=4 browse 0.46 duration 6.5 min borrows 2.0 narrow screen 0.86 roles: external, graduate, staff, undergraduate groups by role undergraduate P01 P03 P04 n=3 graduate P02 P10 P12 n=3 faculty P05 P09 n=2 external P06 P11 n=2 staff P07 P08 n=2 grouping groups within-cluster sum of squares explained variance by role 5 4.885 22.3% by behavior 3 0.261 95.8% total variance (single group): 6.286 cluster medoid within-cluster distance sum P01+P05+P08+P12 P01 0.538 P02+P04+P09+P11 P02 0.605 P03+P06+P07+P10 P03 0.450 ``` #### Role Is Not a Persona The result of the comparison reads in a single line. Grouping by role produces **five** groups and explains 22.3% of the variance in behavior; grouping by behavior explains 95.8% with **three** groups. This comparison favors role, because more groups always tend to produce smaller within-group variance. Higher explanation with fewer groups shows that the difference comes from the data. The reason shows up in the composition of the clusters: each of the three behavior clusters contains four distinct roles. Two people with the same role do not resemble each other; four people with different roles do. Undergraduate P01 and faculty member P05 are in the same cluster — both want to find a known record and leave. Undergraduate P04 and faculty member P09 are in another cluster — both browse by subject. Role is cheap to measure because it is already written in the library record; it has no counterpart in design because it does not explain behavior. The averages of the three clusters give the core of the personas. - **Known-Record Searcher.** Subject-browsing share 0.16, session 4.3 minutes, 6.8 borrows a month. For this persona, the first row of the result list is decisive; the user does not scan the list, they confirm the first match and leave. - **Subject Browser.** Subject-browsing share 0.83, session 21.5 minutes, 2.0 borrows a month. This persona actually reads the list, compares, and leaves without borrowing in most sessions. What matters to them is not the transaction but being able to return to the list. - **Narrow-Screen Viewer.** 0.86 of sessions on a narrow screen, duration 6.5 minutes. This persona's constraint is the screen: the same result list shows them fewer rows, and the same action bar does not fit. The third table gives each cluster's **medoid**: the member whose within-cluster distance sum is smallest. When the persona text is written, this participant's notes are used, because it is a real record standing at the center of the cluster. It is not an average person; it is the person closest to the average. #### The Ethical Limit of a Persona Because a persona is derived from research data, it inherits every obligation that participant data carries. **A single-member cluster is not a persona.** If a cluster reduces to a single member, that persona is one person's behavior record; even with the name removed, someone on the team recognizes it. The rule is this: if a persona does not represent the behavior of at least three participants, it is not written as a persona — it stays an observation. All three clusters above have four members. **The medoid's quote is anonymized too.** When a participant's words are brought into the persona text, details that would make the person identifiable are removed. The criterion set in the previous lesson applies here as well: it must not be possible to trace back from the data to identity. **The persona stays within the scope of consent.** When the participant joined the research, they consented to their data being used in design decisions, not to their name appearing on a persona card. This is the only defensible reason for giving personas made-up names, but the made-up name is also chosen so that it does not evoke the real one. **A persona must not turn into a tool for exclusion.** The sentence "this persona is not our user" can be written as a scope decision, but it is written together with who was not measured. If none of the three personas were derived from a participant who uses a screen reader, that is not a finding but a sample gap, and it is recorded in the persona document as such. #### Summary - A persona is the triad of goal, behavior, and constraint; a design decision must be demonstrable for every line in the persona. - Participants are clustered by measured behavior; the measures are moved into a common range and merged by distance. - In the sample data, grouping by role explained 22.3% of the variance with five groups, and grouping by behavior explained 95.8% with three groups; role is cheap to measure but does not explain behavior. - Each cluster's medoid is the real participant whose within-cluster distance sum is smallest; the persona text is written from this record, not from an average person. - A persona is not written unless it represents at least three participants; a persona must not make a participant re-identifiable, and it must not present those left outside the sample as a finding. #### Next Step Three personas were obtained, but a persona alone is not a to-do list. The Subject Browser persona's session lasts twenty-one and a half minutes — what exactly do they do in that time, which steps do they follow in order, and what should be expected at each step? The next lesson turns the need into action: it addresses the form in which a user story is written, how a solution seeps into it, and how tasks are ranked by frequency and failure data. ### User Stories and Tasks Source: https://academia.sh/en/courses/user-experience/user-stories-and-tasks Turning a need into a story and a story into a task; checking for solution bias in a story, and ranking the task inventory by frequency, failure rate, and failed attempts. The previous lesson produced three personas: Known-Record Searcher, Subject Browser, and Narrow-Screen Viewer. Personas say who is coming, not what they do. The Subject Browser persona's session lasts twenty-one and a half minutes; what steps happen in that time, which step requires which other step, and which one should the design prioritize? This lesson turns the need into action. It first establishes the distinction among need, story, and task; then it checks how a solution seeps into a story; and finally it ranks the task inventory drawn from observation data by three separate criteria, and shows that these rankings do not agree with each other. #### Need, Story, and Task The three concepts are often used interchangeably, and wherever they are confused, the design decision becomes blurred. **A need** is independent of the interface. "Knowing what has been written on a subject" is a need; it exists without the catalog interface too, and it can also be met by asking a librarian. **A user story** ties the need to a role and a goal. Its format is short and carries three parts: who, what, why. A story does not propose a solution on its own; it narrows the solution space. **A task** is the observable counterpart of a story: a sequence of actions with a start, an end, and a success criterion. "Browse by subject" is a task; it starts, it ends, and whether it counts as successful can be measured. Tasks break down into subtasks: within browsing by subject there are steps such as "narrow the results," "open a record," and "return to the list." The chain is one-directional. It descends from need to story, from story to task, from task to interface decision. If walked in reverse — if an interface element is thought of first and then a story is written to justify it — research data does not produce the decision; it decorates it. #### Solution Bias in Stories The most common way a story breaks down is when an interface element gets written in place of the need. The sentence "I want a publication-year filter" looks like a need but is a solution; the need is "telling recent sources apart," and a filter is not its only counterpart. The check below scans stories at the level of format, role, and wording. ```js // story.mjs — checking user stories for format and solution bias const STORIES = [ "As an undergraduate student, I want a publication-year filter on search results, so that I can tell recent sources apart", "As a subject browser, I want to return to my result list after opening a record, so that I can continue comparing", "As a faculty member, I want to select a language from a dropdown menu", "As a library staff member, I want to see whether a record is on the shelf, so that I can tell the reader whether the book is available", "As a user, I want a better search screen, so that I can search more conveniently", "As an external member, I want to extend my borrowing period, so that I can keep working without returning the book", ]; // Format: As a , I want , so that const FORMAT = /^As an? (.+?), I want (.+?), so that (.+)$/; // Solution vocabulary: words naming an interface element const SOLUTION = ["filter", "button", "dropdown menu", "tab", "checkbox", "screen", "page", "form", "slider"]; // Generic names that do not stand in for a role const GENERIC_ROLE = ["user", "person", "everyone", "people"]; console.log("no format role finding"); STORIES.forEach((s, i) => { const m = s.match(FORMAT); const findings = []; if (!m) findings.push("format incomplete (goal not written)"); const role = m ? m[1] : (s.match(/^As an? ([^,]+)/)?.[1] ?? ""); if (GENERIC_ROLE.includes(role.toLowerCase())) findings.push("role generic"); const leaked = SOLUTION.filter((c) => s.toLowerCase().includes(c)); if (leaked.length) findings.push(`solution bias: ${leaked.join(", ")}`); console.log( `${String(i + 1).padStart(2)} ${(m ? "complete" : "incomplete").padEnd(10)} ${role.slice(0, 22).padEnd(22)} ${findings.length ? findings.join(" | ") : "-"}` ); }); const clean = STORIES.filter((s) => { const m = s.match(FORMAT); const role = m ? m[1] : ""; return m && !GENERIC_ROLE.includes(role.toLowerCase()) && !SOLUTION.some((c) => s.toLowerCase().includes(c)); }); console.log(`\nstories passing the check: ${clean.length} / ${STORIES.length}`); ``` ``` no format role finding 1 complete undergraduate student solution bias: filter 2 complete subject browser - 3 incomplete faculty member format incomplete (goal not written) | solution bias: dropdown menu 4 complete library staff member - 5 complete user role generic | solution bias: screen 6 complete external member - stories passing the check: 3 / 6 ``` Three of the six stories pass the check. The check is not a measure of correctness but a review tool: because it relies on a word list, it does not catch a solution outside the list, and it can flag an innocent usage by mistake. Its function is to move the discussion from the level of wording to the level of structure. Each of the three findings shows a different flaw. In the first story, the goal is written correctly, but a solution has taken the need's place; its corrected form becomes "being able to see publications in order of recency," and this leaves multiple solutions open — filter, sort, date range. In the third story, there is no goal at all: because why the language selection is needed is never written, the story cannot be tested. In the fifth story, the role is "user" — which of the three personas this is remains unclear, and "more conveniently" cannot be measured. #### Task Inventory and Priority Once stories are turned into tasks, the question becomes which one to take on first. During the observation period, the attempt and success counts were logged for each task. ```js // task.mjs — ranking the task inventory by frequency, failure rate, and failed attempts // Task attempts logged during a four-week observation period (data constructed for this lesson) const TASKS = [ { name: "find a known record", attempts: 148, successes: 132 }, { name: "find the shelf location", attempts: 121, successes: 79 }, { name: "borrow", attempts: 94, successes: 88 }, { name: "browse by subject", attempts: 86, successes: 51 }, { name: "renew a loan", attempts: 37, successes: 19 }, { name: "reserve", attempts: 22, successes: 12 }, { name: "add to reading list", attempts: 14, successes: 13 }, ]; for (const t of TASKS) { t.failed = t.attempts - t.successes; t.failureRate = t.failed / t.attempts; } console.log("task attempts successes failed failure rate"); for (const t of TASKS) { console.log( `${t.name.padEnd(24)} ${String(t.attempts).padStart(8)} ${String(t.successes).padStart(10)} ${String(t.failed).padStart(7)} ${(t.failureRate * 100).toFixed(1)}%` ); } const rankedNames = (key) => [...TASKS].sort((a, b) => b[key] - a[key]).map((t) => t.name); const RANKINGS = { "frequency (attempts)": rankedNames("attempts"), "failure rate": rankedNames("failureRate"), "failed attempts": rankedNames("failed"), }; console.log("\nrank frequency failure rate failed attempts"); for (let i = 0; i < TASKS.length; i++) { console.log( `${String(i + 1).padStart(4)} ${RANKINGS["frequency (attempts)"][i].padEnd(25)} ` + `${RANKINGS["failure rate"][i].padEnd(25)} ${RANKINGS["failed attempts"][i]}` ); } // The rank of the same task across the three rankings console.log("\ntask frequency rate failed largest rank gap"); let largest = { name: "", gap: -1 }; for (const t of TASKS) { const r = Object.values(RANKINGS).map((l) => l.indexOf(t.name) + 1); const gap = Math.max(...r) - Math.min(...r); if (gap > largest.gap) largest = { name: t.name, gap }; console.log(`${t.name.padEnd(24)} ${String(r[0]).padStart(9)} ${String(r[1]).padStart(5)} ${String(r[2]).padStart(7)} ${gap}`); } console.log(`\ntask with the largest rank shift: ${largest.name} (${largest.gap} ranks)`); console.log(`total failed attempts: ${TASKS.reduce((t, x) => t + x.failed, 0)}`); const topTwo = [...TASKS].sort((a, b) => b.failed - a.failed).slice(0, 2); console.log( `${((topTwo.reduce((t, x) => t + x.failed, 0) / TASKS.reduce((t, x) => t + x.failed, 0)) * 100).toFixed(1)}% of failed attempts are in two tasks: ` + topTwo.map((t) => t.name).join(", ") ); ``` ``` task attempts successes failed failure rate find a known record 148 132 16 10.8% find the shelf location 121 79 42 34.7% borrow 94 88 6 6.4% browse by subject 86 51 35 40.7% renew a loan 37 19 18 48.6% reserve 22 12 10 45.5% add to reading list 14 13 1 7.1% rank frequency failure rate failed attempts 1 find a known record renew a loan find the shelf location 2 find the shelf location reserve browse by subject 3 borrow browse by subject renew a loan 4 browse by subject find the shelf location find a known record 5 renew a loan find a known record reserve 6 reserve add to reading list borrow 7 add to reading list borrow add to reading list task frequency rate failed largest rank gap find a known record 1 5 4 4 find the shelf location 2 4 1 3 borrow 3 7 6 4 browse by subject 4 3 2 2 renew a loan 5 1 3 4 reserve 6 2 5 4 add to reading list 7 6 7 1 task with the largest rank shift: find a known record (4 ranks) total failed attempts: 128 60.2% of failed attempts are in two tasks: find the shelf location, browse by subject ``` The three rankings produce three separate lists, and no task sits at the same rank in all three. **By frequency**, "find a known record" ranks first; it is the most attempted task. But its failure rate is 10.8%, and only sixteen of the hundred and forty-eight attempts are failures. Frequency alone does not set design priority; a task that is done a lot but works well is not an area for improvement. **By failure rate**, "renew a loan" ranks first: 48.6%. This rate correctly describes the severity of the problem but not its size; the task was attempted thirty-seven times and produced eighteen failures. **By failed attempts**, "find the shelf location" ranks first: forty-two failures. This task is second by frequency, fourth by failure rate — first in neither list, but it produces the largest share of the total damage. Of the total hundred and twenty-eight failed attempts, 60.2% are concentrated in two tasks. Priority ranking is built not by picking one of the three but by defining what the problem is. If the goal is to reduce total damage, the failed-attempts ranking is used. If the goal is to raise a task's reliability above a threshold, the failure rate is used. Frequency alone is not a priority criterion; it is a multiplier for the other two: a high rate on a rarely attempted task can produce fewer failures than a low rate on a frequently attempted one. A caveat is needed: these numbers count only tasks that were **attempted**. A task the user wanted to do but could not find the way to, and so never attempted, does not enter the inventory. The fourteen attempts at "add to reading list" may show not that this task is rare, but that it is rare to be **found**. This is the inventory's blind spot, and it can only be closed by an interview. #### The Ethical Limit of Task Data The task inventory is produced from per-person session logs; this is the most identifying form of research data. The three rules apply here too. A session log is collected from **consenting** participants; logging everyone on a publicly open interface is not research but surveillance. The data that enters analysis is **aggregated**: the table above has no people in it, only tasks. Per-person records are deleted within the **retention limit** once aggregation is done; only the numbers remain. A related design rule follows from this: every new field collected to enrich the inventory opens one more path back to identity. Knowing whether a task succeeded does not require knowing who attempted it. #### Acceptance Criteria Make a Story Testable Even a story that passes the check is not enough on its own; when it is satisfied must be written down. **Acceptance criteria** are a story's verifiable counterpart, and they are written in terms of the task. The second story — "being able to return to my result list after opening a record" — is tied to these criteria: on return, the search term, the applied refinements, and the position in the list are preserved; a record the user has already opened stays marked in the list; the return does not require re-running the search. All three are observable; all three are testable. The solution is again kept out while writing the criterion. "A back button is added" is not a criterion but a decision; a criterion states what the result must be, regardless of which button achieves it. #### Summary - A need is independent of the interface, a story ties the need to a role and a goal, and a task is the story's observable counterpart with a start, an end, and a success criterion. - Solution bias is the most common way a story breaks down; the wording and format check is a review tool, not a measure of correctness, and it flagged three of the six sample stories. - The same task inventory produces three separate rankings by frequency, failure rate, and failed attempts; in the sample data, one task shifted by four ranks across the three rankings. - The priority criterion is derived from the definition of the problem: failed attempts is used for total damage, failure rate for reliability; frequency alone does not set priority. - The inventory counts only attempted tasks; a task that is never attempted is invisible, and this blind spot can only be closed by an interview. - A session log is collected from consenting participants, enters analysis in aggregated form, and per-person records are deleted within the retention limit. #### Next Step The task inventory shows each task as a separate row, yet the user performs them back to back: they find a record, check the shelf location, go to the shelf, cannot find the book, and return to the catalog. The task producing the largest share of failed attempts, "find the shelf location," is a step that starts on screen and ends off screen. The next lesson traces this chain end to end: it shows what columns a journey map consists of, why stage duration and drop-off points do not sit in the same place, and how off-screen steps enter the map. ### Journey Maps Source: https://academia.sh/en/courses/user-experience/journey-maps Breaking the end-to-end experience into stages; calculating channel share, drop-off rate, and wasted person-minutes; and why the longest stage and the most-abandoned stage are not the same. The task inventory showed each task as a separate row, yet the user performs them back to back. The "find the shelf location" task starts on screen, ends in front of the shelf, and when it fails, the user goes back to the catalog. The inventory cannot see this chain; it counts each task as an independent event. A **journey map** is the end-to-end tracking of a single goal: where the user starts, which stages they pass through, how long they stay at each stage, and where they give up. This lesson builds the map's columns, calculates the drop-off points, and shows why the longest stage is not the worst stage. #### The Map's Columns A journey map assigns a column to four questions. - **Stage.** A step the user takes toward their goal. A stage boundary sits where the user makes a decision or the environment changes; it does not map one-to-one onto interface screens. - **Channel.** The medium the stage passes through. The catalog interface has three channels: screen, the library's physical space, and the borrowing desk. A map with no channel column assumes a journey that starts and ends on screen. - **Measurement.** How many people entered the stage, how many advanced to the next stage, and how much time passed at the stage. This column comes from observation data. - **Finding.** What happened at the stage and why. This column comes from the interview; it is not a number but a coded phenomenon. Keeping the last two columns separate matters. Measurement says where something was lost, the finding says why it was lost, and one does not substitute for the other. #### Measuring the Journey The calculation below tracks the "find a known record and borrow it" goal from a hundred starts. For each stage, the number entered and advanced, along with the average time spent at the stage, was logged. ```js // journey.mjs — end-to-end journey: stage-by-stage duration, drop-off, and wasted time // The "find a known record and borrow it" journey tracked from 100 starts // (data constructed for this lesson). channel: the medium the stage passes through. const STAGES = [ { name: "open the catalog", channel: "screen", entered: 100, advanced: 97, duration: 0.4 }, { name: "type the search", channel: "screen", entered: 97, advanced: 92, duration: 1.1 }, { name: "scan the result list", channel: "screen", entered: 92, advanced: 74, duration: 2.3 }, { name: "read the record detail", channel: "screen", entered: 74, advanced: 68, duration: 1.6 }, { name: "note the shelf code", channel: "screen", entered: 68, advanced: 61, duration: 0.5 }, { name: "walk to the shelf", channel: "physical", entered: 61, advanced: 58, duration: 6.2 }, { name: "find the book on the shelf", channel: "physical", entered: 58, advanced: 39, duration: 4.8 }, { name: "complete the borrowing transaction", channel: "desk", entered: 39, advanced: 37, duration: 2.1 }, ]; console.log("stage channel entered advanced dropped drop rate duration cumulative"); let totalDuration = 0; for (const s of STAGES) { s.dropped = s.entered - s.advanced; s.dropRate = s.dropped / s.entered; totalDuration += s.duration; console.log( `${s.name.padEnd(37)} ${s.channel.padEnd(9)} ${String(s.entered).padStart(6)} ${String(s.advanced).padStart(8)} ` + `${String(s.dropped).padStart(8)} ${(s.dropRate * 100).toFixed(1).padStart(9)}% ${s.duration.toFixed(1).padStart(6)} ${totalDuration.toFixed(1).padStart(10)}` ); } console.log(`\nend-to-end completion: ${STAGES.at(-1).advanced} / ${STAGES[0].entered} (${STAGES.at(-1).advanced}%)`); console.log(`end-to-end duration (for completers): ${totalDuration.toFixed(1)} min`); // Channel share: how much of the journey happened on screen const channelDuration = {}; for (const s of STAGES) channelDuration[s.channel] = (channelDuration[s.channel] ?? 0) + s.duration; console.log("\nchannel duration journey share"); for (const [c, d] of Object.entries(channelDuration)) { console.log(`${c.padEnd(9)} ${d.toFixed(1).padStart(5)} min ${((d / totalDuration) * 100).toFixed(1)}%`); } // Three separate definitions of "worst stage" do not point to the same stage const maxBy = (key) => [...STAGES].sort((a, b) => b[key] - a[key])[0]; for (const s of STAGES) s.wastedPersonMinutes = s.dropped * s.duration; console.log("\nmetric worst stage value"); console.log(`${"longest stage".padEnd(28)} ${maxBy("duration").name.padEnd(35)} ${maxBy("duration").duration.toFixed(1)} min`); console.log(`${"most dropped".padEnd(28)} ${maxBy("dropped").name.padEnd(35)} ${maxBy("dropped").dropped} people`); console.log(`${"highest drop rate".padEnd(28)} ${maxBy("dropRate").name.padEnd(35)} ${(maxBy("dropRate").dropRate * 100).toFixed(1)}%`); console.log(`${"most wasted person-minutes".padEnd(28)} ${maxBy("wastedPersonMinutes").name.padEnd(35)} ${maxBy("wastedPersonMinutes").wastedPersonMinutes.toFixed(1)} person-minutes`); // Time spent by those who dropped: total minutes for those who entered but never exited the journey let spent = 0, completerMinutes = STAGES.at(-1).advanced * totalDuration; for (let i = 0; i < STAGES.length; i++) { spent += STAGES[i].entered * STAGES[i].duration; } console.log(`\ntotal time spent: ${spent.toFixed(1)} person-minutes`); console.log(`completers' time: ${completerMinutes.toFixed(1)} person-minutes`); console.log(`share of time with no result: ${(((spent - completerMinutes) / spent) * 100).toFixed(1)}%`); ``` ``` stage channel entered advanced dropped drop rate duration cumulative open the catalog screen 100 97 3 3.0% 0.4 0.4 type the search screen 97 92 5 5.2% 1.1 1.5 scan the result list screen 92 74 18 19.6% 2.3 3.8 read the record detail screen 74 68 6 8.1% 1.6 5.4 note the shelf code screen 68 61 7 10.3% 0.5 5.9 walk to the shelf physical 61 58 3 4.9% 6.2 12.1 find the book on the shelf physical 58 39 19 32.8% 4.8 16.9 complete the borrowing transaction desk 39 37 2 5.1% 2.1 19.0 end-to-end completion: 37 / 100 (37%) end-to-end duration (for completers): 19.0 min channel duration journey share screen 5.9 min 31.1% physical 11.0 min 57.9% desk 2.1 min 11.1% metric worst stage value longest stage walk to the shelf 6.2 min most dropped find the book on the shelf 19 people highest drop rate find the book on the shelf 32.8% most wasted person-minutes find the book on the shelf 91.2 person-minutes total time spent: 1249.2 person-minutes completers' time: 703.0 person-minutes share of time with no result: 43.7% ``` #### What the Map Says **End-to-end completion is 37%.** None of the eight stages has a drop-off rate higher than a third, but their product brings a hundred down to thirty-seven. Stages looking "acceptable" one by one does not mean the chain is acceptable. A journey map's first function is to show the product of numbers that look good stage by stage. **The screen is only 31.1% of the journey.** Of the nineteen-minute journey, eleven minutes pass in the physical space and two at the desk. This is a region the catalog interface's measurement cannot see: screen measurement goes blind the moment the user closes the catalog. The User Profiles lesson's session duration of 4.3 minutes for the Known-Record Searcher persona is a result of this blindness; the user is on screen for six minutes and off screen for thirteen on this journey. **The longest stage is not the worst stage.** "Walk to the shelf" is the longest stage at 6.2 minutes, but its drop-off rate is 4.9%. The user expects this duration and accepts it; it is an expected part of the journey. The "find the book on the shelf" stage takes 4.8 minutes and loses nineteen of the fifty-eight people who enter it. Length is not a sign of a problem; the problem is that the time spent produces no result. **Wasted person-minutes convert stages into a common unit.** Multiplying the number of people who dropped off by the time spent at that stage gives the time wasted at that stage. "Find the book on the shelf" produces 91.2 person-minutes; of the total one thousand two hundred forty-nine person-minutes, 43.7% is time that reached no result. This ratio changes the design conversation: the problem is framed not as "37% completion is low" but as "forty-three points of the time spent bought nothing." **An on-screen decision affects an off-screen stage.** The largest loss being off screen does not mean the solution is off screen too. At the "note the shelf code" stage, seven of sixty-eight people drop off; some of the rest may also note the code incorrectly, and the cost of that is paid at the next stage. How the shelf code is shown in the record detail directly determines the 4.8 minutes spent at the shelf. The map makes this link visible; the task inventory does not. #### Separating the Drop-Off Point from the Finding What the table does not say is why nineteen people could not find the book at the shelf. The possible reasons call for very different solutions from each other: the book is checked out but the catalog is not current, the book was shelved in the wrong place, the shelf code was read but the shelving scheme was not understood, or the user stood in front of the right shelf and missed the book with their eyes. Which of these applies is determined not by measurement but by a short interview conducted right after the stage. The `shelf-code-unclear` code, coded in the first lesson, appeared in seven of twelve participants; this finding supports the third possibility but does not prove it on its own. The map's finding column records which stage a code belongs to — a code cannot be turned into a design decision until it is tied to a stage. #### The Ethical Limit of Off-Screen Observation A journey map requires observing the user in the physical space, and this creates obligations different from an on-screen recording. **Consent for shadowing is obtained separately.** Consent given for catalog use does not cover being followed among the shelves. In shadowing, the participant knows the observer is present; covert surveillance is not a research method. **Third parties do not enter the record.** In observation carried out in the physical space, other users who have not consented are also within view. When notes are taken, only the participant's behavior is written down; the image, voice, or identifying detail of people nearby is not recorded. **Location data is reduced to a stage as soon as possible.** A record in the form "third floor, shelf 4, at 11:20" identifies a person in time and place. The data that enters analysis is the stage name and duration; the raw location and time are deleted within the retention limit once aggregation is done. #### Summary - A journey map consists of stage, channel, measurement, and finding columns; measurement says where something was lost, the finding says why, and one does not substitute for the other. - Drop-off rates that look acceptable stage by stage get multiplied; in the sample data, even though none of the eight stages exceeded a third, end-to-end completion stayed at 37%. - Only 31.1% of the sample journey passes on screen; screen measurement goes blind once the user closes the interface, and session duration is not journey duration. - The longest stage is not the worst stage; the wasted-person-minutes metric converts stages into a common unit, and in the sample data 43.7% of the time spent reached no result. - Consent is obtained separately for observation in the physical space, third parties do not enter the record, and raw location-time data is deleted once it is reduced to a stage. #### Next Step The map shows where our own interface loses people, but it does not say whether these losses are unavoidable. Other systems solve the same task, and which decisions they made is measurable data: in how many steps do they complete it, which information do they show on which screen, and where do they diverge from each other in their decisions? The next lesson addresses how to read existing solutions: comparing step counts, telling conventions apart from deviations, and understanding why a decision turns out the same across everyone. ### Competitive and Current-State Analysis Source: https://academia.sh/en/courses/user-experience/competitive-and-current-state-analysis Reading existing solutions decision by decision; spotting a convention through its adoption rate, measuring the cost of a deviation in step count, and the limits observed during the review. The journey map showed where our own interface loses people, but it did not say whether these losses were unavoidable. Other systems doing the same job exist, and the decisions they made can be observed from the outside. This lesson turns that observation into a dataset. The goal is not imitation. Reading existing solutions gives two separate pieces of information. First, which decisions are **conventions**: a problem solved the same way across every system is a pattern the user has already learned, and deviating from it produces a learning cost. Second, which decisions are **open**: the places where systems diverge from each other are the places where the designer will actually make a decision. #### Breaking Systems Down into Decisions The analysis does not start with the question "which is better" but with "which decisions were made." Every system is reduced to a decision list, and the list is the same across all systems; only then are they comparable. A decision must be observable and binary: is it present in the system or not. "Search works well" is not a decision but a judgment. "The search term is preserved when returning to the list after opening a record" is a decision; you look, and you write down the answer. ```js // current-state.mjs — spotting conventions and comparing step counts // A: our current catalog interface. B-E: four other systems doing the same job, reviewed. // (data constructed for this lesson) const SYSTEMS = ["A", "B", "C", "D", "E"]; const DECISIONS = [ { name: "search field top-center", present: ["A", "B", "C", "D", "E"] }, { name: "borrow action in record detail", present: ["A", "B", "C", "D", "E"] }, { name: "search preserved on return", present: ["B", "C", "D", "E"] }, { name: "result count shown", present: ["A", "B", "D", "E"] }, { name: "on-shelf badge in list", present: ["B", "C", "D"] }, { name: "refinements on the left", present: ["A", "B", "D"] }, { name: "reading list", present: ["A", "C", "E"] }, { name: "subject-tree navigation", present: ["C", "D"] }, { name: "shelf code in list", present: ["C"] }, { name: "record opens in a new tab", present: ["E"] }, ]; const N = SYSTEMS.length; const classify = (share) => (share >= 0.8 ? "convention" : share >= 0.4 ? "split" : "differentiator"); console.log("decision adopters share class ours"); const deviations = []; for (const d of DECISIONS) { const share = d.present.length / N; const ours = d.present.includes("A"); if (classify(share) === "convention" && !ours) deviations.push(d.name); console.log( `${d.name.padEnd(30)} ${d.present.join("").padEnd(11)} ${share.toFixed(2)} ${classify(share).padEnd(14)} ${ours ? "yes" : "NO"}` ); } console.log(`\ndeviation from a convention: ${deviations.length ? deviations.join(", ") : "none"}`); // How many screen steps the same task takes to complete on each system (observed by counting) const TASKS = [ { name: "find a known record and borrow it", steps: { A: 7, B: 5, C: 6, D: 5, E: 8 } }, { name: "browse by subject and compare three records", steps: { A: 13, B: 9, C: 11, D: 12, E: 10 } }, ]; const median = (values) => { const s = [...values].sort((x, y) => x - y); const m = s.length >> 1; return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; }; console.log("\n" + "task".padEnd(45) + " " + SYSTEMS.map((s) => s.padStart(2)).join(" ") + " min median our difference"); for (const t of TASKS) { const values = SYSTEMS.map((s) => t.steps[s]); const med = median(values); const diff = t.steps.A - med; console.log( `${t.name.padEnd(45)} ${values.map((v) => String(v).padStart(2)).join(" ")} ` + `${String(Math.min(...values)).padStart(3)} ${String(med).padStart(6)} ${(diff >= 0 ? "+" : "") + diff} steps` ); } // Step-by-step model of the comparison task: state preserved or not function stepCount(statePreserved, recordCount) { let a = 3; // type the term, run the search, apply the refinement for (let i = 0; i < recordCount; i++) { a += 2; // open the record, return to the list if (!statePreserved && i < recordCount - 1) a += 2; // retype the term, reapply the refinement } return a; } console.log("\ncomparison-task model (3 records)"); console.log(` state preserved : ${stepCount(true, 3)} steps`); console.log(` state not preserved : ${stepCount(false, 3)} steps`); console.log(` source of the gap : ${stepCount(false, 3) - stepCount(true, 3)} steps, from a single decision`); console.log(` observed A : ${TASKS[1].steps.A} steps, observed min (B): ${Math.min(...SYSTEMS.map((s) => TASKS[1].steps[s]))} steps`); // How the gap grows as record count increases console.log("\nrecords compared preserved not preserved gap"); for (const k of [2, 3, 5, 8]) { const p = stepCount(true, k), np = stepCount(false, k); console.log(`${String(k).padStart(16)} ${String(p).padStart(9)} ${String(np).padStart(13)} ${np - p}`); } ``` ``` decision adopters share class ours search field top-center ABCDE 1.00 convention yes borrow action in record detail ABCDE 1.00 convention yes search preserved on return BCDE 0.80 convention NO result count shown ABDE 0.80 convention yes on-shelf badge in list BCD 0.60 split NO refinements on the left ABD 0.60 split yes reading list ACE 0.60 split yes subject-tree navigation CD 0.40 split NO shelf code in list C 0.20 differentiator NO record opens in a new tab E 0.20 differentiator NO deviation from a convention: search preserved on return task A B C D E min median our difference find a known record and borrow it 7 5 6 5 8 5 6 +1 steps browse by subject and compare three records 13 9 11 12 10 9 11 +2 steps comparison-task model (3 records) state preserved : 9 steps state not preserved : 13 steps source of the gap : 4 steps, from a single decision observed A : 13 steps, observed min (B): 9 steps records compared preserved not preserved gap 2 7 9 2 3 9 13 4 5 13 21 8 8 19 33 14 ``` #### Three Decision Classes Adoption rate splits decisions into three classes, and each class calls for a different design stance. **Convention (0.80 and above).** The search field sitting top-center and the borrow action being in the record detail are the same in all five of the five systems. These decisions are not opened up for debate; the user has learned them elsewhere, and the interface inherits that learning. Deviating from a convention is not forbidden, but a measurable gain must be shown in exchange for the deviation. **Split decision (0.40 – 0.80).** Showing on-shelf status in the list, the position of the refinements, whether a reading list exists: systems split into two camps. These decisions are genuine design decisions, and they are resolved with our own user data. In a split decision, the argument "everyone does it this way" cannot be used, because not everyone does. **Differentiator (below 0.40).** The shelf code shown in the list exists only in C, and a record opening in a new tab exists only in E. These decisions are either an advantage or a choice that was tried and never caught on; the only way to tell the two apart is to ask, in that system's own context, why it made this choice. These are the decisions that look most tempting to copy, and they are also the riskiest. #### The Cost of a Deviation Is Measurable The decision list has exactly one deviation from a convention: search being preserved on return exists in four systems but not in ours. We are looking at the same phenomenon as the `search-lost-on-return` code, which appeared in four of the twelve interviews in the first lesson; now its cost is being tied to a number. The step model builds the three-record comparison task under two conditions. When state is preserved, the task takes nine steps; when it is not, thirteen. The gap is four steps, and all of it comes from a single decision. The observed numbers confirm the model: our interface takes thirteen steps, and the system that completes it in the fewest steps takes nine. The last table shows the real constraint. The gap is not constant; it grows together with the number of records compared. Two records, a two-step gap; three records, four steps; eight records, fourteen steps. The Subject Browser persona built in the second lesson does exactly this task, and their session lasts twenty-one and a half minutes. The user who pays the highest cost for the deviation is the user who uses the interface the most. The rule that follows from this determines how the comparison is read: **step count is calculated not for a single task but for the task's range of sizes.** A gap that looks trivial in a small example can grow linearly in real use. #### The Difference Between the Median and the Best The table has two comparison metrics, and they answer two separate questions. The gap against the median tells how far the interface has drifted from the customary; one step in the known-record task, two steps in the comparison task. The gap against the system that completes it in the fewest steps tells the reachable limit: four steps in the comparison task. If the median is chosen as the target, the interface stays average. If the best is chosen, the target is clear, but that system may carry other constraints; system E, which completes the task in eight steps, opens records in a new tab — a decision that slows it down in this task but pays off in another. A system's edge in one task does not mean all of its decisions are correct. #### Limits of the Review Reviewing existing systems means coming into contact with someone else's interface, and often with someone else's data. **The review is conducted as an ordinary user.** The decision list is made of things observable in the publicly open interface. Creating a fake membership to reach closed sections, bulk-downloading another institution's records, or bypassing access restrictions is not review, and it falls outside the analysis. **If someone else's users are to be observed, the same rules apply.** Observing another library's users carries the same consent and anonymization obligation as observing our own participants; the institution's permission is obtained separately. **Decisions are carried over, not content.** A system's record descriptions, classification data, or texts are that institution's labor. What is taken from the analysis is information such as "the search term is preserved," not the text itself. **The decision list is kept together with its source.** Which system, on which date, and by attempting which task it was observed is written down. Interfaces change; a decision list with no date cannot be verified a year later. #### Summary - The analysis is done not with the question "which is better" but with a decision list that is common and binary across all systems. - Adoption rate splits decisions into conventions, split decisions, and differentiators; deviating from a convention requires a measurable gain, and split decisions are resolved with our own data. - In the sample data, the one deviation from a convention is "search not preserved on return"; the step model showed that this single decision adds four steps in a three-record comparison, and it matches the observed numbers. - The cost of a deviation grows with task size: in a session comparing eight records, the gap rises to fourteen steps, and the cost is paid most by whoever uses the interface the most. - The review is limited to the publicly open interface; fake membership, bulk data downloads, and carrying over content fall outside the analysis, and the decision list is kept together with its date. #### Next Step Data accumulated over five lessons: ten codes, three personas, seven tasks, one journey map, and one decision list. This accumulation is not yet a design job — it is a pile of material. The next lesson reduces the pile to a single sentence: the problem statement. It addresses the criterion by which findings are ranked, which parts a problem statement consists of, and why a solution seeping into it is the most common mistake. ### Problem Statement Source: https://academia.sh/en/courses/user-experience/problem-definition Merging five lessons' findings into a single table; ranking by evidence strength and lost time, the parts of a problem statement, and preventing a solution from seeping into the statement. Data accumulated across five lessons: ten codes, three personas, seven tasks, one journey map, and one decision list. This accumulation is not yet a design job; to become something workable, it must be reduced to a single question: **which problem is being solved?** Skipping this question is the most common way research goes to waste. Findings are collected, written into a document, and then work moves straight to drawing screens. The drawn screen cannot say which finding it answers; whether it is good or bad cannot be tested, because what it must solve is never written down. This lesson first ranks the findings, then turns the finding at the top of the ranking into a problem statement. #### Converting Findings into a Common Unit Findings come from different methods and cannot be compared directly: one is "seven participants mentioned it," another is "forty-two failed attempts," a third is "four extra steps." A common unit is needed for comparison. The most defensible unit is the time the user pays. ```js // problem.mjs — ranking findings by evidence strength and lost time // Data from five lessons merged into a single table (data constructed for this lesson). // mentions: how many of the 12 interviews it appeared in. sources: data types // independently showing the finding. failedAttempts: failed-attempt count in the // four-week period. minutesPerFailure: observed average cost of a failed attempt (minutes). const FINDINGS = [ { name: "book not found on shelf", mentions: 7, sources: ["interview", "task", "journey"], failedAttempts: 42, minutesPerFailure: 4.8 }, { name: "search lost on return", mentions: 4, sources: ["interview", "task", "current state"], failedAttempts: 35, minutesPerFailure: 3.1 }, { name: "same-name record causes confusion", mentions: 7, sources: ["interview", "task"], failedAttempts: 16, minutesPerFailure: 1.6 }, { name: "renewal path not found", mentions: 2, sources: ["interview", "task"], failedAttempts: 18, minutesPerFailure: 2.4 }, { name: "refinements not visible", mentions: 3, sources: ["interview"], failedAttempts: 12, minutesPerFailure: 1.9 }, { name: "list unreadable on narrow screen", mentions: 2, sources: ["interview", "task"], failedAttempts: 9, minutesPerFailure: 2.7 }, ]; const PERIODS = 13; // 13 four-week periods per year for (const f of FINDINGS) f.lost = f.failedAttempts * f.minutesPerFailure; const total = FINDINGS.reduce((t, f) => t + f.lost, 0); console.log("finding mentions sources lost min/4 wks share annual hours"); for (const f of [...FINDINGS].sort((a, c) => c.lost - a.lost)) { console.log( `${f.name.padEnd(37)} ${String(f.mentions).padStart(6)} ${String(f.sources.length).padStart(8)} ` + `${f.lost.toFixed(1).padStart(15)} ${((f.lost / total) * 100).toFixed(1).padStart(4)}% ${((f.lost * PERIODS) / 60).toFixed(1).padStart(11)}` ); } console.log(`\ntotal: ${total.toFixed(1)} min / 4 wks = ${((total * PERIODS) / 60).toFixed(1)} hours / year`); // The two rankings do not agree const rankBy = (key) => [...FINDINGS].sort((a, b) => b[key] - a[key]).map((f) => f.name); const mentionRank = rankBy("mentions"), lostRank = rankBy("lost"); console.log("\nrank mentioned in interviews lost time"); for (let i = 0; i < FINDINGS.length; i++) { console.log(`${String(i + 1).padStart(4)} ${mentionRank[i].padEnd(37)} ${lostRank[i]}`); } // Cumulative share: how many findings account for most of the loss console.log("\ncumulative share"); let cumulative = 0; [...FINDINGS].sort((a, b) => b.lost - a.lost).forEach((f, i) => { cumulative += f.lost; console.log(` first ${i + 1} finding${i === 0 ? "" : "s"}: ${((cumulative / total) * 100).toFixed(1)}%`); }); // Evidence strength: a single-source finding is verified before entering the problem statement const weak = FINDINGS.filter((f) => f.sources.length < 2); console.log(`\nsingle-source finding: ${weak.map((f) => f.name).join(", ") || "none"} (verified before entering the problem statement)`); ``` ``` finding mentions sources lost min/4 wks share annual hours book not found on shelf 7 3 201.6 47.3% 43.7 search lost on return 4 3 108.5 25.5% 23.5 renewal path not found 2 2 43.2 10.1% 9.4 same-name record causes confusion 7 2 25.6 6.0% 5.5 list unreadable on narrow screen 2 2 24.3 5.7% 5.3 refinements not visible 3 1 22.8 5.4% 4.9 total: 426.0 min / 4 wks = 92.3 hours / year rank mentioned in interviews lost time 1 book not found on shelf book not found on shelf 2 same-name record causes confusion search lost on return 3 search lost on return renewal path not found 4 refinements not visible same-name record causes confusion 5 renewal path not found list unreadable on narrow screen 6 list unreadable on narrow screen refinements not visible cumulative share first 1 finding: 47.3% first 2 findings: 72.8% first 3 findings: 82.9% first 4 findings: 88.9% first 5 findings: 94.6% first 6 findings: 100.0% single-source finding: refinements not visible (verified before entering the problem statement) ``` Three things can be read from this. **Mentions are not cost.** "Same-name record causes confusion" was mentioned seven times in interviews and ranks second by mentions, fourth by lost time. The reason is that its cost is low: the user who opens the wrong record notices the mistake within a minute and a half and goes back. Phenomena mentioned often in interviews are the most annoying, not the most expensive. Annoyance and cost are separate metrics, and both are written down, but they are not conflated. **Loss is concentrated.** The first two findings account for 72.8% of the total loss, the first three for 82.9%. Trying to fix all six findings means three times the work of fixing the first two, and it adds only a quarter more of the gain. This is what the priority decision rests on. **Evidence strength is a separate column.** The "refinements not visible" finding comes only from interviews; it has no independent counterpart in either the task data or the decision list. A single-source finding need not be wrong, but it is verified with a second method before it enters the problem statement. Writing a low-evidence finding into the statement lends research's authority to an unverified claim. The annual-hours column is an estimate, and its limit must be written down explicitly: it is a four-week observation multiplied by thirteen periods, and it does not account for seasonal variation. It assumes that attempts per period stay the same outside the observed period too. Left unstated, this assumption makes the number read like a measurement rather than a product. #### The Parts of a Problem Statement The finding at the top of the ranking is not a problem — it is an observation. A problem statement carries five parts. - **Who.** Which persona is affected. Not "users" — which of the three personas built in the second lesson. - **Context.** Which task, which stage, and under what condition the problem occurs. - **Fact.** What does not happen. It is written as an observable shortfall, not as a request. - **Evidence.** With which measurement and which method. The number and the source are written together. - **Criterion.** If the problem counts as solved, what will the number become. A statement with no criterion can never close. The first finding's statement is written as follows: > Users in the Known-Record Searcher and Narrow-Screen Viewer personas cannot find the > book once they reach the shelf with the shelf code. Nineteen of the fifty-eight users > who enter the shelf-search stage give up at that stage (32.8%); 201.6 person-minutes > are spent with no result in a four-week period. The finding is seen independently in > three sources: seven of twelve interviews, forty-two failed attempts in the task data, > and the drop-off point in the journey map. The problem counts as solved once the > give-up rate at the shelf-search stage falls below 15%. No interface element appears in this text. How the shelf code will be shown, whether a map is added, whether a floor plan needs to be placed — all of this falls outside the statement; all are candidates that could satisfy it, and a choice will be made among them. #### Ways a Statement Breaks Down Three breakdowns are common, and all three make the statement untestable. **A solution seeping into the statement.** "The record detail is missing a shelf map" is not a problem statement — it is a chosen solution. The sign of the leak is this: if the statement names an interface element, the design decision has already been made, and research is being used to justify it. The distinguishing criterion is the same one set in the third lesson — at least two different solutions must be derivable from the statement. **Blaming the user.** The sentence "users do not read the shelf code correctly" writes the fact as the user's fault and puts the design out of scope. When the same fact is written as "the shelf code is not presented in a form that can be matched while standing at the shelf," the solution space reopens. The rule is this: **a problem statement describes what the interface fails to do, not what the user fails to do.** **A missing criterion.** A statement with no criterion never closes; every new design counts as "better," and none counts as enough. When the criterion is chosen, a reachable target is set; aiming to bring a 32.8% rate down to zero means ignoring the cases where the book really is checked out. #### The Ethical Burden of a Problem Statement The statement is research's public-facing side, and it carries every obligation the data has. **The statement does not show the participant.** The numbers that appear in the text must be aggregated. The sentence "the participant who is a night-shift attendant cannot find it at the shelf" points to one person; this information stays not in the statement but in the deleted raw note. **Those left out of scope are written down.** If the sample has no participant who uses a screen reader, the statement says nothing about that group's problem. Silence is not absence; a line reading "this group was not measured" is written under the statement. The rule set for personas in the second lesson is repeated here at the level of the statement. **A number's source stays traceable.** Every number in the statement must be traceable back to which method and which period it came from. A number whose source cannot be shown is removed from the statement; a number with no source is decoration, not evidence. #### Summary - Findings from different methods cannot be compared until they are converted into a common unit — the time the user pays. - The finding mentioned most in interviews is not the most expensive finding; in the sample data, one finding ranked second by mentions and fourth by cost. - Loss is concentrated: in the sample data, the first two findings accounted for 72.8% of the total, and the priority decision rests on this concentration. - Evidence strength is a separate column; a single-source finding is verified with a second method before it enters the problem statement. - A problem statement carries the parts who, context, fact, evidence, and criterion; a statement with no criterion never closes, and at least two different solutions must be derivable from it. - The statement describes what the interface fails to do, not what the user fails to do; it does not show the participant, and it explicitly writes down groups that were not measured. #### Next Step The problem is defined: the give-up rate at the shelf-search stage will be reduced, and the Subject Browser's comparison path will be shortened. The solution space is still open, and the first question is not the screens themselves but the structure between them. The catalog interface carries thousands of records, dozens of subjects, and a handful of operations; which headings these get grouped under is not a matter of style. The next lesson shows how structure is derived from content: extracting a content inventory, collecting card-sorting data, and building a similarity matrix from the pattern of how participants group items together. ## Information Architecture and Flow From content inventory to grouping, from navigation models to flow diagrams; wireframe and prototype fidelity levels, usability heuristics, and finding problems with a small sample. ### Content Inventory and Grouping Source: https://academia.sh/en/courses/user-experience/content-inventory-and-grouping Deriving structure from content; the inventory's columns, building a similarity matrix from card-sorting data, threshold-based grouping, and detecting cards without consensus. The problem is defined and the solution space is clear. The first question is not the screens themselves but the structure between them. The catalog interface carries thousands of records, dozens of subjects, and a set of operations; which headings these fall under is not a matter of style. A wrong grouping leads the user to search for what they want in the wrong place, a loss no search box can rescue. **Information architecture** is the grouping and naming of content and the establishment of the relationships between its pieces. This lesson shows where the structure is derived from: first an inventory that counts what exists, then a study that measures how users group these items. #### What the Inventory Counts **Content inventory** is the list of every item the interface carries. Grouping is impossible without the list; what is being grouped is unknown otherwise. The inventory has five columns. - **Item.** The name of the content. Not the screen name, but the content itself: "borrowing terms" is an item, "help page" is a container. - **Type.** Is it a record, an explanatory text, an operation, or an external link. Type determines not how the item is displayed, but how it can be grouped. - **Task link.** Which task the item serves. It connects to the task inventory built in the third lesson. An item that cannot be linked to any task stays in the inventory as a question mark. - **Owner.** Who updates the content. Content without an owner grows stale, and stale content is wrong information. - **Last updated.** How long the item has gone unchanged. The last two columns do not directly affect the grouping decision, but they are the inventory's most useful part: content that should be deleted before the structure is built shows up here. In the catalog interface, an item that cannot be linked to any task and has not been updated in two years is not an item to group but one to remove. #### User Grouping Is Measured Having the design team do the grouping results in the organization copying its own structure onto the interface. In the library's internal organization, the "circulation unit" and the "reference unit" may be separate; the user does not know this distinction and does not search for the "interlibrary loan" item by which unit runs it. **Card sorting** is giving participants the inventory's items as cards and asking them to group them their own way. If the participant assigns the group names, it is open sorting; if the names are given in advance, it is closed sorting. Open sorting measures structure and naming together; closed sorting serves to test an existing structure. The result is a stack of patterns and cannot be read one by one. Ten participants' ten separate sortings are reduced to how often the cards were grouped together. ```js // card-sorting.mjs — similarity matrix and grouping from card-sorting data // The 15 content items in the inventory and each item's expected cluster (a hypothesis to verify) const CARDS = [ ["search tips", "discover"], ["subject headings", "discover"], ["new arrivals", "discover"], ["databases", "discover"], ["borrowing terms", "borrow"], ["late fees", "borrow"], ["renewal", "borrow"], ["reservation", "borrow"], ["interlibrary loan", "borrow"], ["shelf plan", "on-site"], ["study rooms", "on-site"], ["opening hours", "on-site"], ["membership signup", "account"], ["password reset", "account"], ["contact", "unclear"], ]; const CATEGORIES = ["discover", "borrow", "on-site", "account"]; // Deterministic pseudo-random generator (mulberry32) function makeRng(seed) { let a = seed >>> 0; return () => { a = (a + 0x6d2b79f5) >>> 0; let t = Math.imul(a ^ (a >>> 15), 1 | a); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; } const rng = makeRng(20260101); // How the 10 participants group the cards: 80% match to the expected category, the "unclear" card is free const PARTICIPANTS = 10; const sorts = []; for (let k = 0; k < PARTICIPANTS; k++) { sorts.push( CARDS.map(([, expected]) => expected === "unclear" || rng() > 0.8 ? CATEGORIES[Math.floor(rng() * CATEGORIES.length)] : expected ) ); } // Similarity matrix: how many participants put i and j in the same group const n = CARDS.length; const M = Array.from({ length: n }, () => new Array(n).fill(0)); for (const s of sorts) for (let i = 0; i < n; i++) for (let j = 0; j < n; j++) if (i !== j && s[i] === s[j]) M[i][j]++; console.log("similarity matrix (how many of the 10 participants grouped them together)"); console.log(" ".repeat(24) + CARDS.map((_, i) => String(i + 1).padStart(3)).join("")); CARDS.forEach(([name], i) => { console.log( `${String(i + 1).padStart(2)} ${name.padEnd(21)}` + M[i].map((v, j) => (i === j ? " ." : String(v).padStart(3))).join("") ); }); // Distance = 1 - similarity; merge with average linkage, stop at threshold 0.5 const dist = (i, j) => 1 - M[i][j] / PARTICIPANTS; let groups = CARDS.map((_, i) => [i]); const linkageDistance = (a, b) => { let t = 0; for (const i of a) for (const j of b) t += dist(i, j); return t / (a.length * b.length); }; const THRESHOLD = 0.5; while (groups.length > 1) { let best = Infinity, pair = null; for (let i = 0; i < groups.length; i++) for (let j = i + 1; j < groups.length; j++) { const d = linkageDistance(groups[i], groups[j]); if (d < best) { best = d; pair = [i, j]; } } if (best > THRESHOLD) break; const [i, j] = pair; groups = groups.filter((_, x) => x !== i && x !== j).concat([groups[i].concat(groups[j])]); } groups = groups.map((g) => g.sort((a, b) => a - b)).sort((a, b) => a[0] - b[0]); console.log(`\ngroups formed at threshold ${THRESHOLD} (${groups.length} groups)`); for (const g of groups) console.log(` ${g.map((i) => CARDS[i][0]).join(", ")}`); // Each card's affinity gap between its own group and the nearest second group console.log("\ncard own group nearest other gap"); const avgSimilarity = (i, g) => { const d = g.filter((x) => x !== i); return d.length ? d.reduce((t, j) => t + M[i][j], 0) / d.length / PARTICIPANTS : 0; }; for (let i = 0; i < n; i++) { const own = groups.find((g) => g.includes(i)); const a = avgSimilarity(i, own); const others = groups.filter((g) => g !== own).map((g) => avgSimilarity(i, g)); const b = others.length ? Math.max(...others) : 0; console.log( `${CARDS[i][0].padEnd(20)} ${a.toFixed(2).padStart(9)} ${b.toFixed(2).padStart(13)} ${(a - b).toFixed(2)}${a - b < 0.2 ? " UNCLEAR" : ""}` ); } ``` ``` similarity matrix (how many of the 10 participants grouped them together) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 search tips . 9 8 10 1 1 0 0 1 0 1 1 2 0 2 2 subject headings 9 . 7 9 1 1 0 0 1 0 2 2 2 0 2 3 new arrivals 8 7 . 8 0 2 1 1 2 0 0 1 2 1 3 4 databases 10 9 8 . 1 1 0 0 1 0 1 1 2 0 2 5 borrowing terms 1 1 0 1 . 8 8 9 7 0 1 0 1 0 2 6 late fees 1 1 2 1 8 . 8 9 7 0 0 1 0 0 3 7 renewal 0 0 1 0 8 8 . 9 7 0 1 0 1 1 3 8 reservation 0 0 1 0 9 9 9 . 8 0 1 0 0 0 3 9 interlibrary loan 1 1 2 1 7 7 7 8 . 0 1 0 1 1 3 10 shelf plan 0 0 0 0 0 0 0 0 0 . 6 8 2 1 5 11 study rooms 1 2 0 1 1 0 1 1 1 6 . 7 0 1 3 12 opening hours 1 2 1 1 0 1 0 0 0 8 7 . 1 0 3 13 membership signup 2 2 2 2 1 0 1 0 1 2 0 1 . 7 2 14 password reset 0 0 1 0 0 0 1 0 1 1 1 0 7 . 1 15 contact 2 2 3 2 2 3 3 3 3 5 3 3 2 1 . groups formed at threshold 0.5 (5 groups) search tips, subject headings, new arrivals, databases borrowing terms, late fees, renewal, reservation, interlibrary loan shelf plan, study rooms, opening hours membership signup, password reset contact card own group nearest other gap search tips 0.90 0.20 0.70 subject headings 0.83 0.20 0.63 new arrivals 0.77 0.30 0.47 databases 0.90 0.20 0.70 borrowing terms 0.80 0.20 0.60 late fees 0.80 0.30 0.50 renewal 0.80 0.30 0.50 reservation 0.88 0.30 0.57 interlibrary loan 0.72 0.30 0.42 shelf plan 0.70 0.50 0.20 UNCLEAR study rooms 0.65 0.30 0.35 opening hours 0.75 0.30 0.45 membership signup 0.70 0.20 0.50 password reset 0.70 0.10 0.60 contact 0.00 0.37 -0.37 UNCLEAR ``` #### Reading the Matrix The matrix shows a structure that cannot be extracted from the individual sortings. The first and fourth cards — search tips and databases — were placed in the same group by ten of the ten participants. The fifth and eighth cards were together for nine participants. These values are a measure of agreement: a high count shows that users regard the two items as part of the same task. The threshold was set at 0.5; that is, two items enter the same group if they were grouped together by at least half the participants. The threshold is not a law of nature but an auditable decision. Raising it produces more, smaller groups; lowering it merges groups. What matters is that the threshold is written down: the sentence "these should stay together" is debatable, while "co-grouping rate 0.7, threshold 0.5" is auditable. Five groups emerged, and four of them are internally consistent. The fifth "group" carries a single card. #### The Card Without Consensus Is the Structure's Most Important Finding The affinity gap in the last table is the difference between each card's average affinity to its own group and its affinity to the nearest second group. A large gap means the card has found its place; a small gap means the participants did not reach consensus. The **contact** card did not stay inside any group. Its highest affinity is 0.37, to the on-site group; it does not clear any threshold. This is not a data flaw but a finding: users do not agree on where to look for contact information. Three options exist for such a card, and the choice is made from the data. - **It is shown in multiple places.** If the information is cheap and short, repeating it is harmless; contact information can appear in both the on-site group and the account group. - **It is taken out of the structure.** Some items belong not to groups but to an area reachable from every screen. The footer exists for this. - **It is split.** "Contact" may not be a single item: asking a question about a shelf and reporting a membership problem are separate tasks and go to separate groups. The lack of consensus can be a sign that the item itself splits in two. The **shelf plan** card was also flagged: its affinity to its own group is 0.70, but its co-grouping rate with the contact card is 0.50. Half the participants treat asking where a shelf is and looking at the shelf plan as the same task. This is the information-architecture side trace of the "book cannot be found on the shelf" problem measured in the fourth lesson: the user does not know where to look for shelf information. #### Naming the Groups The groups now exist but have no names. In the open form of card sorting, the name is derived from the names the participants gave themselves; among the twelve phrases naming the same group, the most frequently used word is chosen. The organization's internal vocabulary must not intrude here: if users do not say "circulation services," the heading does not become "circulation services." Two criteria apply to naming. The name must cover the group's **entirety**; the heading "renewal" names only one of the five items in the borrowing group. The name must be **distinct** from a neighboring group; when the headings "library information" and "general information" stand side by side, the user cannot choose which to enter and, unable to choose, tries both. #### The Ethical Framework of Card Sorting Card sorting is a research session, and the rules established in the User Research Methods lesson apply here as well: the participant gives consent knowing what the sorting is for, the session record is kept under a code name, and the raw recording is deleted within the retention limit after analysis. No single participant's sorting appears in the matrix above; only the co-grouping counts do. Two more rules apply to the inventory itself. **Content carrying personal data is flagged in the inventory**: borrowing history, membership information, and search history are tied to a person, and their grouping decision is made together with the access decision. **Every personal-data item in the inventory has its retention period written down**; an item without a written period is being stored for an unspecified duration before it is even placed anywhere in the interface. #### Summary - The content inventory carries item, type, task-link, owner, and last-updated columns; an item that cannot be linked to any task and is not updated is one to remove, not to group. - Card sorting ensures that structure is derived from the user's mental grouping rather than the organization's internal structure; the open form measures structure and naming together. - Individual sortings cannot be read; the data is reduced to a similarity matrix giving the co-grouping count for each pair of cards and grouped by a written threshold. - A card with a low affinity gap is a finding, not a flaw; in the sample data, one card could not be attached to any group, and the solution is repetition, moving it outside the structure, or splitting the item. - Group names are chosen from the participants' own words; the name must cover the group's entirety and be distinct from a neighboring group. - Content carrying personal data is flagged in the inventory and written down together with its retention period. #### Next Step Five groups and group names have been obtained, but this is not yet a navigation. How many of the groups are shown at once, how many levels the sub-breakdowns descend, and how many clicks a user needs to reach an item are separate decisions. The next lesson compares hierarchical, flat, and matrix navigation models, computes the trade-off between menu width and depth, and finds the width that minimizes total scanning cost. ### Navigation Models Source: https://academia.sh/en/courses/user-experience/navigation-models Comparing hierarchical, flat, and matrix navigation; computing the trade-off between menu width and depth, and how reorientation cost shifts the best width. Card sorting produced five groups and group names. This is not yet a navigation. How many of the groups are shown at once, how many levels the sub-breakdowns descend, and how many steps a user needs to reach an item are separate decisions, and they depend on each other: as the menu narrows the tree deepens, and as it widens the tree shallows. This lesson distinguishes three navigation models, then computes the trade-off between depth and width with a cost model. The result is not a rule of the "a menu should have seven items" kind, but a computation that gives which width is cheapest under which condition. #### Three Models **Hierarchical navigation** places items in nested groups. The user starts at the root, makes a choice at each level, and descends to a leaf. This is the catalog interface's subject tree. Its strength is that it reduces the options at every level; its weakness is that the hierarchy imposes a **single order** — the question of subject-first or material-type-first must have one answer, and the user pays a cost if the order in their mind differs. **Flat navigation** shows all items at a single level. There is no depth, the user never gets lost, everything is one step away. It does not scale: as the item count grows, scanning cost grows linearly. **Matrix navigation** gives access to the same items from multiple independent axes. If a record in the catalog interface can be reached both by subject and by material type, matrix navigation is present. Its strength is that the user can pick their own axis; its cost is that each axis needs separate maintenance. #### The Trade-off Between Width and Depth With the number of leaf nodes fixed in a tree, menu width and depth are inversely proportional: with width `b` and leaf count `N`, depth is approximately `log_b N`. As width grows, the items to read at each level grow; as depth shrinks, page changes and reorientation shrink. Which width is cheaper depends on the ratio between these two costs. ```js // depth-and-width.mjs — the trade-off between menu width and depth const N = 1000; // number of leaf nodes to reach const REORIENT = 1.2; // page change and reorientation per level (seconds) const ITEM = 0.12; // reading one menu item (seconds) const depth = (b) => Math.ceil(Math.log(N) / Math.log(b)); console.log(`N = ${N} leaf nodes, ${REORIENT} s reorientation per level, ${ITEM} s reading per item`); console.log("width depth items scanned total time"); let best = { b: 0, time: Infinity }; for (const b of [2, 3, 4, 5, 6, 7, 8, 10, 12, 16, 20, 32, 64]) { const d = depth(b); const time = d * (REORIENT + ITEM * b); if (time < best.time) best = { b, time }; console.log( `${String(b).padStart(5)} ${String(d).padStart(5)} ${String(b * d).padStart(13)} ${time.toFixed(2).padStart(10)} s` ); } console.log(`\nsmallest time: width ${best.b}, ${best.time.toFixed(2)} s`); // Which width would be chosen if only the number of items scanned were measured let fewestItems = { b: 0, items: Infinity }; for (let b = 2; b <= 64; b++) { const t = b * depth(b); if (t < fewestItems.items) fewestItems = { b, items: t }; } console.log(`if only items scanned were measured: width ${fewestItems.b}, ${fewestItems.items} items`); // How the best width shifts as reorientation cost changes console.log("\nreorientation per level best width depth time"); for (const s of [0.0, 0.4, 1.2, 3.0, 6.0]) { let top = { b: 0, time: Infinity, d: 0 }; for (let b = 2; b <= 64; b++) { const d = depth(b), time = d * (s + ITEM * b); if (time < top.time) top = { b, time, d }; } console.log( `${s.toFixed(1).padStart(23)} ${String(top.b).padStart(10)} ${String(top.d).padStart(5)} ${top.time.toFixed(2)} s` ); } ``` ``` N = 1000 leaf nodes, 1.2 s reorientation per level, 0.12 s reading per item width depth items scanned total time 2 10 20 14.40 s 3 7 21 10.92 s 4 5 20 8.40 s 5 5 25 9.00 s 6 4 24 7.68 s 7 4 28 8.16 s 8 4 32 8.64 s 10 3 30 7.20 s 12 3 36 7.92 s 16 3 48 9.36 s 20 3 60 10.80 s 32 2 64 10.08 s 64 2 128 17.76 s smallest time: width 10, 7.20 s if only items scanned were measured: width 2, 20 items reorientation per level best width depth time 0.0 2 10 2.40 s 0.4 4 5 4.40 s 1.2 10 3 7.20 s 3.0 10 3 12.60 s 6.0 32 2 19.68 s ``` Three results emerge. **The best width depends on the metric.** If only the number of items scanned were counted, the binary menu would win: twenty items are read, the smallest value in the table. But ten levels must be descended, and each level carries a reorientation cost. When duration is measured, the binary menu becomes one of the most expensive options: 14.40 seconds. **Total time is a flat curve.** Between width 6 and 12, time stays between 7.20 and 8.64 seconds. Within this range, the choice belongs not to the computation but to other constraints: how many items fit on a narrow screen, the length of group names, the natural number of groups that came out of card sorting. The computation does not decide; it says within which range the decision is free. **If reorientation is expensive, the menu widens.** If per-level reorientation were zero, the best width would be 2; it rises to 10 at 1.2 seconds and to 32 at 6.0 seconds. The design equivalent of this is a direct rule: if page transitions are slow, if the user's place in the list is lost during a transition, or if the user has to reorient at every level, **a deep tree becomes expensive and the menu is widened**. Conversely, if the level transition is instant and context is preserved, a deep tree is cheap. This connects directly to a phenomenon measured in the fifth lesson: in our interface, the search state is not preserved on return. The state not being preserved raises the reorientation cost, and the higher reorientation cost makes depth more expensive. The same flaw bills twice, in two separate places. #### The User's Axis and the Menu's Axis The order the hierarchy imposes produces a measurable cost. The catalog collection can be classified along two axes: twenty-four subject headings and five material types. ```js // navigation-models.mjs — scanning cost of flat, hierarchical, and matrix navigation const TOPIC = 24; // number of subject headings const TYPE = 5; // number of material types const LEAVES = TOPIC * TYPE; // 120 leaf nodes const BACKTRACK = 1; // back step after starting on the wrong axis // Average scan: half the list is read before the target is found const flat = LEAVES / 2; const matched = TOPIC / 2 + TYPE / 2; // the user's axis matches the menu's axis const mismatched = TOPIC + TYPE / 2 + BACKTRACK; // user thinks in type first, menu starts with topic const matrix = TOPIC / 2 + TYPE / 2; // user starts from their own axis console.log(`collection: ${TOPIC} topics x ${TYPE} types = ${LEAVES} leaf nodes`); console.log("\nmodel entry points steps to leaf avg items scanned"); console.log(`${"flat".padEnd(18)} ${String(LEAVES).padStart(13)} ${String(1).padStart(14)} ${flat.toFixed(1).padStart(19)}`); console.log(`${"hierarchical".padEnd(18)} ${String(TOPIC).padStart(13)} ${String(1).padStart(14)} ${"variable".padStart(19)}`); console.log(`${"matrix".padEnd(18)} ${String(TOPIC + TYPE).padStart(13)} ${String(2).padStart(14)} ${matrix.toFixed(1).padStart(19)}`); console.log("\nin the hierarchical model, does the user's axis match the menu"); console.log(` matches : ${matched.toFixed(1)} items`); console.log(` does not match: ${mismatched.toFixed(1)} items`); console.log("\nshare thinking by type axis hierarchical matrix gap"); for (const p of [0, 0.1, 0.2, 0.3, 0.4, 0.5]) { const h = matched * (1 - p) + mismatched * p; console.log( `${p.toFixed(2).padStart(28)} ${h.toFixed(2).padStart(12)} ${matrix.toFixed(2).padStart(6)} ${(h - matrix).toFixed(2)}` ); } // The matrix's cost: both axes must cover every leaf node console.log("\nmatrix's maintenance load"); console.log(` links to maintain on the topic axis : ${LEAVES}`); console.log(` links to maintain on the type axis : ${LEAVES}`); console.log(` links to maintain in the hierarchy : ${LEAVES}`); console.log(` matrix total : ${2 * LEAVES} (${((2 * LEAVES) / LEAVES).toFixed(1)}x)`); ``` ``` collection: 24 topics x 5 types = 120 leaf nodes model entry points steps to leaf avg items scanned flat 120 1 60.0 hierarchical 24 1 variable matrix 29 2 14.5 in the hierarchical model, does the user's axis match the menu matches : 14.5 items does not match: 27.5 items share thinking by type axis hierarchical matrix gap 0.00 14.50 14.50 0.00 0.10 15.80 14.50 1.30 0.20 17.10 14.50 2.60 0.30 18.40 14.50 3.90 0.40 19.70 14.50 5.20 0.50 21.00 14.50 6.50 matrix's maintenance load links to maintain on the topic axis : 120 links to maintain on the type axis : 120 links to maintain in the hierarchy : 120 matrix total : 240 (2.0x) ``` Flat navigation makes the user scan an average of sixty items out of one hundred twenty; hierarchy and matrix, fourteen and a half. This shows that grouping itself saves more than a quarter, and it gives flat navigation's limit: it is good up to a few dozen items, not at hundreds. The difference between hierarchy and matrix depends on how many users think along the menu's axis. If everyone thinks by subject, the gap is zero. If a fifth think by material type, the hierarchy is 2.6 items more expensive; if half do, 6.5 items. The matrix's gain is not a fixed advantage but a gain directly proportional to **the rate of axis mismatch**. This rate is not estimated, it is measured. The card-sorting data from the previous lesson exists exactly for this: whichever axis the participants grouped the cards along is navigation's primary axis. The matrix's cost is in the last block: because every leaf node sits on both axes, the number of links to maintain doubles. This is not a runtime cost but a maintenance cost. When a new subject heading is added, both axes must be updated, and if one axis falls out of date, the same content shows two different realities in two different places. Matrix navigation is for teams that can commit to keeping two navigations consistent. #### The Model's Limits The calculations above rest on two assumptions, and both must be stated explicitly. First, it is assumed that the user scans the menu start to finish and recognizes the right item. In reality the user sometimes does **not recognize** the right item: someone looking for "interlibrary loan" may not think to enter the "borrowing operations" group. In this case the cost is not scanning but entering the wrong branch and backtracking, and it is not in the model. Second, the search box was not accounted for. In an interface where search exists, navigation is not the only path; some users search without ever entering a menu. Navigation structure is still needed: search serves those who know what they are looking for, navigation serves those who do not. The **Subject Browser** persona built in the second lesson is exactly in the second group. #### Summary - Hierarchical navigation reduces the options at every level but imposes a single order; flat navigation does not scale; matrix navigation gives access from multiple axes. - Width and depth are inversely proportional, and the best width depends on the metric: a narrow menu wins if the number of items scanned is minimized, a wide menu wins if duration is minimized. - In the sample model, total time stays flat between width 6 and 12; the computation does not decide, it says within which range the decision is free. - As per-level reorientation cost rises, the best width grows; flaws such as state not being preserved make reorientation more expensive and, with it, deep trees. - The matrix's advantage over the hierarchy is proportional to how many users think outside the menu's axis, and its cost is that the number of links to maintain doubles. #### Next Step Navigation structure explains how items are reached, but not the decisions inside a task. The act of borrowing is not a single click: membership is verified, the record's status is checked, a reservation is offered if it's checked out, and the transaction stops if the member has an overdue book. When these branches are described in prose, which state connects to which gets lost. The next lesson builds the flow as a graph and searches for two flaws by computation: a state unreachable from the start, and a node with no exit. ### Flow Diagrams Source: https://academia.sh/en/courses/user-experience/flow-diagrams Building the flow as a graph; searching by computation for unreachable states and dead-end nodes, distinguishing whether a cycle is a flaw, and how path count determines testing load. Navigation structure explains how items are reached, but not the decisions inside a task. Borrowing is not a single click: whether the session is active is checked, the record's status is checked, a reservation is offered if the record is checked out, and the transaction stops if the member has an overdue book. When these branches are described in prose, which state connects to which gets lost, and missing branches go unnoticed. A **flow diagram** makes this structure explicit: every state is a node, every transition is an edge. The diagram's value is not in being a drawing but in being **auditable**. The graph concept introduced in the Data Structures course is put to direct use here: the moment a flow is written as a graph, the questions asked of it are answered by computation. #### The Flow's Nodes and Transitions There are three node types, and they must be kept separate. - **State node.** A place the user is: the login screen, record detail, borrow confirmation. The user sees something here. - **Decision node.** A point where the system or the user produces a branch: is the session active, is there an overdue book. A decision node's outputs must be **mutually exclusive** and must **cover every case**; if only one branch leaves a two-way decision, the diagram is incomplete. - **End node.** Where the flow completes. A flow can have more than one ending, and not all of them have to be success; giving up is also an ending, and it is written down. An ending that is not written is an ending that was not designed. The user still gives up, but where they land is undefined. #### Searching for Two Flaws Once the graph is written, two flaws are searched for by computation. An **unreachable state** is a node no sequence of transitions from the start can reach; it is a screen that was designed but never wired into the flow. A **dead-end node** is a node that cannot reach any ending; it is a place the user can fall into and never leave. ```js // flow.mjs — unreachable states, dead-end nodes, and path counting in the flow graph // Nodes and transitions of the borrowing flow (data built for this lesson) const START = "start"; const ENDINGS = ["borrow successful", "reservation successful", "give up"]; const TRANSITIONS = { "start": ["session active?"], "session active?": ["record status", "login screen"], "login screen": ["record status", "login error"], "login error": ["login screen", "give up"], "record status": ["on shelf", "checked out"], "on shelf": ["overdue book?"], "overdue book?": ["pay fine", "borrow confirmation"], "pay fine": ["borrow confirmation"], "borrow confirmation": ["borrow successful", "system error"], "checked out": ["reservation offer"], "reservation offer": ["reservation confirmation", "give up"], "reservation confirmation": ["reservation successful"], "not a member notice": ["give up"], "system error": [], "borrow successful": [], "reservation successful": [], "give up": [], }; const NODES = Object.keys(TRANSITIONS); // 1. Nodes reachable from the start const traverse = (start, neighbor) => { const g = new Set([start]), stack = [start]; while (stack.length) for (const k of neighbor(stack.pop())) if (!g.has(k)) { g.add(k); stack.push(k); } return g; }; const reachable = traverse(START, (d) => TRANSITIONS[d]); // 2. Nodes that can reach an ending (traversal on the reverse graph) const REVERSE = {}; for (const d of NODES) REVERSE[d] = []; for (const [d, neighbors] of Object.entries(TRANSITIONS)) for (const k of neighbors) REVERSE[k].push(d); const canReachEnd = new Set(); for (const b of ENDINGS) for (const d of traverse(b, (x) => REVERSE[x])) canReachEnd.add(d); console.log("node in out from start to an ending"); for (const d of NODES) { console.log( `${d.padEnd(24)} ${String(REVERSE[d].length).padStart(3)} ${String(TRANSITIONS[d].length).padStart(4)} ` + `${(reachable.has(d) ? "reachable" : "UNREACHABLE").padStart(13)} ${(canReachEnd.has(d) ? "reaches" : "NO EXIT").padStart(9)}` ); } const unreachable = NODES.filter((d) => !reachable.has(d)); const deadEnd = NODES.filter((d) => !canReachEnd.has(d)); console.log(`\nunreachable node : ${unreachable.join(", ") || "none"}`); console.log(`dead-end node : ${deadEnd.join(", ") || "none"}`); // 3. Is there a cycle (a path that loops back) const COLOR = {}; const cycles = []; (function search(d, path) { COLOR[d] = 1; for (const k of TRANSITIONS[d]) { if (COLOR[k] === 1) cycles.push([...path.slice(path.indexOf(k)), k].join(" -> ")); else if (!COLOR[k]) search(k, [...path, k]); } COLOR[d] = 2; })(START, [START]); console.log(`cycle : ${cycles.join(" | ") || "none"}`); // 4. Simple paths from the start to the endings const paths = []; (function walk(d, path) { if (ENDINGS.includes(d)) { paths.push(path); return; } for (const k of TRANSITIONS[d]) if (!path.includes(k)) walk(k, [...path, k]); })(START, [START]); console.log(`\nsimple path count: ${paths.length}`); const lengths = paths.map((y) => y.length - 1); console.log(`shortest path: ${Math.min(...lengths)} transitions, longest path: ${Math.max(...lengths)} transitions`); console.log("\nending path count shortest"); for (const b of ENDINGS) { const these = paths.filter((y) => y.at(-1) === b); console.log( `${b.padEnd(23)} ${String(these.length).padStart(9)} ${these.length ? Math.min(...these.map((y) => y.length - 1)) : "-"}` ); } console.log("\nshortest successful path:"); console.log(" " + paths.filter((y) => y.at(-1) === "borrow successful").sort((a, b) => a.length - b.length)[0].join(" -> ")); ``` ``` node in out from start to an ending start 0 1 reachable reaches session active? 1 2 reachable reaches login screen 2 2 reachable reaches login error 1 2 reachable reaches record status 2 2 reachable reaches on shelf 1 1 reachable reaches overdue book? 1 2 reachable reaches pay fine 1 1 reachable reaches borrow confirmation 2 2 reachable reaches checked out 1 1 reachable reaches reservation offer 1 2 reachable reaches reservation confirmation 1 1 reachable reaches not a member notice 0 1 UNREACHABLE reaches system error 1 0 reachable NO EXIT borrow successful 1 0 reachable reaches reservation successful 1 0 reachable reaches give up 3 0 reachable reaches unreachable node : not a member notice dead-end node : system error cycle : login screen -> login error -> login screen simple path count: 9 shortest path: 4 transitions, longest path: 8 transitions ending path count shortest borrow successful 4 6 reservation successful 2 6 give up 3 4 shortest successful path: start -> session active? -> record status -> on shelf -> overdue book? -> borrow confirmation -> borrow successful ``` #### What the Two Flaws Mean No transition enters the **not a member notice** node. This screen was designed: it tells a non-member visitor that they cannot borrow, and it connects to giving up. But the flow never lands on this state. Two branches leave the login screen — success and error — and the "this person is not a member yet" case is in neither. The flaw is not in the screen but in the decision node's missing branch. Unreachable screens are usually evidence that a decision was written incompletely. No transition leaves the **system error** node. If the user runs into a system error during borrow confirmation, they fall into this node and cannot reach any ending. The rule set in the Error and Warning States lesson of the Fundamentals of Interface Design course is restated here in the language of graphs: **every error state must have at least one exit.** The exit can be "retry," it can be "give up"; having none is not an option. The `start` node having an in-count of zero is not a flaw; that is true of a start node by definition. The start is declared separately so the check can tell the difference. #### A Cycle Is Not a Flaw The check found a cycle: between the login screen and the login error. When the user enters the wrong password, they fall into the error screen, return to the login screen from there, and try again. This is not a flaw but a designed retry. There is a single criterion for whether a cycle is a flaw: **does an edge leave the cycle?** The login-error node has two exits — the login screen and give up — so the user is never locked in the loop. A cycle with no exit is the multi-node version of a dead-end node. The cycle's second question is how many times it can be looped. Unlimited retries are a problem for password security, zero retries are a problem for usability; in the diagram this is stated as a condition written on the transition, and if the condition is not written, no decision has been made. #### Path Count Determines the Testing Load There are nine simple paths from the start to the endings. This number is directly a testing budget: fully testing the flow means running nine separate scenarios. The shortest path is four transitions (the path to giving up), the longest is eight. A successful borrow's shortest path is six transitions, and it has four different paths. The source of the four paths is two binary decisions: is the session active, and is there an overdue book. Two binary decisions produce four combinations; the count stays at four because the system-error branch is not added to it. The rapid growth of path count gives flow design's most practical metric: **every new binary decision roughly doubles the number of paths to test.** Whether a decision is really necessary is asked together with this cost. The "is there an overdue book" decision is necessary because it is the institution's rule; the "has the user viewed this record before" decision might offer convenience, but it doubles the testing load. The same computation also suggests a small simplification. The pay-fine node has a single exit, going straight to borrow confirmation; that is, no failure branch is written for after the fine is paid. If payment is an operation that can fail, the diagram is incomplete; if it cannot fail, the node is not a decision but a notice, and it can be dropped from the flow. Single-exit nodes always invite this question. #### The Diagram's Limit The flow graph shows states and transitions, not **time**. How long a transition takes, how long the user waits at a node, and what they see while waiting are not in the graph; these belong to the journey map and to component states. The second limit is that the graph shows the system's flow, not the flow **in the user's mind**. The user does not make a decision called "is there an overdue book"; they are surprised to see their fine on the screen. The diagram being correct does not mean the flow is understandable. Understandability is tested separately, and that is the work of this topic's last two lessons. #### Summary - A flow diagram is a graph: state nodes, decision nodes, and end nodes; a decision node's branches must be mutually exclusive and must cover every case. - An unreachable node is found by forward traversal, a dead-end node by traversing the reverse graph from the endings; in the sample flow, one uncovered a missing branch, the other a dead-end error state. - A cycle is not a flaw; the flaw is the absence of an edge leaving the cycle and the retry count not being tied to a condition. - Simple path count gives the testing load; every new binary decision roughly doubles the path count, and a decision's necessity is asked together with this cost. - Single-exit nodes point either to a missing failure branch or to a notice that can be dropped from the flow. #### Next Step The flow has been checked and the navigation structure built; both are consistent on paper. A consistent structure is not the same as a usable interface, and the only way to find out is to show it to someone. But what should be shown — a crude box outline, a clickable copy? The next lesson takes on the fidelity-level decision: it computes which question is answered at which level, how many tasks can be tested end to end for how many nodes a prototype covers, and why the two ratios are not equal. ### Wireframe and Prototype Source: https://academia.sh/en/courses/user-experience/wireframe-and-prototype Fidelity's three independent axes — visual, content, and interaction; separating a prototype's node coverage from its task coverage, and how the fidelity level affects preparation cost. The flow has been checked, the navigation structure built. Both are consistent on paper, but a consistent structure does not mean a usable interface. The only way to find out is to show the structure to someone and give them a task. The question is what to show. A crude outline made of boxes, a clickable copy, or a visually complete screen filled with real text? This lesson takes on the fidelity-level decision and computes how many tasks can be tested for how many screens a prototype covers. #### Wireframe and Prototype Answer Different Questions A **wireframe** shows a single screen's structure: what information exists, in what hierarchy it sits, what actions are available. The question it answers is "are the right things on this screen." A wireframe is static and single-screen; it cannot test a flow. A **prototype** also includes the transitions between screens. The question it answers is "can the user complete this task." A prototype is the flow diagram's runnable counterpart: nodes correspond to screens, transitions to clicks. The difference is not a difference in maturity but a difference in question. If a screen's content is what's in question, a wireframe is enough, and building a prototype is wasted effort. If whether the task is completed is what's in question, a wireframe says nothing. #### Fidelity Is Not a Single Axis Fidelity level is usually thought of as a single slider between "low" and "high." In reality there are three independent axes, and each can be set separately. - **Visual fidelity.** How close the color, typography, spacing, and state design are to the real thing. The low end is gray boxes; the high end is every decision built in the Fundamentals of Interface Design course, applied. - **Content fidelity.** Whether the text and data on the screen are real. The low end is placeholder text; the high end is the catalog's real records — including long titles, same-titled volumes, and empty metadata fields. - **Interaction fidelity.** How many transitions work. The low end has a single clickable path; the high end has every branch. The independence of these axes has a practical consequence: a prototype with **low visual fidelity and high content fidelity** is an efficient combination for catching microcopy and information-architecture problems. When real record titles are placed inside gray boxes, the truncation problem of long titles and the same-titled-record problem measured in the first topic become visible, while color decisions wait without diluting the discussion. The reverse combination — high visual fidelity, low content fidelity — is the most misleading. The participant assumes the screen they see is finished and comments on color rather than structure; also, because every placeholder text is the same length, no truncation problem ever surfaces. #### How Many Screens, How Many Tasks While preparing a prototype, how many nodes to implement is a budget decision. This decision's metric is not the share of screens covered but the share of tasks that can be tested end to end. ```js // prototype-coverage.mjs — a prototype's node coverage and task coverage are not the same const ALL_NODES = [ "start", "session active?", "login screen", "login error", "record status", "on shelf", "overdue book?", "pay fine", "borrow confirmation", "checked out", "reservation offer", "reservation confirmation", "borrow successful", "reservation successful", "give up", ]; // Each task's required sequence of nodes in the flow const TASKS = { "borrow (session active)": ["start", "session active?", "record status", "on shelf", "overdue book?", "borrow confirmation", "borrow successful"], "borrow (login required)": ["start", "session active?", "login screen", "record status", "on shelf", "overdue book?", "borrow confirmation", "borrow successful"], "pay fine and borrow": ["start", "session active?", "record status", "on shelf", "overdue book?", "pay fine", "borrow confirmation", "borrow successful"], "reserve a checked-out record": ["start", "session active?", "record status", "checked out", "reservation offer", "reservation confirmation", "reservation successful"], "give up after wrong password": ["start", "session active?", "login screen", "login error", "give up"], "give up on reservation": ["start", "session active?", "record status", "checked out", "reservation offer", "give up"], }; // Prototype versions: each version adds nodes to the previous one const ADDED = [ ["P1 main path", ["start", "session active?", "record status", "on shelf", "overdue book?", "borrow confirmation", "borrow successful"]], ["P2 login", ["login screen", "login error", "give up"]], ["P3 reservation", ["checked out", "reservation offer", "reservation confirmation", "reservation successful"]], ["P4 fine", ["pay fine"]], ]; const taskNames = Object.keys(TASKS); const covered = (set, g) => TASKS[g].every((d) => set.has(d)); console.log("version node node coverage testable tasks task coverage"); const set = new Set(); const versions = []; for (const [name, added] of ADDED) { added.forEach((d) => set.add(d)); const testable = taskNames.filter((g) => covered(set, g)); versions.push({ name, node: set.size, task: testable.length, set: new Set(set) }); console.log( `${name.padEnd(15)} ${String(set.size).padStart(5)} ${((set.size / ALL_NODES.length) * 100).toFixed(1).padStart(13)}% ` + `${String(testable.length).padStart(16)} ${((testable.length / taskNames.length) * 100).toFixed(1).padStart(15)}%` ); } // Of the nodes missing after a version, which one unlocks the most tasks console.log("\nafter P1: missing node tasks it blocks"); const p1 = versions[0].set; const missing = ALL_NODES.filter((d) => !p1.has(d)); const blocking = missing .map((d) => [d, taskNames.filter((g) => !covered(p1, g) && TASKS[g].includes(d)).length]) .sort((a, b) => b[1] - a[1]); for (const [d, c] of blocking) console.log(` ${d.padEnd(24)} ${c}`); // Fidelity level's cost: assumed preparation time per node const RATE = { "low (paper)": 0.5, "medium (clickable)": 2, "high (visual complete)": 6 }; console.log("\nfidelity level P1 P2 P3 P4 (hours)"); for (const [name, hours] of Object.entries(RATE)) { console.log( `${name.padEnd(24)} ${versions.map((s) => (s.node * hours).toFixed(1).padStart(5)).join(" ")}` ); } const largest = versions.at(-1).node; console.log(`\ntesting every task requires ${largest} nodes`); console.log(`${(largest * RATE["low (paper)"]).toFixed(1)} hours at low fidelity, ${(largest * RATE["high (visual complete)"]).toFixed(1)} hours at high fidelity (${(RATE["high (visual complete)"] / RATE["low (paper)"]).toFixed(0)}x)`); ``` ``` version node node coverage testable tasks task coverage P1 main path 7 46.7% 1 16.7% P2 login 10 66.7% 3 50.0% P3 reservation 14 93.3% 5 83.3% P4 fine 15 100.0% 6 100.0% after P1: missing node tasks it blocks login screen 2 checked out 2 reservation offer 2 give up 2 login error 1 pay fine 1 reservation confirmation 1 reservation successful 1 fidelity level P1 P2 P3 P4 (hours) low (paper) 3.5 5.0 7.0 7.5 medium (clickable) 14.0 20.0 28.0 30.0 high (visual complete) 42.0 60.0 84.0 90.0 testing every task requires 15 nodes 7.5 hours at low fidelity, 90.0 hours at high fidelity (12x) ``` #### Counting Screens Is Not Counting Tasks The first table's two columns do not track each other. The main-path prototype covers 46.7% of the nodes but can test only 16.7% of the tasks end to end. By the third version, node coverage climbs to 93.3%, while task coverage stays at 83.3%. The reason is a compound condition: a task is testable only if **all** the nodes it passes through are implemented. A single missing node renders the entire task unusable. A prototype that is half-ready can test not half the tasks but far fewer. This has a direct practical consequence: a prototype is built **from the task list, not the screen list.** The tasks to be tested are chosen first, then the set of nodes those tasks pass through is derived, and the prototype covers that set. Proceeding screen by screen produces a prototype where most of the prepared screens cannot be tested. The second table says which node to build first. After the main-path prototype, the nodes that unlock the most tasks are the login screen, the checked-out state, the reservation offer, and giving up; each blocks two tasks. This is the same logic as the flow diagram's path count, and it ties the priority discussion to a number. #### Cost Ratio, Not Cost Itself The hour values in the third table are assumed unit costs; the real numbers depend on the team, the tooling, and the screen's complexity. What matters is not the absolute durations but the ratio between them: a low-fidelity prototype costs twelve times less per node than a high-fidelity one. A prototype covering every task is 7.5 hours at low fidelity, 90 hours at high fidelity. Here is where this ratio determines the decision: **with the same budget, you can test one task in detail or twelve tasks roughly.** Which question is being asked determines this choice. The structural question — can the user complete the task — is answered with a wide, cheap prototype. The visual-decision question — do these two states stand apart clearly enough — is answered with a narrow, expensive prototype. A cheap prototype has a second advantage, more important than cost: because it is **disposable**, it is easy to criticize. A prototype that took ninety hours makes it hard for the team working on it to accept change proposals; both the designer and the participant lean toward defending the investment. #### Questions a Prototype Cannot Answer A prototype is a copy, and it systematically misleads on certain questions. **Data scale.** The prototype has ten records, the catalog has thousands. A results list working well with ten records proves nothing; truncation, sorting, and narrowing problems surface with scale. The cheapest way to raise content fidelity is to keep not the record count but the records' **variety** realistic: the longest title, two same-titled volumes, a record with missing metadata. **Error frequency.** In the prototype, the connection never drops, an operation never fails, the session never expires. The dead-end error node found in the flow diagram never appears in the prototype, because there is no path that falls into it. Error states are tested only if they are deliberately placed in the prototype. **Time.** In the prototype, every transition is instant. The design of waiting states — the Loading and Empty States lesson in the Fundamentals of Interface Design course — is not naturally tested in a prototype; unless delay is added by hand, the user never waits. **Learning.** A prototype session is a one-time encounter. A structure that looks hard on first use might be the fastest by the third use; a structure that looks easy on first use might be tiring in the long run. A prototype measures the first encounter, and this must be stated explicitly. #### Summary - A wireframe questions a single screen's content, a prototype questions whether a task can be completed; the difference between them is not maturity but the question asked. - Fidelity is three independent axes — visual, content, and interaction; the low-visual, high-content combination is most efficient at catching information-architecture and copy problems. - Node coverage and task coverage do not grow at the same rate; in the sample data, 46.7% node coverage could test only 16.7% of the tasks, because a single missing node blocks a task entirely. - A prototype is built from the task list, not the screen list; missing nodes are ranked by computing how many tasks each one unlocks. - What the fidelity level means is a ratio, not an absolute cost; with the same budget, either one task is tested in detail or many tasks are tested roughly. - A prototype systematically understates data scale, error frequency, wait time, and learning; these limits are written alongside the findings. #### Next Step The prototype is ready, but a cheap check can be run before calling in participants. A large share of known usability problems are found by reviewing the interface with an established list of evaluation criteria in hand. The next lesson takes on these criteria one principle at a time, computes how much multiple evaluators' findings overlap, and shows with a number why a single expert is not enough. ### Usability Heuristics Source: https://academia.sh/en/courses/user-experience/usability-heuristics Applying an established set of evaluation criteria principle by principle; computing how much multiple evaluators' findings overlap, and why a single evaluator misses the most severe problem. The prototype is ready, but a cheap check can be run before calling in participants. A portion of usability problems are found without ever showing the interface to a user, by reviewing it with a known set of criteria in hand. This method does not replace the participant session; it is done first so the session is not spent needlessly. A **usability heuristic** is a general criterion used when evaluating an interface. The name "heuristic" comes from its not being a strict rule: when a criterion is not met, the result is not a definite flaw but a candidate that needs examining. This lesson applies an established set of criteria principle by principle and computes how many people's evaluation is needed. #### The Established Set of Criteria Ten principles long used in the field are read here, matched against the catalog's decisions. - **Visibility of system status.** The user always knows what is happening. A loading state while search runs, a progress indicator during borrowing. - **Match between the system and the real world.** The interface speaks the user's vocabulary, not the institution's. "Circulation operations" is the institution's vocabulary; "borrowing" is the user's. - **User control and freedom.** A wrong step can be undone, a transaction can be abandoned partway through, an earlier state can be returned to. - **Consistency and standards.** The same thing has the same name and the same form everywhere. This is the structural-level counterpart of the rule set for visuals in the Repetition and Consistency lesson. - **Error prevention.** Better than a good error message is an error that never occurs. If a late fee is going to be charged, it is confirmed first. - **Recognition rather than recall.** The user does not have to memorize what they saw on a previous screen. Showing the shelf code again at the borrowing step instead of making the user hold it in memory. - **Flexibility and efficiency of use.** A shortcut for frequent users, an explicit path for new users; both exist in the same interface. - **Aesthetic and minimalist design.** No unnecessary information sits on the screen; every added element reduces the visibility of the information that matters. - **Help users recognize, diagnose, and recover from errors.** The message states what happened, why, and what to do about it. - **Help and documentation.** If help is needed, it is searchable, short, and task-focused; not needing help at all is better still. The names of the criteria are not used as templates. The output of an evaluation is not "the consistency principle was violated" but is written as "the borrow action is called *Take* in the list, *Borrow* in the record detail, and *On Hold* on the confirmation screen." The principle gives the finding's class; the finding does not stand in for the principle. #### Running the Session The evaluation follows three rules. **Independent.** Each evaluator walks through the interface alone and writes findings to their own list. If they walk through it together, the first person to speak steers the others' attention, and the number of independent findings drops. **Two-pass.** The first pass follows the flow as a whole and grasps the overall structure; the second pass scans each screen against the criteria list one by one. An evaluation done in a single pass judges a screen without knowing the flow's context. **Severity-scored.** Every finding gets a severity. The computation below uses a four-level scale: 1 a cosmetic flaw, 2 a minor obstacle, 3 a problem that noticeably hinders the task, 4 a problem that makes the task impossible to complete or produces irreversible damage. The score does the ranking after findings are merged. #### One Evaluator Is Not Enough Five evaluators reviewed the prototype independently, and the findings were merged. ```js // heuristic-evaluation.mjs — overlap and severity agreement of five evaluators' findings // Each finding: which principle it belongs to and which evaluator gave it what severity (1 minor, 4 major) const FINDINGS = [ { name: "loading state not shown", principle: "status visibility", score: { D1: 2, D2: 3, D3: 2, D4: 2, D5: 3 } }, { name: "same action, different names", principle: "consistency", score: { D1: 3, D2: 3, D4: 2 } }, { name: "borrowing cannot be undone", principle: "control and freedom", score: { D2: 4, D3: 3, D5: 4 } }, { name: "error message does not say why", principle: "error recovery", score: { D1: 3, D3: 4, D4: 3, D5: 3 } }, { name: "institutional jargon used", principle: "real-world match", score: { D2: 3, D5: 2 } }, { name: "search lost on return", principle: "control and freedom", score: { D1: 4, D2: 4, D3: 4, D4: 3, D5: 4 } }, { name: "fee not confirmed beforehand", principle: "error prevention", score: { D3: 4 } }, { name: "placeholder text used as label", principle: "recognition over recall", score: { D1: 3, D4: 3 } }, { name: "shelf code is not explained", principle: "real-world match", score: { D2: 3, D3: 3, D4: 4 } }, { name: "no shortcut to repeat a search", principle: "flexibility and efficiency", score: { D5: 2 } }, { name: "three actions carry equal weight", principle: "minimalist design", score: { D1: 3, D2: 2 } }, { name: "help is buried", principle: "help and documentation", score: { D4: 1 } }, ]; const EVALUATORS = ["D1", "D2", "D3", "D4", "D5"]; const median = (d) => { const s = [...d].sort((a, b) => a - b), o = s.length >> 1; return s.length % 2 ? s[o] : (s[o - 1] + s[o]) / 2; }; console.log("finding principle found median range"); for (const b of FINDINGS) { const p = Object.values(b.score); console.log( `${b.name.padEnd(34)} ${b.principle.padEnd(27)} ${String(p.length).padStart(5)} ${median(p).toFixed(1).padStart(8)} ${Math.min(...p)}-${Math.max(...p)}` ); } console.log("\nevaluator findings coverage"); for (const d of EVALUATORS) { const n = FINDINGS.filter((b) => d in b.score).length; console.log(`${d.padEnd(11)} ${String(n).padStart(9)} ${((n / FINDINGS.length) * 100).toFixed(1)}%`); } // Expected union over all subsets of k evaluators const subsets = (arr, k) => k === 0 ? [[]] : arr.flatMap((x, i) => subsets(arr.slice(i + 1), k - 1).map((r) => [x, ...r])); console.log("\nevaluators subsets expected found expected coverage"); for (let k = 1; k <= EVALUATORS.length; k++) { const groups = subsets(EVALUATORS, k); const total = groups.reduce( (t, group) => t + FINDINGS.filter((b) => group.some((d) => d in b.score)).length, 0); const avg = total / groups.length; console.log( `${String(k).padStart(10)} ${String(groups.length).padStart(7)} ${avg.toFixed(2).padStart(14)} ${((avg / FINDINGS.length) * 100).toFixed(1).padStart(17)}%` ); } // Findings caught by only one person, and severity agreement const soloFindings = FINDINGS.filter((b) => Object.keys(b.score).length === 1); console.log(`\nfindings caught by only one evaluator: ${soloFindings.length} / ${FINDINGS.length} (${((soloFindings.length / FINDINGS.length) * 100).toFixed(1)}%)`); console.log(` ${soloFindings.map((b) => b.name).join("; ")}`); const widelyFound = FINDINGS.filter((b) => Object.keys(b.score).length >= 3); const spread = widelyFound.map((b) => Math.max(...Object.values(b.score)) - Math.min(...Object.values(b.score))); console.log(`\nseverity spread on the ${widelyFound.length} findings caught by three or more people: average ${(spread.reduce((a, c) => a + c, 0) / spread.length).toFixed(2)} points, max ${Math.max(...spread)} points`); // Findings per principle const principles = new Map(); for (const b of FINDINGS) principles.set(b.principle, (principles.get(b.principle) ?? 0) + 1); console.log("\nprinciple findings"); for (const [i, c] of [...principles].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))) console.log(`${i.padEnd(27)} ${c}`); ``` ``` finding principle found median range loading state not shown status visibility 5 2.0 2-3 same action, different names consistency 3 3.0 2-3 borrowing cannot be undone control and freedom 3 4.0 3-4 error message does not say why error recovery 4 3.0 3-4 institutional jargon used real-world match 2 2.5 2-3 search lost on return control and freedom 5 4.0 3-4 fee not confirmed beforehand error prevention 1 4.0 4-4 placeholder text used as label recognition over recall 2 3.0 3-3 shelf code is not explained real-world match 3 3.0 3-4 no shortcut to repeat a search flexibility and efficiency 1 2.0 2-2 three actions carry equal weight minimalist design 2 2.5 2-3 help is buried help and documentation 1 1.0 1-1 evaluator findings coverage D1 6 50.0% D2 7 58.3% D3 6 50.0% D4 7 58.3% D5 6 50.0% evaluators subsets expected found expected coverage 1 5 6.40 53.3% 2 10 9.00 75.0% 3 10 10.50 87.5% 4 5 11.40 95.0% 5 1 12.00 100.0% findings caught by only one evaluator: 3 / 12 (25.0%) fee not confirmed beforehand; no shortcut to repeat a search; help is buried severity spread on the 6 findings caught by three or more people: average 1.00 points, max 1 points principle findings control and freedom 2 real-world match 2 consistency 1 error prevention 1 error recovery 1 flexibility and efficiency 1 help and documentation 1 minimalist design 1 recognition over recall 1 status visibility 1 ``` #### What the Numbers Say **A single evaluator finds about half.** Each of the five people found roughly half of the twelve findings; expected coverage over all single-person subsets is 53.3%. Having one person do the evaluation is agreeing in advance not to see about half the problems. **Gains grow with diminishing returns.** Expected coverage is 75% with two evaluators, 87.5% with three, 95% with four. Going from one to two gains 21.7 points, going from three to four gains 7.5 points. The evaluator count is chosen between three and four; a fifth person adds only 5 points in this example, and the same effort pays off more in a participant session. **Only one person found the most severe finding.** The "fee not confirmed beforehand" finding has a severity of 4 — the highest value on the scale — and only the third evaluator found it. A single evaluator chosen at random from the pool of five has a one-in-five chance of finding it. This is the real risk of working with a single evaluator: a missed finding does not have to be as unimportant as the average of missed findings. **Agreement is good on severity, poor on coverage.** For the six findings caught by three or more people, the severity spread averages one point; no one gave the same finding scores ranging from 1 to 4. When evaluators see a problem, they think similarly about its weight; where they diverge is whether they see the problem at all. So what gets discussed at the merge stage is not the scores but the finding missing from the list. **Two principles stand out.** Control and freedom, and real-world match, each have two findings; both point to the same place as the problems measured in user research — search lost on return and the unexplained shelf code. When a heuristic evaluation and field data converge on the same finding, evidence strength grows; the multi-source criterion set in the sixth lesson applies here too. #### The Limit of Heuristic Evaluation The method is cheap, and this cheapness is also the source of its limit. **Findings are hypotheses.** The evaluator sees the interface not as a user but as an expert walking through it with a criteria list. The "help is buried" finding was given a severity of 1; if real users never look for help, its severity is zero, and if they try to look for it and cannot find it, it is far higher. Only measurement tells us which, not the heuristic evaluation. **It can produce false findings.** Walking through with a criteria list breeds an attitude oriented toward hunting violations; every deviation looks like a flaw. A decision that deliberately departs from an established criterion — the differentiator decisions discussed in the fifth lesson — gets flagged as a violation in a heuristic evaluation. This is why findings are reviewed together with their design rationale. **It does not say what the user actually wanted to do.** A heuristic evaluation checks the interface's existing state against criteria. It cannot discover that the user actually wanted to do something else, that the information they were looking for is not in the interface at all, or that the task is entirely unnecessary. These belong to the domain of research and user testing. #### Summary - A usability heuristic is a criterion, not a rule; when it is not met, the result is a candidate to examine, not a definite flaw, and a finding is written as a concrete fact, not the principle's name. - The evaluation is run independently, in two passes, and with severity scoring; walking through together lowers the count of independent findings. - In the sample data, a single evaluator's expected coverage is 53.3%; it is 87.5% with three evaluators, 95% with four, and the gains diminish. - Only one of the five evaluators found the highest-severity finding; a missed finding cannot be assumed to be as unimportant as the average. - Evaluators agree on severity and diverge on coverage; what gets discussed at the merge is not the scores but what is missing from the list. - A heuristic evaluation produces hypotheses, can yield false findings, and cannot say what the user wanted to do. #### Next Step The heuristic evaluation produced twelve candidates, and each is a hypothesis. Which ones actually stop a real user cannot be known without measuring it. The next lesson puts the prototype in a participant's hands: it shows how the session is set up, how many participants are enough, and under which conditions the "five users are enough" claim is true and under which it is misleading, by computing the discovery curve. ### Usability Testing Source: https://academia.sh/en/courses/user-experience/usability-testing Finding problems with a small sample; computing the discovery curve, why the five-participant claim collapses on a non-uniform problem set, and the uncertainty of duration measurement in a small sample. The heuristic evaluation produced twelve candidates, and each is a hypothesis. Which ones actually stop a real user, and which go entirely unnoticed, cannot be known without measuring it. The way to measure this is to put the prototype in a participant's hands and give them a task. A **usability test** is observing a participant as they try to complete a predetermined task. Its purpose is not collecting opinions; what is measured is where the user pauses, where they click, and where they give up. This lesson takes on how the session is set up, how many participants it requires, and what a small sample can and cannot say, by computation. #### Setting Up the Session The session has three parts, and each part has its own rule. **Task scenario.** The task is given to the participant with a context, not an instruction. "Press the Borrow button" is an instruction, and it measures nothing. "A friend recommended a book to you; find out whether the library has it, and if so, how to get it" is a scenario; it requires the user to find their own way. The scenario does not use the interface's own words — if the word "narrowing" appears in the scenario, whether that word is understood cannot be measured. **Think-aloud.** The participant is asked to say out loud what they are doing and what they expect. This surfaces what observation alone cannot: we learn that the user saw an item but did not understand it, or was looking for something not on the screen, only from what they say. Think-aloud lengthens the duration; so if duration is to be measured, a separate task is run in silence. **What the moderator does not do.** The moderator does not point the way, does not give hints, does not say "over there." When the participant gets stuck, the moderator waits; when they truly stop, the task is ended, and this is a finding. The moderator also does not defend the design: the sentence "actually, we did put it there" silences the participant and devalues the rest of the session. #### The Test's Ethical Framework A usability test is a session where the participant's failure counts as data; this makes it, ethically, one of the forms of research that demands the most care. **What is being tested is the interface, not the participant.** This sentence is said explicitly at the start of the session, and it is not left as a formality: when a task cannot be completed, the participant is reminded that this is a finding about the interface. The participant feeling that they are being tested is both an ethical problem and something that corrupts the data. **Consent is detailed and reversible.** The participant gives consent knowing what is being recorded — screen, audio, click trail — and who will see the recording. They can stop the session at any moment and do not have to justify it. If they ask for their recording to be deleted at the end of the session, it is deleted. **Anonymization and the retention limit.** Findings are written with codes like `P01`. If the participant's account appears in the screen recording, a test account is used instead of a real membership; if that is not possible, identifying fields in the recording are scrubbed before analysis. Once the raw recording has been turned into a coded note, it is deleted within the written retention period. **Using a quote is a separate consent.** Putting the participant's words into a report is a different use from analyzing their recording. If a direct quote is going to be used, this is asked for separately. #### The Five-Participant Claim There is a well-known computation defending small samples. If a problem's probability of appearing in a single participant is `p`, its probability of showing up at least once across `n` participants is `1 - (1 - p)^n`. This curve saturates quickly and leads to the conclusion that five participants are enough to find most problems. The computation itself is correct; the assumption it carries is not. ```js // discovery-curve.mjs — the 1 - (1 - p)^n curve and non-uniform finding probabilities // Uniform assumption: every problem surfaces with the same probability for every participant console.log("share of problems found at uniform p"); console.log("participants p=0.10 p=0.20 p=0.31 p=0.50"); for (const n of [1, 2, 3, 5, 8, 10, 15, 20]) { const s = [0.1, 0.2, 0.31, 0.5].map((p) => ((1 - (1 - p) ** n) * 100).toFixed(1).padStart(6)); console.log(`${String(n).padStart(12)} ${s.join(" ")}`); } // The real problem set is not uniform: some problems show up for everyone, some rarely const PROBLEMS = [ { name: "search lost on return", p: 0.90, severity: 4 }, { name: "shelf code not understood", p: 0.75, severity: 3 }, { name: "same-titled records confused", p: 0.60, severity: 3 }, { name: "no loading state", p: 0.50, severity: 2 }, { name: "same action, different names", p: 0.40, severity: 3 }, { name: "placeholder instead of label", p: 0.30, severity: 3 }, { name: "three actions carry equal weight", p: 0.25, severity: 2 }, { name: "error message has no reason", p: 0.20, severity: 3 }, { name: "renewal path hidden", p: 0.15, severity: 3 }, { name: "help is buried", p: 0.10, severity: 1 }, { name: "no shortcut", p: 0.08, severity: 2 }, { name: "fee not confirmed beforehand", p: 0.05, severity: 4 }, ]; const avgP = PROBLEMS.reduce((t, s) => t + s.p, 0) / PROBLEMS.length; console.log(`\naverage p of the problem set: ${avgP.toFixed(4)}`); console.log("\nparticipants at uniform p actual set severity-weighted lowest problem's probability"); const severityTotal = PROBLEMS.reduce((t, s) => t + s.severity, 0); for (const n of [1, 3, 5, 8, 12, 20, 32]) { const uniform = (1 - (1 - avgP) ** n) * 100; const actual = (PROBLEMS.reduce((t, s) => t + (1 - (1 - s.p) ** n), 0) / PROBLEMS.length) * 100; const weighted = (PROBLEMS.reduce((t, s) => t + s.severity * (1 - (1 - s.p) ** n), 0) / severityTotal) * 100; const lowest = Math.min(...PROBLEMS.map((s) => 1 - (1 - s.p) ** n)) * 100; console.log( `${String(n).padStart(12)} ${uniform.toFixed(1).padStart(11)}% ${actual.toFixed(1).padStart(10)}% ${weighted.toFixed(1).padStart(17)}% ${lowest.toFixed(1).padStart(37)}%` ); } // Each problem's probability of being found with 5 participants console.log("\nproblem p severity found with 5 participants"); for (const s of PROBLEMS) { const b = (1 - (1 - s.p) ** 5) * 100; console.log(`${s.name.padEnd(33)} ${s.p.toFixed(2)} ${String(s.severity).padStart(8)} ${b.toFixed(1).padStart(26)}%`); } // How many participants are needed to find each problem with at least 80% probability const rarest = PROBLEMS.reduce((a, b) => (a.p < b.p ? a : b)); const needed = Math.ceil(Math.log(0.2) / Math.log(1 - rarest.p)); console.log(`\nrarest problem "${rarest.name}" (p = ${rarest.p}): ${needed} participants needed for an 80% chance of finding it`); ``` ``` share of problems found at uniform p participants p=0.10 p=0.20 p=0.31 p=0.50 1 10.0 20.0 31.0 50.0 2 19.0 36.0 52.4 75.0 3 27.1 48.8 67.1 87.5 5 41.0 67.2 84.4 96.9 8 57.0 83.2 94.9 99.6 10 65.1 89.3 97.6 99.9 15 79.4 96.5 99.6 100.0 20 87.8 98.8 99.9 100.0 average p of the problem set: 0.3567 participants at uniform p actual set severity-weighted lowest problem's probability 1 35.7% 35.7% 38.7% 5.0% 3 73.4% 61.0% 63.3% 14.3% 5 89.0% 72.3% 73.9% 22.6% 8 97.1% 81.4% 82.2% 33.7% 12 99.5% 87.9% 88.1% 46.0% 20 100.0% 94.0% 93.7% 64.2% 32 100.0% 97.5% 97.1% 80.6% problem p severity found with 5 participants search lost on return 0.90 4 100.0% shelf code not understood 0.75 3 99.9% same-titled records confused 0.60 3 99.0% no loading state 0.50 2 96.9% same action, different names 0.40 3 92.2% placeholder instead of label 0.30 3 83.2% three actions carry equal weight 0.25 2 76.3% error message has no reason 0.20 3 67.2% renewal path hidden 0.15 3 55.6% help is buried 0.10 1 41.0% no shortcut 0.08 2 34.1% fee not confirmed beforehand 0.05 4 22.6% rarest problem "fee not confirmed beforehand" (p = 0.05): 32 participants needed for an 80% chance of finding it ``` #### Reading the Curve **The uniform assumption is what keeps the claim standing.** If every problem is assumed to have the same `p` value, five participants find 84.4% of the problems at `p = 0.31`. This computation is done correctly, and the result is internally consistent. **The real problem set is not uniform.** The sample set's average `p` value is 0.3567; if treated as uniform, expected coverage at five participants is 89.0%. In the real set, which carries the same average but a wide spread, expected coverage is 72.3%. The 16.7-point gap comes entirely from the spread: high-`p` problems already show up with the first participant and pull the curve upward, while low-`p` problems approach very slowly as participants are added. Computing from the average overcounts the easy-to-find ones. **The problem that gets missed is not chosen at random.** The last table ranks problems by their probability of being found, and at the very bottom sits a severity-4 problem: the fee not being confirmed beforehand. Its probability of being found with five participants is 22.6%. This was also the problem only one person found in the heuristic evaluation in the previous lesson. Both methods miss the same problem for the same reason: **rare, high-consequence problems cannot be reliably found by any frequency-based method.** These are searched for with structural methods, such as flow-diagram auditing and criteria review. **Weighting by severity does not save the table.** Weighted coverage at five participants is 73.9%, very close to unweighted coverage. Because some severe problems are common and some are rare, the weighting balances out. In an interface where severe problems happen to be especially rare, weighted coverage would fall below unweighted. **Finding every problem is expensive.** Finding the rarest problem with 80% probability requires thirty-two participants. This number cannot be met in most conditions, and it does not need to be. The correct conclusion is this: small-sample testing is a method that **finds common problems cheaply** and is good for fixing what it finds; it does not prove the absence of what it does not find. The sentence "we tested with five users and no problems came up" is not evidence. In practice, this translates into running the test as **successive small rounds** rather than one large session: five participants, fixes, five new participants. Because each round clears out the most common problems, rarer problems become visible in the next round. #### The Limit of Measuring Duration in a Small Sample The test's second output is duration measurement, and here the limit of a small sample is harsher. ```js // duration-measurement.mjs — median duration and its uncertainty in a small sample // Five participants' screen time on the "find the known record and borrow it" task (seconds) const DURATIONS = [42, 55, 61, 78, 154]; const sorted = [...DURATIONS].sort((a, b) => a - b); const mean = DURATIONS.reduce((a, b) => a + b, 0) / DURATIONS.length; const median = sorted[(sorted.length - 1) / 2]; const geometric = Math.exp(DURATIONS.reduce((t, x) => t + Math.log(x), 0) / DURATIONS.length); console.log(`measured durations : ${sorted.join(", ")} s`); console.log(`arithmetic mean : ${mean.toFixed(1)} s`); console.log(`geometric mean : ${geometric.toFixed(1)} s`); console.log(`median : ${median.toFixed(1)} s`); console.log(`mean / median : ${(mean / median).toFixed(2)}`); const largest = Math.max(...DURATIONS); console.log(`mean with the largest value (${largest} s) removed: ${(DURATIONS.filter((x) => x !== largest).reduce((a, b) => a + b, 0) / 4).toFixed(1)} s`); // Distribution-free confidence interval for the median, from order statistics const binom = (n, k) => { let r = 1; for (let i = 0; i < k; i++) r = (r * (n - i)) / (i + 1); return r; }; const coverage = (n, i, j) => { let t = 0; for (let k = i; k < j; k++) t += binom(n, k); return t / 2 ** n; }; console.log("\nfor n = 5, probability that an order-statistic interval covers the median"); for (const [i, j] of [[1, 5], [2, 4]]) { console.log( ` [order ${i}, order ${j}] = [${sorted[i - 1]}, ${sorted[j - 1]}] s -> ${(coverage(5, i, j) * 100).toFixed(2)}%` ); } // The narrowest order-statistic interval giving at least 90% coverage const narrowestInterval = (n) => { for (let g = 1; g < n; g++) for (let i = 1; i + g <= n; i++) if (coverage(n, i, i + g) >= 0.9) return [i, i + g, g, coverage(n, i, i + g)]; return null; }; console.log("\nparticipants narrowest interval at 90% coverage orders spanned share of sample"); for (const n of [5, 8, 12, 20, 30, 50]) { const a = narrowestInterval(n); console.log( `${String(n).padStart(12)} ${`[order ${a[0]}, order ${a[1]}]`.padStart(33)} ${String(a[2] + 1).padStart(15)} ${(((a[2] + 1) / n) * 100).toFixed(1).padStart(16)}%` ); } ``` ``` measured durations : 42, 55, 61, 78, 154 s arithmetic mean : 78.0 s geometric mean : 70.1 s median : 61.0 s mean / median : 1.28 mean with the largest value (154 s) removed: 59.0 s for n = 5, probability that an order-statistic interval covers the median [order 1, order 5] = [42, 154] s -> 93.75% [order 2, order 4] = [55, 78] s -> 62.50% participants narrowest interval at 90% coverage orders spanned share of sample 5 [order 1, order 5] 5 100.0% 8 [order 2, order 7] 6 75.0% 12 [order 3, order 9] 7 58.3% 20 [order 6, order 14] 9 45.0% 30 [order 11, order 20] 10 33.3% 50 [order 19, order 31] 13 26.0% ``` Task durations are right-skewed: the lower bound is the time it physically takes to do the task, and there is no upper bound. When one participant gets stuck and spends 154 seconds, the arithmetic mean rises to 78 seconds; this value is larger than four of the five participants and represents none of them. Removing that single value brings the mean down to 59 seconds. The median is 61 seconds and is not affected by a single outlier. The median is used as the duration metric. The second table gives the real warning. With five participants, the interval that covers the true median with 93.75% probability is the **entire sample**: between 42 and 154 seconds. A narrower interval — 55 to 78 seconds — covers only 62.5%. The 61-second median that comes out of five participants is not a measurement but the midpoint of a very wide interval. The last table shows how this narrows. At thirty participants, the interval covering 90% spans the middle ten of the sorted values — a third of the sample; at fifty participants, a quarter. If a duration comparison is going to be made, the sample it requires is an order of magnitude larger than what is needed to find problems. The practical consequence of this is a hard distinction: **small-sample testing finds problems, it does not produce numbers.** A five-participant session does not yield the sentence "task duration is 61 seconds"; it yields "four of the five participants finished in under a minute, one got stuck searching for the shelf code." The second sentence is both true and produces a decision. #### Summary - A usability test does not collect opinions; a task scenario is given, think-aloud is requested, and the moderator does not point the way or defend the design. - What is tested is the interface; consent is detailed and reversible, findings are written with codes, the raw recording is deleted within the retention limit, and using a quote is asked for separately. - The discovery curve assumes a uniform finding probability; in the sample set, the uniform computation with the same average gave 89.0% at five participants, the real distribution gave 72.3%. - Missed problems are not random: rare and severe problems cannot be found by frequency-based methods and are searched for with structural audits; one of the sample set's most severe problems shows up with only 22.6% probability at five participants. - Small-sample testing finds common problems cheaply and is run in successive rounds; no findings turning up does not prove there is no problem. - The duration metric is the median; at five participants, the interval that covers the true median with 93.75% probability is the entire sample, so small-sample testing finds problems, it does not produce numbers. #### Next Step The test shows where the user gets stuck. The participant who got stuck searching for the shelf code is seen looking at that information on the screen but not using it; the participant who picked the wrong one of two same-titled volumes is seen not reading the full title. Observation records both but explains neither. Why did the user not use the information they looked at, why did they choose without reading? These are not carelessness; they are questions about how the decision was made. The next topic's first lesson, Dual-Process Thinking, distinguishes whether decisions are made through a fast, intuitive process or a slow, deliberate one, and asks which the interface speaks to. ## Behavior Design Intuitive and deliberative decision-making, cognitive load, the motivation–ability–trigger framework, the habit loop, the effect of defaults, and recognizing dark patterns. ### Dual-Process Thinking Source: https://academia.sh/en/courses/user-experience/dual-process-thinking Whether a decision is made by the fast intuitive system or the slow deliberative system, how salience in a result list determines the choice, and computing the time cost of switching to the deliberative system. A usability test shows **where** the user got stuck. The participant paused on the result list, opened the third record, went back, opened the second record, and borrowed it — you kept a log, wrote down the durations, pinned the problem to a single step. What the test does not say is **why** that pause happened. The same observation can arise from two different causes: the user may not have found the information they were looking for in the list, or they may have found it but trusted the record the visual layout pointed to, then noticed the mistake afterward. The two causes call for different design decisions. This topic addresses the decision mechanisms beneath observed behavior. The first question is this: when a user selects a record in the catalog, how much do they read, and how much do they merely see? #### Two Decision Modes A human decision does not come from a single mechanism. The **dual-process thinking** model holds that a decision can be made by two separate operating modes. The **intuitive system** is fast, works by pattern matching, requires no effort, and is almost always on. When you look at a list, you know which item "stands out" without reading it; this knowledge comes from size, weight, position, and contrast — the quantities measured under the name **visual weight** in the Fundamentals of Interface Design course. The intuitive system reads this weight as a signal of relevance. The **deliberative system** is slow, sequential, requires attention, and tires. Reading a record's title, comparing the year published against the edition you are looking for, inferring which of two records is the volume you want — these are this system's work. The two systems do not compete; one hands off to the other. The intuitive system constantly produces an answer; the deliberative system engages only when it detects a contradiction, an ambiguity, or a cost signal. What the design determines is **which one makes the decision.** #### Which System Decides in the Result List In the catalog interface, a search returns eight records. What the user is looking for is the record that best matches the query. The interface, however, sorts and emphasizes records by some criterion. The two criteria do not have to be the same. The simulation below compares three interface options: one that emphasizes records by year published, one that emphasizes records that have a cover image, and one that emphasizes records by match to the query. The user can behave in two modes: choosing the most salient record (intuitive) or reading and comparing all eight records (deliberative). ```js // salience.mjs — how salience in a result list determines the choice // Deterministic pseudo-random generator (linear congruential generator). function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } const LIST_LENGTH = 8; const TRIALS = 500; // Produce a search result list: each record's relevance (0-100) and year. function list(rnd) { const records = []; for (let i = 0; i < LIST_LENGTH; i++) { records.push({ relevance: Math.round(rnd() * 100), // actual match to the query year: 1980 + Math.round(rnd() * 45), // year published cover: rnd() < 0.35, // has a cover image }); } return records; } // Three interfaces: each assigns a record a visual weight. const INTERFACES = { "by year": (r) => r.year - 1980, // newest is on top and biggest "by cover image": (r) => (r.cover ? 80 : 20) + (r.year - 1980) / 10, "by relevance": (r) => r.relevance, // visual weight = match }; const READING_TIME = 1.4; // seconds / record — reading and evaluating one record const SCAN_TIME = 0.6; // seconds — scanning the list to pick the most salient const largest = (records, metric) => records.reduce((a, b) => (metric(b) > metric(a) ? b : a)); console.log("interface intuitive correct intuitive time deliberative correct deliberative time"); for (const [name, weight] of Object.entries(INTERFACES)) { const rnd = generator(20240701); let intuitiveCorrect = 0; for (let d = 0; d < TRIALS; d++) { const records = list(rnd); const mostRelevant = largest(records, (r) => r.relevance); const mostSalient = largest(records, weight); if (mostSalient.relevance === mostRelevant.relevance) intuitiveCorrect++; } console.log( `${name.padEnd(22)} ${((intuitiveCorrect / TRIALS * 100).toFixed(1) + " %").padStart(14)} ${(SCAN_TIME.toFixed(1) + " s").padStart(14)}` + ` ${("100.0 %").padStart(14)} ${((SCAN_TIME + LIST_LENGTH * READING_TIME).toFixed(1) + " s").padStart(13)}` ); } // Mixed behavior: the user trusts the salient record, but switches to reading // when the first two records are close in visual weight. console.log("\nmixed behavior (reads when the top two records are within 15% in visual weight)"); console.log("interface correct rate average time switch to reading"); for (const [name, weight] of Object.entries(INTERFACES)) { const rnd = generator(20240701); let correct = 0, time = 0, reading = 0; for (let d = 0; d < TRIALS; d++) { const records = list(rnd); const mostRelevant = largest(records, (r) => r.relevance); const sorted = [...records].sort((a, b) => weight(b) - weight(a)); const diff = weight(sorted[0]) === 0 ? 1 : (weight(sorted[0]) - weight(sorted[1])) / Math.abs(weight(sorted[0])); if (diff < 0.15) { reading++; time += SCAN_TIME + LIST_LENGTH * READING_TIME; correct++; } else { time += SCAN_TIME; if (sorted[0].relevance === mostRelevant.relevance) correct++; } } console.log( `${name.padEnd(22)} ${((correct / TRIALS * 100).toFixed(1) + " %").padStart(10)} ${((time / TRIALS).toFixed(2) + " s").padStart(14)}` + ` ${((reading / TRIALS * 100).toFixed(1) + " %").padStart(14)}` ); } // Cost of the wrong choice: relevance gap between the chosen record and the best one console.log("\ninterface average relevance loss (intuitive choice)"); for (const [name, weight] of Object.entries(INTERFACES)) { const rnd = generator(20240701); let loss = 0; for (let d = 0; d < TRIALS; d++) { const records = list(rnd); loss += largest(records, (r) => r.relevance).relevance - largest(records, weight).relevance; } console.log(`${name.padEnd(22)} ${(loss / TRIALS).toFixed(1).padStart(10)} points`); } ``` ``` interface intuitive correct intuitive time deliberative correct deliberative time by year 13.6 % 0.6 s 100.0 % 11.8 s by cover image 12.6 % 0.6 s 100.0 % 11.8 s by relevance 100.0 % 0.6 s 100.0 % 11.8 s mixed behavior (reads when the top two records are within 15% in visual weight) interface correct rate average time switch to reading by year 70.6 % 7.95 s 65.6 % by cover image 87.8 % 10.16 s 85.4 % by relevance 100.0 % 7.97 s 65.8 % interface average relevance loss (intuitive choice) by year 39.5 points by cover image 37.8 points by relevance 0.0 points ``` #### The Intuitive System Does Not Err, It Is Fed Wrong The first column of the first table is the core of this lesson. The intuitive mode runs on the same rule — "choose the most salient one" — but its accuracy ranges from 13 percent to 100 percent. The only thing that changes is what the interface makes salient. This does not mean the intuitive system is flawed. The rule is a sound rule: in the physical world, what catches the eye is most often what is being looked for. The rule's output is only as good as its input. The designer chooses the input. An interface that emphasizes records by year published feeds the user's intuitive system the information that "the newest one is the best match"; the user did not choose this information, the interface supplied it. The third table gives the size of the error. In the list emphasized by year published, the intuitive choice results in a record that is, on average, 39.5 relevance points worse. On a hundred-point scale, this means picking a record from the middle of the ranking instead of the list's best match. The user borrows the wrong book and only realizes it once the book is in hand. The first design rule this yields: **visual weight is tied to the same quantity as the user's decision criterion.** When the sorting criterion and the emphasis criterion diverge, the interface is giving the user wrong information — not in writing, but through form. #### The Cost of Switching to the Deliberative System Is Time The second table measures something more interesting. Here the user is not rigid: when the two most salient records are close to each other — when the emphasis does not give a clear answer — the user switches to reading. In the interface emphasized by year published, the user switches to reading in 65.6 percent of trials, the average time rises from 0.6 seconds to 7.95 seconds, and accuracy stays at 70.6 percent. In the interface emphasized by cover image, the switch to reading rises to 85.4 percent, the time rises to 10.16 seconds, and accuracy reaches 87.8 percent. Reading more produces a more accurate result but lengthens the time by twenty-eight percent. In the interface emphasized by relevance, the switch-to-reading rate is 65.8 percent, the time is 7.97 seconds — nearly identical to the interface emphasized by year published. But accuracy is 100 percent. Same time, same amount of reading, different result. The difference lies in **what the reading is for**: in one case the user reads to correct the interface's error, in the other to confirm the answer the interface already gave. This distinction separates the two causes of the pause observed in a usability test. If the user reads and ends up taking the record the interface recommended, the reading was a confirmation; it carries a time cost but the outcome is correct. If the user reads and takes a record different from the interface's recommendation, the interface misled them and the reading was a repair. In the test log, the two look the same: "the participant paused on the list." #### The Limits of This Model The numbers above come from a simulation, not a measurement. A reading time of 1.4 seconds, a scan time of 0.6 seconds, and a 15 percent proximity threshold are assumptions; other values give other numbers. What the model carries is not the numbers but their **direction**: when the emphasis criterion diverges from the decision criterion, the intuitive choice breaks down and the correction is paid for in time. This direction does not change across a reasonable range of assumptions. There is one more thing the model does not measure. Here, once the user switches to reading, they always make the correct decision. A real user tires; the deliberative system is not unlimited. Comparing eight records does not happen as eight separate decisions but out of an attention budget that runs out far faster. How that budget is spent is the subject of the next lesson. #### Where the Designer's Responsibility Begins Because the designer chooses the intuitive system's input, the accuracy of that input is the designer's responsibility. This is the first form of a criterion that spans this entire topic: **Would the user make the same decision if they knew what the interface was telling them?** In the list emphasized by relevance, the answer is yes: knowing "this record is on top because it best matches your query" would not change the user's choice. In the list emphasized by year published, the answer is no: had the user known "this record is on top because it is the newest edition," they would have switched to reading the list. The second interface draws the power of its decision from the user's ignorance. At this point, this criterion looks like a design-quality measure; by the end of the topic it will turn into an ethical one. The only difference is whether the divergence was built by accident or on purpose. #### Summary - Dual-process thinking separates whether a decision is made by the fast, effortless intuitive system or the slow, attention-demanding deliberative system; the interface determines which one decides. - The intuitive system applies the rule "choose the most salient one"; the rule's accuracy depends on what the interface makes salient — in the simulation, the same rule produced results ranging from 13 percent to 100 percent. - When the emphasis criterion diverges from the decision criterion, the cost of the intuitive choice is measurable: an average loss of 39.5 relevance points in an eight-record list. - Switching to the deliberative system reduces error but increases time by roughly a factor of twenty; the user's reading is either a confirmation or a repair of the interface's error. - The "pause" observation in a usability test does not distinguish between these two cases; the distinction is made by looking at which record was chosen. - Because the designer chooses the intuitive system's input, the question of whether the user would make the same decision knowing that information is the designer's question. #### Next Step This lesson showed that the deliberative system has a cost but assumed the cost was fixed: 1.4 seconds per record. In reality the cost grows with the number of options, and the growth is not linear. The next lesson measures that growth: how the number of options in a list determines decision time, how independent decision points on a screen are counted, and why grouping options affects decision time in the opposite direction from what is expected. ### Cognitive Load Source: https://academia.sh/en/courses/user-experience/cognitive-load Computing the effect of the number of options on decision time with the Hick–Hyman relation, grouping's counterintuitive result, counting independent decision points on a screen, and the effect of decision fatigue on error rate. The previous lesson showed that switching to the deliberative system has a cost but assumed the cost was fixed: 1.4 seconds per record. That assumption holds only as long as the number of records in the list does not change. In the catalog interface, though, the number of options takes a different value on every screen: twenty-four criteria in the filter panel, five in the sort dropdown, six in the delivery branch list. This lesson establishes the relation between the number of options and decision time, counts the independent decision points in the borrowing flow, and computes how that load converts into an error rate. One of the results will show that a familiar piece of design advice does not hold up against measurement. #### Decision Time Is the Logarithm of the Number of Options The time it takes to choose among $n$ equiprobable options is proportional not to the number of options but to the **amount of information** the options carry. The **Hick–Hyman relation** writes this as: $$T = a + b \cdot \log_2(n + 1)$$ Here $a$ is the decision-independent fixed delay (perception and action), and $b$ is the decision time per bit. The $n + 1$ term accounts for the "choose none" option. The intuition the relation carries is this: choosing works like a binary search. The user does not review options one by one; they successively halve the set. Doubling the number of options adds a single bit to the time. **Cognitive load** is the sum of these bits: the amount of information the user must process to complete a task. Because load is a measurable quantity, the design decision becomes measurable as well. #### Counting the Borrowing Flow's Load The computation below does three things: it derives the options-versus-time curve, compares a flat list against a grouped list, and counts the independent decision points in the catalog interface's borrowing flow to give the total load. ```js // load.mjs — Hick-Hyman decision time, grouping, and on-screen decision points const A = 0.20; // s — decision-independent fixed delay (perception + action) const B = 0.15; // s/bit — decision time per bit const log2 = (x) => Math.log(x) / Math.LN2; const equalOddsTime = (n) => A + B * log2(n + 1); // Hick's law const entropy = (p) => -p.reduce((t, x) => t + (x > 0 ? x * log2(x) : 0), 0); const entropyTime = (p) => A + B * entropy(p); // Hyman generalization // 1) Effect of the number of options in the filter panel on decision time console.log("options info (bits) decision time"); for (const n of [2, 4, 8, 12, 16, 24, 32]) { console.log(`${String(n).padStart(7)} ${log2(n + 1).toFixed(3).padStart(12)} ${(equalOddsTime(n).toFixed(3) + " s").padStart(13)}`); } // 2) Comparing a flat list against two-stage grouping (24 filter options) console.log("\nlayout comparison for 24 filter options"); const flat = equalOddsTime(24); const grouped = equalOddsTime(4) + equalOddsTime(6); // first the group, then within the group const grouped3 = equalOddsTime(3) + equalOddsTime(8); const grouped2 = equalOddsTime(2) + equalOddsTime(12); console.log(`flat list (24) ${flat.toFixed(3)} s`); console.log(`4 groups x 6 options ${grouped.toFixed(3)} s (diff ${(grouped - flat).toFixed(3)} s)`); console.log(`3 groups x 8 options ${grouped3.toFixed(3)} s (diff ${(grouped3 - flat).toFixed(3)} s)`); console.log(`2 groups x 12 options ${grouped2.toFixed(3)} s (diff ${(grouped2 - flat).toFixed(3)} s)`); // 3) When usage distribution is not equiprobable: Hyman entropy console.log("\nwhen usage distribution is not equiprobable (24 options)"); const equalProb = Array(24).fill(1 / 24); // Observed usage: three filters heavily weighted, the remaining 21 rare const observed = []; const heavy = [0.34, 0.26, 0.17]; for (const p of heavy) observed.push(p); for (let i = 0; i < 21; i++) observed.push((1 - 0.77) / 21); console.log(`equiprobable distribution entropy ${entropy(equalProb).toFixed(3)} bits time ${entropyTime(equalProb).toFixed(3)} s`); console.log(`observed distribution entropy ${entropy(observed).toFixed(3)} bits time ${entropyTime(observed).toFixed(3)} s`); console.log(`gain from ordering by frequency: ${(entropyTime(equalProb) - entropyTime(observed)).toFixed(3)} s`); console.log(`gain from grouping (under the equiprobable assumption): ${(flat - grouped).toFixed(3)} s`); // 4) Counting independent decision points in the borrowing flow // Each decision point: an area where the user must make a choice, independent of the others. const FLOW = [ { screen: "search", points: [["search field", 1], ["scope selection", 3], ["sorting", 5]] }, { screen: "result list", points: [["filter panel", 24], ["sorting", 5], ["page size", 4], ["record selection", 20]] }, { screen: "record detail", points: [["copy selection", 3], ["borrow / reserve / add to list", 3]] }, { screen: "borrow confirmation", points: [["delivery branch", 6], ["duration selection", 3], ["reminder preference", 4], ["confirmation", 2]] }, ]; console.log("\nscreen decision points total bits decision time"); let totalBits = 0, totalTime = 0, totalPoints = 0; for (const { screen, points } of FLOW) { const bits = points.reduce((t, [, n]) => t + log2(n + 1), 0); const time = points.reduce((t, [, n]) => t + equalOddsTime(n), 0); totalBits += bits; totalTime += time; totalPoints += points.length; console.log(`${screen.padEnd(22)} ${String(points.length).padStart(15)} ${bits.toFixed(2).padStart(11)} ${(time.toFixed(2) + " s").padStart(13)}`); } console.log(`${"TOTAL".padEnd(22)} ${String(totalPoints).padStart(15)} ${totalBits.toFixed(2).padStart(11)} ${(totalTime.toFixed(2) + " s").padStart(13)}`); // 5) Decision fatigue: error probability rises as bits spent accumulate // p_error(x) = p0 + k * (x / budget)^2 ; x = bits spent so far const P0 = 0.02, K = 0.16, BUDGET = 32; function flowError(flow) { let spent = 0, expectedError = 0; for (const { points } of flow) { for (const [, n] of points) { const bits = log2(n + 1); const p = Math.min(1, P0 + K * Math.pow(spent / BUDGET, 2)); expectedError += p; spent += bits; } } return { spent, expectedError }; } const before = flowError(FLOW); console.log(`\nbits spent ${before.spent.toFixed(2)}, expected number of wrong choices ${before.expectedError.toFixed(3)}`); // 6) Reduced flow: fields with defaults stop being decision points const REDUCED = [ { screen: "search", points: [["search field", 1]] }, { screen: "result list", points: [["filter panel (4 groups)", 4], ["in-group selection", 6], ["record selection", 20]] }, { screen: "record detail", points: [["borrow / reserve / add to list", 3]] }, { screen: "borrow confirmation", points: [["confirmation", 2]] }, ]; const after = flowError(REDUCED); const reducedTime = REDUCED.reduce((t, e) => t + e.points.reduce((s, [, n]) => s + equalOddsTime(n), 0), 0); const reducedPoints = REDUCED.reduce((t, e) => t + e.points.length, 0); console.log(`reduced flow: ${reducedPoints} decision points, ${after.spent.toFixed(2)} bits, ${reducedTime.toFixed(2)} s, expected wrong choices ${after.expectedError.toFixed(3)}`); console.log(`gain: ${(totalPoints - reducedPoints)} decision points, ${(totalTime - reducedTime).toFixed(2)} s, ${((1 - after.expectedError / before.expectedError) * 100).toFixed(1)} % fewer expected errors`); ``` ``` options info (bits) decision time 2 1.585 0.438 s 4 2.322 0.548 s 8 3.170 0.675 s 12 3.700 0.755 s 16 4.087 0.813 s 24 4.644 0.897 s 32 5.044 0.957 s layout comparison for 24 filter options flat list (24) 0.897 s 4 groups x 6 options 1.169 s (diff 0.273 s) 3 groups x 8 options 1.175 s (diff 0.279 s) 2 groups x 12 options 1.193 s (diff 0.296 s) when usage distribution is not equiprobable (24 options) equiprobable distribution entropy 4.585 bits time 0.888 s observed distribution entropy 2.967 bits time 0.645 s gain from ordering by frequency: 0.243 s gain from grouping (under the equiprobable assumption): -0.273 s screen decision points total bits decision time search 3 5.58 1.44 s result list 4 13.94 2.89 s record detail 2 4.00 1.00 s borrow confirmation 4 8.71 2.11 s TOTAL 13 32.24 7.44 s bits spent 32.24, expected number of wrong choices 0.943 reduced flow: 6 decision points, 14.11 bits, 3.32 s, expected wrong choices 0.170 gain: 7 decision points, 4.12 s, 82.0 % fewer expected errors ``` #### Adding an Option Is Cheap, Adding a Decision Point Is Expensive The first table shows what logarithmic growth means in practice. As the number of options rises from two to thirty-two — a sixteenfold increase — decision time rises from 0.438 seconds to 0.957 seconds, only a 2.2-fold increase. The cost of going from eight options to sixteen is 0.138 seconds. This means adding an option to a single dropdown is cheap. What is expensive is something else, and it shows up in the fourth table: **adding a separate decision point.** Every decision point charges the $a$ constant again. The two decision points on the third screen cost 1.00 second total; 0.40 seconds of that, 40 percent, comes not from the number of options but from the number of decision points. The rule is this: **a single field with twelve options is cheaper than four fields with three options each.** The computation confirms it — a single field with twelve options costs 0.755 seconds; four fields with three options each cost $4 \times 0.500 = 2.000$ seconds. #### Grouping's Effect Is the Opposite of Expected The second table contradicts a piece of design advice given often. A flat list of twenty-four options costs 0.897 seconds; the same twenty-four options split into four groups cost 1.169 seconds. Grouping **increases** decision time **by 0.273 seconds**, it does not reduce it. The reason is the rule from the previous paragraph: grouping turns a single decision point into two, and the $a$ constant is paid twice. This result does not mean grouping is wrong; it means grouping **is not done for decision time.** Grouping reduces a different cost: the cost of **finding** the option the user is looking for in the list. In an unlabeled list of twenty-four items, the user has to read the items; in a list with four headings, they read under only one heading. The Hick–Hyman relation assumes the option is already known — it does not account for the search cost. The two costs are measured separately and justified separately. The third table shows a bigger gain. When filter usage is not equiprobable — when 77 percent of users use only three criteria — entropy drops from 4.585 bits to 2.967 bits, and time drops from 0.888 seconds to 0.645 seconds. The gain from putting frequently used filters first is 0.243 seconds; grouping's "gain" is minus 0.273 seconds. The most efficient way to reduce load is not rearranging the options but **reflecting the usage distribution in the interface.** This has one precondition: knowing the usage distribution. That distribution is obtained by measurement, not by guessing; it is the subject of the measurement topic. #### Fatigue Is the Load's Second Bill The fifth computation models a second cost of load beyond time. The user's attention is limited, and error probability rises as it is spent. In the model, error probability grows with the square of the number of bits spent so far. The entire borrowing flow costs thirteen decision points and 32.24 bits. At this load, the expected number of wrong choices is 0.943 — every user who completes the flow gets, on average, one decision wrong. A wrong decision tends to be a decision near the end of the flow; the most tired moments come on the last screen. In the catalog interface, that screen is the borrow confirmation: the delivery branch, the duration, and the reminder preference are chosen there. A book ordered to the wrong branch is the user's most expensive mistake. The sixth computation removes seven decision points. The removed fields do not disappear; they take on a **default** value. The delivery branch defaults to the branch the user belongs to, the duration to the longest duration, the reminder preference to on. The user can change them if they want, but as long as they do not, they are not forced to decide. The result: 3.32 seconds instead of 7.44, and 82 percent fewer expected errors. Two separate conclusions follow from this, and they must not be conflated. The first is a design gain: a decision not asked of the user is better than a decision left unmade. The second is a warning: setting a default takes the decision away from the user and gives it to the designer. The three defaults above serve the user's interest — the user would have chosen the same thing had they known. The same mechanism can be built against the user's interest. The power of defaults, and the limit of that power, is the subject of a separate lesson. #### The Practical Way to Measure Load Measuring a screen's load does not require a design tool; counting is enough. **Count the independent decision points.** Every field where the user must assign a value is a point. If the value is pre-filled and the user can move on without touching it, the point does not count. **Write down each point's number of options and sum $\log_2(n+1)$.** The total is the screen's load in bits. Comparisons between screens are made with this number. **Track the total bits across the same flow.** Fatigue accumulates over the flow; screens that look reasonable one at a time may not be reasonable once summed. **Order options by the usage distribution.** Frequency ordering yields a bigger gain than grouping and adds no decision point. #### Summary - The Hick–Hyman relation ties decision time to the logarithm of the number of options: $T = a + b \log_2(n+1)$; doubling the number of options adds one bit to the time. - Adding an option is cheap, adding a decision point is expensive; a single field with twelve options is chosen roughly 2.6 times faster than four fields with three options each. - Grouping does not reduce decision time — in the computation it adds 0.273 seconds; grouping is justified not by decision time but by the cost of finding an option in the list. - When the usage distribution is not equiprobable, entropy drops; ordering filters by frequency gains 0.243 seconds and adds no decision point. - Load accumulates over a flow and raises error probability; when a thirteen-point borrowing flow is reduced to six points, expected wrong choices drop by 82 percent. - Setting a default takes the decision away from the user and gives it to the designer; both the gain and the risk come from that. #### Next Step This lesson measured **how hard** the user finds a decision. What it did not measure is not making the decision at all. The catalog user may never complete the borrowing flow; they may not even start it, even when the load is low. The next lesson establishes which three conditions must hold at the same time for a behavior to occur, and computes which of those conditions is worth intervening on to bring more users into the flow. ### Behavior Models Source: https://academia.sh/en/courses/user-experience/behavior-models That motivation, ability, and a trigger must hold at the same time for a behavior to occur; computing the effect of two types of intervention on a population; and why the motivation gain fades. The previous lesson reduced the borrowing flow's load from thirteen decision points to six and pulled the time from 7.44 seconds to 3.32 seconds. That is a gain that makes things easier for the user who **enters** the flow. The unanswered question is this: why would the user enter the flow at all? Most catalog users open a record's detail, read it, and leave without borrowing. Even when the load is low, the behavior may not occur. This lesson establishes which conditions must hold at the same time for a behavior to occur, and computes which of these conditions is worth intervening on to bring more users to act. #### Three Conditions Are Required at Once A widely used **behavior model** holds that a behavior occurs at the intersection of three conditions. **Motivation** is the degree to which the user wants to perform that behavior. The motivation to borrow a record in the catalog comes from the user's need for that book; the interface does not create this need, at most it makes it visible. **Ability** is the ease of performing the behavior. Ability is not the user's skill alone; it is determined together with the task's cost. The same user has lower ability in a flow with thirteen decision points than in a flow with six. A **trigger** is the signal that starts the behavior at that moment. The Borrow button on the record detail is a trigger; so is a message notifying the user that a book's due date is approaching. The three combine **multiplicatively**: if one is zero, the behavior does not happen. A user with high motivation who does not see the trigger does not borrow. A user who sees the trigger but lacks the ability to complete the flow does not borrow either. This means an observed "low conversion" can have at least three different causes, and which one it is cannot be found by guessing. #### The Effect of Two Interventions on the Population The computation below builds a population of two thousand users. Each user has a motivation and a base skill. Ability is a quantity determined jointly by skill and flow cost: as cost rises, ability falls. The behavior occurs when the product of motivation and ability clears a threshold and the trigger is seen. Two interventions are compared: raising motivation by fifteen points (a text that makes the benefit of borrowing visible) and reducing the flow from thirteen decisions to six. ```js // behavior.mjs — computing the motivation, ability, and trigger framework over a population function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } const POPULATION = 2000; const THRESHOLD = 20; // motivation x ability must clear this threshold // Population: each user's motivation (0-100) and base skill (0.2-1.0) function population() { const rnd = generator(19770401); const people = []; for (let i = 0; i < POPULATION; i++) { people.push({ motivation: rnd() * 100, skill: 0.2 + rnd() * 0.8 }); } return people; } // Flow cost pulls ability down; cost is the decision time from lesson 02. const ability = (skill, cost) => skill * (4 / (4 + cost)); function rate(people, { cost, motivationBoost = 0, triggerRate = 1 }) { let acted = 0; for (const p of people) { const m = Math.min(100, p.motivation + motivationBoost); if (m * ability(p.skill, cost) >= THRESHOLD) acted++; } return (acted / people.length) * triggerRate * 100; } const PEOPLE = population(); const FULL_FLOW = 7.44; // lesson 02: flow with 13 decision points const REDUCED = 3.32; // lesson 02: flow with 6 decision points console.log("intervention flow cost acted"); const baseline = rate(PEOPLE, { cost: FULL_FLOW }); console.log(`none (baseline) ${FULL_FLOW.toFixed(2)} s ${baseline.toFixed(1).padStart(6)} %`); const motivated = rate(PEOPLE, { cost: FULL_FLOW, motivationBoost: 15 }); console.log(`motivation +15 ${FULL_FLOW.toFixed(2)} s ${motivated.toFixed(1).padStart(6)} %`); const abled = rate(PEOPLE, { cost: REDUCED }); console.log(`flow reduced from 13 to 6 decisions ${REDUCED.toFixed(2)} s ${abled.toFixed(1).padStart(6)} %`); const both = rate(PEOPLE, { cost: REDUCED, motivationBoost: 15 }); console.log(`both together ${REDUCED.toFixed(2)} s ${both.toFixed(1).padStart(6)} %`); // Trigger: if the user does not see the call to action at that moment, the behavior does not occur console.log("\ntrigger visibility acted (reduced flow)"); for (const t of [0.3, 0.6, 0.9, 1.0]) { console.log(`${(t * 100).toFixed(0).padStart(21)} % ${rate(PEOPLE, { cost: REDUCED, triggerRate: t }).toFixed(1).padStart(6)} %`); } // Which intervention moves whom: split the population into four zones const zone = (p) => (p.motivation >= 50 ? "high motivation" : "low motivation") + (p.skill >= 0.6 ? " / high skill" : " / low skill"); const ZONES = ["high motivation / high skill", "high motivation / low skill", "low motivation / high skill", "low motivation / low skill"]; function zoneRates(settings) { const count = {}, total = {}; for (const z of ZONES) { count[z] = 0; total[z] = 0; } for (const p of PEOPLE) { const z = zone(p); total[z]++; const m = Math.min(100, p.motivation + (settings.motivationBoost ?? 0)); if (m * ability(p.skill, settings.cost) >= THRESHOLD) count[z]++; } return ZONES.map((z) => ({ z, rate: (count[z] / total[z]) * 100, n: total[z] })); } console.log("\nzone people baseline motivation+15 flow shortening"); const z0 = zoneRates({ cost: FULL_FLOW }); const z1 = zoneRates({ cost: FULL_FLOW, motivationBoost: 15 }); const z2 = zoneRates({ cost: REDUCED }); for (let i = 0; i < ZONES.length; i++) { console.log( `${ZONES[i].padEnd(36)} ${String(z0[i].n).padStart(5)} ${(z0[i].rate.toFixed(1) + " %").padStart(8)}` + ` ${(z1[i].rate.toFixed(1) + " %").padStart(14)} ${(z2[i].rate.toFixed(1) + " %").padStart(14)}` ); } // Second visit: the motivation boost fades, the flow shortening remains console.log("\nvisit motivation intervention flow shortening"); for (const v of [1, 2, 3]) { const boost = v === 1 ? 15 : 0; // the persuasive text is effective only on the first visit const m = rate(PEOPLE, { cost: FULL_FLOW, motivationBoost: boost }); const a = rate(PEOPLE, { cost: REDUCED }); console.log(`${String(v).padStart(5)} ${(m.toFixed(1) + " %").padStart(20)} ${(a.toFixed(1) + " %").padStart(15)}`); } // Threshold curve: the minimum motivation required for a behavior to occur console.log("\nability minimum motivation required"); for (const sk of [0.2, 0.3, 0.4, 0.6, 0.8, 1.0]) { const full = THRESHOLD / ability(sk, FULL_FLOW); const red = THRESHOLD / ability(sk, REDUCED); console.log(`${sk.toFixed(1).padStart(7)} full flow ${(full > 100 ? "unreachable" : full.toFixed(1)).padStart(11)} reduced flow ${(red > 100 ? "unreachable" : red.toFixed(1)).padStart(11)}`); } ``` ``` intervention flow cost acted none (baseline) 7.44 s 14.6 % motivation +15 7.44 s 23.5 % flow reduced from 13 to 6 decisions 3.32 s 34.6 % both together 3.32 s 47.0 % trigger visibility acted (reduced flow) 30 % 10.4 % 60 % 20.8 % 90 % 31.2 % 100 % 34.6 % zone people baseline motivation+15 flow shortening high motivation / high skill 531 55.0 % 84.0 % 96.8 % high motivation / low skill 501 0.2 % 2.2 % 26.3 % low motivation / high skill 485 0.0 % 2.7 % 9.7 % low motivation / low skill 483 0.0 % 0.0 % 0.0 % visit motivation intervention flow shortening 1 23.5 % 34.6 % 2 14.6 % 34.6 % 3 14.6 % 34.6 % ability minimum motivation required 0.2 full flow unreachable reduced flow unreachable 0.3 full flow unreachable reduced flow unreachable 0.4 full flow unreachable reduced flow 91.5 0.6 full flow 95.3 reduced flow 61.0 0.8 full flow 71.5 reduced flow 45.8 1.0 full flow 57.2 reduced flow 36.6 ``` #### Intervening on Ability Delivers More Than Intervening on Motivation The first table compares the two interventions on the same scale. Raising motivation by fifteen points moves the acted-on rate from 14.6 percent to 23.5 percent. Shortening the flow moves the same rate to 34.6 percent. Both together reach 47.0 percent. For this comparison to be fair, both interventions would need to be "fifteen units"; they are not. What makes the comparison meaningful is that the two interventions' **costs** are also similar: writing a text and binding seven fields to defaults are comparable amounts of work. For the same effort, the second intervention brings more than twice as many users into the flow. The reason shows up in the third table. The motivation intervention raises the "high motivation / high skill" zone from 55 percent to 84 percent — a group that was already close to acting. In the other three zones, its effect stays under 3 percent. Shortening the flow, on the other hand, raises the "high motivation / low skill" zone from 0.2 percent to 26.3 percent. This zone is a quarter of the population and consists of users who genuinely want the book but cannot complete the flow. The last table gives the reason for this gap plainly. For users with ability of 0.4 or below, the full flow is **unreachable**: even at a hundred points of motivation, the threshold is not cleared. When the flow is shortened, the motivation required for a user with ability 0.4 drops to 91.5, and for a user with ability 0.6 it drops from 95.3 to 61.0. The motivation intervention can do nothing for these users; their problem is not unwillingness but inability. For a user with ability of 0.3 or below, even shortening the flow is not enough — for this group six decision points are still too many, and what the design needs is not shortening the flow a little more but investigating why that user cannot complete the task. #### Motivation Fades, Ability Stays The fourth table adds the time dimension. The persuasive text gives 23.5 percent on the first visit; on the second visit the user has already read the same text, its effect no longer holds, and the rate returns to the baseline of 14.6 percent. Shortening the flow gives 34.6 percent on every visit. This is the most practical difference between the two types of intervention. The motivation intervention requires repetition; its effect shrinks with each repetition, and preserving the effect requires raising the dose. Raising the dose means exaggerating the message. Beyond a point, exaggeration parts ways with accuracy, and that parting is exactly the boundary this topic addresses at its end. The ability intervention requires no repetition; once the flow is shortened, it stays short. The fact that taking a decision away from the user is permanent also makes it more important that the decision be bound to the correct default. #### The Trigger Is a Multiplier, Not an Addend The second table shows that the trigger is a different kind of quantity. At 30 percent visibility, the acted-on rate is 10.4; at 100 percent, it is 34.6. The relationship is linear, because the trigger multiplies the other two conditions: even if the user is ready, if they do not see the signal at that moment, the behavior does not occur. There are two separate questions in the design of a trigger, and they get conflated. The first is **visibility**: is the call to action where the user is looking? This is the visual hierarchy question from the Fundamentals of Interface Design course. The second is **timing**: does the trigger arrive at the moment the user's motivation and ability clear the threshold? The second question is the source of the distinction. A message notifying the user that a book's due date is approaching arrives at the moment the user needs that information; the user, had they known they would want this message, would still want it. A message sent about something the user has no interest in at all is not a trigger but an interruption. The difference between the two is whether the message rests on the user's existing motivation or tries to manufacture motivation. #### What the Model Does Not Say This framework is a diagnostic tool, not a prediction tool. The threshold value, the multiplicative form, and the motivation–ability distribution are assumptions; other assumptions give other numbers. What the computation carries is this: **an intervention cannot be chosen without first identifying which condition is missing when a behavior fails to occur.** If ability is missing, the persuasive text does not help: in the low-ability region raising motivation lifts participation from 0.2 percent only to 2.2, while shortening the flow lifts it to 26.3. When both conditions are missing, no intervention produces a result. The model does not say one more thing. It does not assume that a behavior occurring is **good**. In the model, one way to raise the acted-on rate is to lower the threshold — that is, to make the user commit to a decision without thinking it through. In the borrowing flow, that leaves the user with a book they did not want. The criterion here is the same: would the user make the same decision if they knew why the designer made this change? For shortening the flow, the answer is yes. It is not yes for a speed-up that prevents the decision from being considered. #### Summary - A behavior occurs when the product of motivation, ability, and trigger clears a threshold; if one of them is zero, the behavior does not happen. - Ability is not the user's skill but a quantity determined jointly by skill and task cost; shortening the flow raises ability. - In the computation, the motivation intervention moved the acted-on rate from 14.6 percent to 23.5 percent, shortening the flow moved it to 34.6 percent; both together reached 47.0 percent. - The motivation intervention affects the group already close to the threshold; for the low-ability group, the full flow is unreachable even at a hundred points of motivation. - The motivation gain fades on the second visit and preserving it requires raising the dose; the ability gain stays the same on every visit. - One way to raise the acted-on rate is to prevent the user from thinking the decision through; the criterion is whether the user would make the same decision knowing about the change. #### Next Step This lesson addressed a single behavior: the user borrows once. The catalog interface's real question is repetition — does the user come back a second, a fifth, a twentieth time? A repeated behavior is explained by a different structure than a one-time behavior. The next lesson builds that structure: the loop formed by cue, routine, and reward, how it strengthens through repetition, and under what condition the designer has the right to build that loop. ### The Habit Loop Source: https://academia.sh/en/courses/user-experience/habit-loop How the loop formed by cue, routine, and reward strengthens with repetition, the cost of weakening one of the three parts, and why habit strength is not the same thing as the benefit delivered to the user. The previous lesson established the conditions for a single behavior to occur: the product of motivation, ability, and trigger must clear a threshold. This framework explains the first borrow. The catalog interface's real question is repetition. Does the user come back a second time, a fifth time, a twentieth time? A repeated behavior is not recomputed from scratch every time. Once repeated enough, the behavior stops being a matter of clearing the motivation threshold and becomes a direct response to a signal. This lesson builds the structure of that transformation and discusses under what condition the designer has the right to build that structure. #### The Loop's Three Parts The **habit loop** consists of three parts. A **cue** is the signal that starts the behavior. Its difference from the trigger in the previous lesson is that the cue is **recurring and tied to context**: a due-date reminder that arrives every Tuesday is a cue; it is not a one-time announcement. The cue's power comes from its predictability. A **routine** is the behavior performed after the cue. In the catalog interface, the routine is extending the duration of a borrowed book or adding a new book to a shelf. The routine's cost was the subject of the previous two lessons: the number of decision points, decision time, ability. A **reward** is the outcome that follows the routine and reinforces the behavior. A reward does not have to be a gamification element; finding the book they wanted on the shelf, having a duration extension confirmed instantly, or knowing they no longer have to track the due date are also rewards. The three parts form a loop: the cue starts the routine, the routine leads to the reward, and the reward strengthens the effect of the next cue. **Habit strength** is a quantity that grows with how many times this loop has been completed. #### The Loop's Strengthening Through Repetition The simulation below follows five hundred users over twenty-six weeks. Each week the cue may or may not arrive; when the cue arrives, the user may or may not complete the routine; when they complete it, the reward may or may not follow. Habit strength grows with a rewarded repetition and decays in a week without repetition. Five designs are compared. Four of them weaken one part of the loop; the fifth strengthens all three parts but lowers the rate at which a visit delivers a benefit to the user. ```js // habit.mjs — how the cue, routine, and reward loop strengthens with repetition function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } const ALPHA = 0.22; // strength gain from each reinforced repetition const DELTA = 0.06; // decay in a week without repetition const WEEK = 26; const PEOPLE = 500; // A design: cue reliability, routine cost (0-1), reward reliability, and the rate // at which a visit actually delivers a benefit to the user. const DESIGNS = [ { name: "no cue", cue: 0.15, cost: 0.45, reward: 0.90, benefit: 0.85 }, { name: "unreliable reward", cue: 0.90, cost: 0.45, reward: 0.30, benefit: 0.85 }, { name: "heavy routine", cue: 0.90, cost: 0.80, reward: 0.90, benefit: 0.85 }, { name: "all three in place", cue: 0.90, cost: 0.25, reward: 0.90, benefit: 0.85 }, { name: "frequent cue, irrelevant content", cue: 0.98, cost: 0.20, reward: 0.95, benefit: 0.18 }, ]; // One person's 26 weeks. If a cutoff is given, the cue thins out after that week. function person(design, rnd, cutoff = null) { let h = 0.05; let visits = 0, useful = 0, threshold = null; const hByWeek = {}; for (let week = 1; week <= WEEK; week++) { const cueRate = cutoff !== null && week > cutoff ? 0.15 : design.cue; const cueArrived = rnd() < cueRate; // Completing the routine: requires the cue; probability rises with habit strength, falls with cost const completes = cueArrived && rnd() < (0.35 + 0.6 * h) * (1 - design.cost); if (completes) { visits++; if (rnd() < design.benefit) useful++; h = rnd() < design.reward ? h + ALPHA * (1 - h) : h * (1 - DELTA); } else { h = h * (1 - DELTA); } if (threshold === null && h >= 0.5) threshold = week; hByWeek[week] = h; } return { hByWeek, visits, useful, threshold }; } function population(design, cutoff = null) { const rnd = generator(20250115); let visits = 0, useful = 0, thresholdCount = 0, thresholdTotal = 0; const h = { 6: 0, 12: 0, 18: 0, 26: 0 }; const cutoffH = {}; for (let i = 0; i < PEOPLE; i++) { const p = person(design, rnd, cutoff); visits += p.visits; useful += p.useful; for (const w of [6, 12, 18, 26]) h[w] += p.hByWeek[w] / PEOPLE; if (cutoff !== null) cutoffH[cutoff] = (cutoffH[cutoff] ?? 0) + p.hByWeek[cutoff] / PEOPLE; if (p.threshold !== null) { thresholdCount++; thresholdTotal += p.threshold; } } return { h, visits: visits / PEOPLE, useful: useful / PEOPLE, thresholdRate: (thresholdCount / PEOPLE) * 100, thresholdWeek: thresholdCount === 0 ? null : thresholdTotal / thresholdCount, cutoffH: cutoff === null ? null : cutoffH[cutoff], }; } console.log("design h(6) h(12) h(18) h(26) crosses 0.5 avg. crossing visits"); const S = {}; for (const d of DESIGNS) { const s = population(d); S[d.name] = s; console.log( `${d.name.padEnd(35)} ${[6, 12, 18, 26].map((w) => s.h[w].toFixed(2)).join(" ")}` + ` ${(s.thresholdRate.toFixed(1) + " %").padStart(11)} ${(s.thresholdWeek === null ? "-" : "week " + s.thresholdWeek.toFixed(1)).padStart(10)} ${s.visits.toFixed(1).padStart(7)}` ); } // Cost of weakening one of the loop's three parts console.log("\nreference: all three in place (h(26) = " + S["all three in place"].h[26].toFixed(2) + ", visits " + S["all three in place"].visits.toFixed(1) + ")"); const ref = S["all three in place"]; for (const name of ["no cue", "unreliable reward", "heavy routine"]) { const s = S[name]; console.log(`${name.padEnd(35)} h(26) is ${(s.h[26] / ref.h[26] * 100).toFixed(0)} % of reference, visits ${(s.visits / ref.visits * 100).toFixed(0)} % of reference`); } // Number of visits and benefit delivered to the user are not the same thing console.log("\ndesign visits useful wasted useful rate"); for (const d of DESIGNS) { const s = S[d.name]; console.log( `${d.name.padEnd(35)} ${s.visits.toFixed(1).padStart(7)} ${s.useful.toFixed(1).padStart(7)}` + ` ${(s.visits - s.useful).toFixed(1).padStart(8)} ${((s.useful / s.visits * 100).toFixed(1) + " %").padStart(12)}` ); } // How long the habit lasts if the cue is thinned out after week 14 console.log("\nif the cue is thinned out after week 14"); for (const name of ["all three in place", "frequent cue, irrelevant content"]) { const d = DESIGNS.find((x) => x.name === name); const s = population(d, 14); console.log(`${name.padEnd(35)} h(14) ${s.cutoffH.toFixed(2)} -> h(26) ${s.h[26].toFixed(2)} (${(s.h[26] / s.cutoffH * 100).toFixed(0)} % remains), visits ${s.visits.toFixed(1)}`); } ``` ``` design h(6) h(12) h(18) h(26) crosses 0.5 avg. crossing visits no cue 0.06 0.07 0.08 0.10 1.0 % week 19.4 0.9 unreliable reward 0.10 0.12 0.14 0.18 10.0 % week 17.1 5.4 heavy routine 0.10 0.14 0.16 0.20 13.2 % week 15.5 2.0 all three in place 0.30 0.49 0.61 0.69 95.4 % week 11.1 10.8 frequent cue, irrelevant content 0.36 0.59 0.72 0.81 98.8 % week 9.3 13.6 reference: all three in place (h(26) = 0.69, visits 10.8) no cue h(26) is 14 % of reference, visits 8 % of reference unreliable reward h(26) is 25 % of reference, visits 50 % of reference heavy routine h(26) is 29 % of reference, visits 19 % of reference design visits useful wasted useful rate no cue 0.9 0.7 0.1 86.0 % unreliable reward 5.4 4.6 0.8 85.6 % heavy routine 2.0 1.7 0.3 85.9 % all three in place 10.8 9.2 1.6 85.5 % frequent cue, irrelevant content 13.6 2.4 11.2 17.8 % if the cue is thinned out after week 14 all three in place h(14) 0.51 -> h(26) 0.31 (62 % remains), visits 5.6 frequent cue, irrelevant content h(14) 0.64 -> h(26) 0.39 (61 % remains), visits 7.2 ``` #### All Three Parts Are Required, But Not in the Same Way The first table gives the result of weakening the loop's parts one at a time. In the design with all three in place, habit strength is 0.30 at week six and 0.69 at week twenty-six; 95.4 percent of users cross the 0.5 threshold. When one part is weakened, strength at week twenty-six drops to between 14 and 29 percent of the reference. The second table shows that the parts fail **in different ways**. Without a cue, the user comes on average 0.9 times over twenty-six weeks: the loop never starts. With an unreliable reward, the number of visits stays at 50 percent of the reference but habit strength stays at 25 percent — the user comes, does the routine, and no reinforcement occurs. With a heavy routine, visits drop to 19 percent, but those who do come get reinforced; the strength ratio (29 percent) is higher than the visit ratio (19 percent). The design conclusion is this: **which part is weak can be read from the observed pattern of numbers.** If visits are low and strength is low, the cue is missing. If visits exist but strength does not, the reward is missing. If visits are low but those who come stay, the routine is heavy. All three collapse into the same "low repetition" indicator, and intervention is not possible without separating them. #### Habit Strength Is Not a Success Metric The third table is the center of gravity of this lesson. The "frequent cue, irrelevant content" design ranks first on every habit indicator: the highest strength (0.81), the highest threshold-crossing rate (98.8 percent), the earliest crossing (week 9.3), and the most visits (13.6). It beats the design with all three in place in every column. In that same table, the last column says the opposite. Of the thirteen and a half visits, only 2.4 deliver a benefit to the user; eleven visits are wasted. In the design with all three in place, 9.2 of 10.8 visits are useful. The second design produces fewer visits but **four times as many useful visits**. Visit count, session duration, and return frequency measure habit, not benefit. If an interface is optimized for these indicators, a design that takes the user's time but gives them nothing looks "successful." The criterion has been the same throughout this course: if the user knew they were sent a cue twice a week and that only two out of ten of those cues were useful to them, would they keep receiving those cues? If the answer is no, the habit serves the indicator, not the user. This does not mean it is wrong for the designer to build a habit. A library user's habit of never missing a due date serves the user's interest; the user would want this reminder even knowing about it. The distinction is who the habit serves, and it is measured by the last column of the third table. #### A Habit Does Not Carry Itself The last table tests an assumption often stated: once a habit is established, does it continue on its own? When the cue is thinned out after week fourteen, habit strength drops to roughly 61 percent in both designs. In the design with all three in place, from 0.51 to 0.31; in the other, from 0.64 to 0.39. There is no meaningful difference between the two designs. The conclusion is this: in this model, the habit stays dependent on the cue. When the cue is cut, the behavior fades. This has two implications. For design: removing a reminder does not mean the user will keep up the behavior; the removal decision is made by measurement. For ethics: a design that makes the user dependent on a cue also holds the power to cut that cue. That power is a service when the cue serves the user's interest; it is a dependency when it does not. The model's limit should be stated here. The simulation does not account for the user's own internal reasons: a real user may look for the book they want even without the reminder. What the model shows is that the **interface-produced portion** of the habit is dependent on the cue; the portion that comes from the user's own motivation is outside this simulation. #### The Rules for Building the Loop The legitimate way to build a habit in the catalog interface is summarized by four rules. **The cue is tied to the user's calendar, not the designer's calendar.** A reminder that arrives as a due date approaches arises from the user's own situation. Sending "new arrivals" on a fixed day of the week arises from the interface's schedule; it does not assume the user has a need at that moment. **The routine is shortened, but the decision is not deleted.** As measured in the previous lesson, lowering the routine's cost strengthens the loop. What gets lowered is the number of decisions, not the user's opportunity to notice what they are borrowing. **The reward is the routine's natural consequence.** The reward for extending a duration is the duration having been extended. A reward unrelated to the routine (badge, counter, streak) feeds the loop but, because it is not tied to the user's benefit, lowers the last column of the third table. **The useful-visit rate is tracked as a separate indicator.** If visit count is reported alone, every change that produces wasted visits looks like an improvement. #### Summary - The habit loop consists of cue, routine, and reward; the three reinforce each other through repetition, and habit strength accumulates through those repetitions. - The parts fail in different patterns: if the cue is missing, both visits and strength drop; if the reward is missing, visits exist but strength does not; if the routine is heavy, visits are few but those who come get reinforced. - A design that ranks first on every habit indicator can stay at a useful-visit rate of 17.8 percent; habit strength is not a success metric. - If the useful-visit rate is not tracked as a separate indicator, every change that takes the user's time looks like an improvement. - In the simulation, the habit stays dependent on the cue: when the cue is thinned out, strength drops to roughly 61 percent; a habit does not carry itself. - The legitimacy criterion is the same: would the user keep receiving the cues if they knew what proportion of them actually served them? #### Next Step This lesson addressed repeating a behavior and showed that lowering the routine's cost strengthens the loop. The most powerful tool for lowering cost appeared in the previous lesson but was not measured: binding decision points to a default. A default is the value the user holds when they do nothing, and its effect is both surprisingly large and the most contested power in the designer's hands. The next lesson computes that effect in two scenarios and ties the criterion that makes a default legitimate to a number. ### The Power of Defaults Source: https://academia.sh/en/courses/user-experience/power-of-defaults Computing choice architecture's effect on participation rate across two settings, defining regret rate as a legitimacy criterion, and how switch cost turns a default into a trap. The previous two lessons showed that lowering the routine's cost strengthens the loop. The most powerful tool for lowering cost appeared in the Cognitive Load lesson but was not measured: binding decision points to a default. That was what reduced the thirteen-point borrowing flow to six. A **default** is the value in effect when the user does nothing. Most users do not change this value — not changing it is not deciding, and not deciding is always cheaper. This is the most powerful and the most contested tool in the designer's hands: setting a checkbox's starting state determines the outcome for the majority of users. This lesson measures that power across two settings and ties the criterion that makes a default legitimate to a number. #### Choice Architecture and Three Layouts **Choice architecture** is the way options are presented to the user: which one is pre-checked, in what order they appear, how many steps changing one takes. Choice architecture cannot be neutral — a checkbox is either checked or not, and either state makes one outcome more likely than the other. The catalog interface's borrow confirmation screen has two settings. The first is the **due date reminder**: should a notification be sent to the user as the due date approaches? The second is the **weekly recommendations newsletter**: should a list of recommendations be sent to the user once a week? Three layouts will be compared: - **Opt-in default**: the setting starts off; whoever wants it turns it on. - **Opt-out default**: the setting starts on; whoever does not want it turns it off. - **Active choice**: there is no default; the user must pick one to continue the flow. Two quantities will be measured. **Participation rate** is the share of users for whom the setting ends up on. **Regret rate** is the share of users who end up in a state different from what they would choose if nothing stood in their way. The second quantity is this lesson's real criterion. #### The Result of Two Settings Across Three Layouts The simulation builds a population of four thousand users. Each user has a preference strength: positive means they want the setting, negative means they do not, and its absolute value gives how much. Switching the default requires a cost; the user switches the default only if their preference strength exceeds that cost. ```js // default.mjs — effect of the default on participation rate and regret rate function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } // Deterministic normal distribution (Box-Muller), for generating preference strength. function normalGenerator(seed) { const rnd = generator(seed); return () => { const u1 = Math.max(rnd(), 1e-12), u2 = rnd(); return Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2); }; } const PEOPLE = 4000; const SWITCH_COST = 0.55; // preference strength required to switch the default // Two settings. mu: center of the preference distribution; positive means the majority wants it. const SETTINGS = [ { name: "due date reminder", mu: 0.78 }, { name: "weekly recommendations newsletter", mu: -0.78 }, ]; function population(mu, seed) { const n = normalGenerator(seed); const prefs = []; for (let i = 0; i < PEOPLE; i++) prefs.push(mu + n()); return prefs; } // Three layouts: opt-out default, opt-in default, active choice. function outcome(pref, layout, cost) { if (layout === "active choice") return pref > 0; // the user chooses directly const defaultOn = layout === "opt-out default"; const switches = defaultOn ? (pref < 0 && -pref > cost) : (pref > 0 && pref > cost); return defaultOn ? !switches : switches; } const LAYOUTS = ["opt-in default", "opt-out default", "active choice"]; console.log("setting informed pref. layout participation regret"); for (const s of SETTINGS) { const u = population(s.mu, 20260318); const informed = u.filter((x) => x > 0).length / PEOPLE; for (const layout of LAYOUTS) { let on = 0, regret = 0; for (const x of u) { const o = outcome(x, layout, SWITCH_COST); if (o) on++; if (o !== x > 0) regret++; } console.log( `${s.name.padEnd(36)} ${((informed * 100).toFixed(1) + " %").padStart(14)} ${layout.padEnd(15)}` + ` ${((on / PEOPLE * 100).toFixed(1) + " %").padStart(7)} ${((regret / PEOPLE * 100).toFixed(1) + " %").padStart(10)}` ); } } // How regret changes as switch cost grows console.log("\nswitch cost due date reminder (opt-out default) newsletter (opt-out default)"); console.log(" participation regret participation regret"); const uA = population(SETTINGS[0].mu, 20260318); const uB = population(SETTINGS[1].mu, 20260318); for (const m of [0.0, 0.25, 0.55, 1.0, 1.5, 2.5]) { const measure = (u) => { let on = 0, regret = 0; for (const x of u) { const o = outcome(x, "opt-out default", m); if (o) on++; if (o !== x > 0) regret++; } return [(on / PEOPLE * 100).toFixed(1), (regret / PEOPLE * 100).toFixed(1)]; }; const [onA, regretA] = measure(uA), [onB, regretB] = measure(uB); console.log(`${m.toFixed(2).padStart(11)} ${(onA + " %").padStart(7)} ${(regretA + " %").padStart(10)} ${(onB + " %").padStart(7)} ${(regretB + " %").padStart(10)}`); } // Active choice zeroes out regret but adds a decision point. // lesson 02: every decision point lengthens the flow; in a longer flow some users drop out. console.log("\nthe cost of active choice (newsletter setting)"); const EXTRA_DECISION_TIME = 0.44; // lesson 02: A + B*log2(4) for a three-option field for (const dropout of [0.0, 0.03, 0.06, 0.10]) { const completing = 1 - dropout; let informedOn = 0; for (const x of uB) if (x > 0) informedOn++; const activeOn = (informedOn / PEOPLE) * completing; const defaultOn = uB.filter((x) => outcome(x, "opt-out default", SWITCH_COST)).length / PEOPLE; const defaultRegret = uB.filter((x) => outcome(x, "opt-out default", SWITCH_COST) !== x > 0).length / PEOPLE; console.log( `dropout ${(dropout * 100).toFixed(0).padStart(2)} % flow completion ${(completing * 100).toFixed(0)} % ` + `active choice participation ${(activeOn * 100).toFixed(1)} % / regret 0.0 % ` + `opt-out default ${(defaultOn * 100).toFixed(1)} % / ${(defaultRegret * 100).toFixed(1)} %` ); } console.log(`time active choice adds to the flow: ${EXTRA_DECISION_TIME.toFixed(2)} s`); ``` ``` setting informed pref. layout participation regret due date reminder 78.0 % opt-in default 58.7 % 19.4 % due date reminder 78.0 % opt-out default 90.8 % 12.8 % due date reminder 78.0 % active choice 78.0 % 0.0 % weekly recommendations newsletter 22.1 % opt-in default 9.0 % 13.1 % weekly recommendations newsletter 22.1 % opt-out default 40.9 % 18.8 % weekly recommendations newsletter 22.1 % active choice 22.1 % 0.0 % switch cost due date reminder (opt-out default) newsletter (opt-out default) participation regret participation regret 0.00 78.0 % 0.0 % 22.1 % 0.0 % 0.25 84.8 % 6.7 % 29.9 % 7.8 % 0.55 90.8 % 12.8 % 40.9 % 18.8 % 1.00 96.2 % 18.1 % 58.2 % 36.1 % 1.50 98.8 % 20.7 % 76.3 % 54.2 % 2.50 99.9 % 21.9 % 95.8 % 73.7 % the cost of active choice (newsletter setting) dropout 0 % flow completion 100 % active choice participation 22.1 % / regret 0.0 % opt-out default 40.9 % / 18.8 % dropout 3 % flow completion 97 % active choice participation 21.4 % / regret 0.0 % opt-out default 40.9 % / 18.8 % dropout 6 % flow completion 94 % active choice participation 20.8 % / regret 0.0 % opt-out default 40.9 % / 18.8 % dropout 10 % flow completion 90 % active choice participation 19.9 % / regret 0.0 % opt-out default 40.9 % / 18.8 % time active choice adds to the flow: 0.44 s ``` #### The Same Layout, Two Different Things in Two Settings The first table's participation column measures the power of the default. The due date reminder gets 58.7 percent with the opt-in default, 90.8 percent with the opt-out default. The newsletter gets 9.0 percent with the opt-in default, 40.9 percent with the opt-out default. For both settings, flipping the default changes participation by roughly thirty-two points; as a ratio that is one and a half times for the due date reminder and four and a half times for the newsletter. The only thing that changed is a checkbox's starting state. Anyone looking only at the participation column reaches the same conclusion for both settings: the opt-out default is better. The regret column refutes that. For the due date reminder, the opt-out default's regret rate is 12.8 percent, the opt-in default's is 19.4 percent. The opt-out default **moves users closer** to their informed preference. The opt-in default leaves a 19.4 percent group who want the reminder but do not bother turning it on without it. For the newsletter, the signs flip. The opt-out default's regret rate is 18.8 percent, the opt-in default's is 13.1 percent. The opt-out default **moves users away** from their informed preference: some of the 3116 users who do not want the newsletter keep receiving it. This yields the lesson's rule. **The default is chosen to minimize the regret rate.** This is the measurable form of saying "the default follows whichever direction the majority's informed preference points." For the due date reminder, the majority wants it, so the default is on. For the newsletter, the majority does not want it, so the default is off. The rule pays no attention to participation rate at all. It is possible to raise newsletter participation from 9 percent to 40.9 percent, and the entirety of that increase comes from users who do not want the setting. #### Switch Cost Turns the Default into a Trap The second table shows where the default's power comes from: switch cost. When cost is zero, regret is zero for both settings. Everyone moves to the state they want; the default has no effect at all. As cost rises, both participation and regret rise. The rates of increase differ between the two settings. For the due date reminder, as cost rises from 0 to 2.5, participation rises from 78 percent to 99.9 percent and regret from 0 percent to 21.9 percent. For the newsletter, participation rises from 22.1 percent to 95.8 percent, while regret rises from 0 percent to **73.7 percent**. At a cost of 2.5, three out of every four users receiving the newsletter do not want it. This number is the quantitative definition of the "hard to cancel" pattern. When turning off a setting is made difficult, participation rate rises; the entirety of the rise comes from regret. On a dashboard that reports only participation rate, the two situations are indistinguishable: 95.8 percent participation shows up as 95.8 percent whether the setting is one users want or one they do not. The only quantity that makes the distinction is the regret rate, and the regret rate cannot be read from interface logs. It requires knowing the user's informed preference, which is the question of the measurement topic. #### The Cost and Place of Active Choice Active choice zeroes out regret for both settings: because the user chooses directly, they never end up in a state they did not want. Its cost is in the third table. Active choice adds one decision point to the flow; by the computation in the Cognitive Load lesson, that is 0.44 seconds. More important than the time is dropout: in a longer flow, some users do not complete the borrow. At a 10 percent dropout rate, the share of users who want the newsletter drops from 22.1 percent to 19.9 percent — what is lost is not the newsletter itself but the borrowing transaction. This means active choice cannot be used everywhere. The decision of when to leave the choice to the user is made with three criteria: **Is the outcome reversible?** A newsletter subscription can be turned off at any time; a book ordered to the wrong branch cannot be corrected after the fact. For irreversible decisions, active choice is worth the dropout cost. **Are preferences split?** If the informed preference shows a distribution like 78 to 22 percent, the default serves the majority. If the distribution is close to 50 percent, no default can lower regret, and active choice is the only correct layout. **Whose interest does the decision serve?** If the setting being on serves the interface's interest and not the user's, choosing the default is a choice the designer makes under a conflict of interest. In that case the decision is left to the user. #### The Rules for Setting a Default **The default is set in the direction of the informed majority's preference.** Knowing this requires measurement; a default based on guessing is not an assumption, it is an imposition. **Switch cost is lowered as much as possible.** A default's legitimacy is directly proportional to how easy it is to change. Every step that makes changing it harder raises the regret rate. **The default is written visibly.** A setting whose current value the user cannot see is not a default but a hidden decision. Writing "delivery branch: main" on the borrow confirmation screen lets the user know that decision has been made. **Participation rate and regret rate are reported together.** When only participation is reported, every change that raises regret looks like an improvement. #### Summary - A default is the value in effect when the user does nothing; in the simulation, flipping the default changed the participation rate by more than thirty points. - Participation rate does not show whether a default is correct; the same layout lowers regret for one setting (from 19.4 percent to 12.8 percent) while raising it for another (from 13.1 percent to 18.8 percent). - The legitimacy criterion is the regret rate: the default is set in the direction that minimizes the share of users who end up in a state different from their informed preference. - The default's power comes from switch cost; at zero cost, the default has no effect, and as cost grows, the entirety of the participation increase comes from regret — in the computation, 95.8 percent participation against 73.7 percent regret. - Active choice zeroes out regret but adds a decision point and produces dropout; it is chosen for irreversible decisions, split preferences, and settings with a conflict of interest. - When participation rate is reported alone, a hard-to-cancel setting and a well-chosen default look the same. #### Next Step A default silently steers the user's decision. The interface also has a tool that speaks openly: showing what others have done. Information in the catalog like "users who borrowed this record also borrowed" or "most borrowed this month" changes the user's decision. The next lesson builds the mechanism of that change, computes whether the displayed number creates a self-reinforcing loop, and addresses the point at which social proof stops being information and turns into steering. ### Social Proof and Authority Source: https://academia.sh/en/courses/user-experience/social-proof-and-authority How visible counts produce an information cascade that suppresses independent judgment, how a popularity list feeds itself, and that an authority signal carries value only when it is more reliable than the user's own knowledge. The previous lesson showed that a default silently steers the user's decision. The interface has one more tool that speaks openly: showing what others have done. In the catalog, "users who borrowed this record also borrowed," "most borrowed this month," or a borrow count shown next to a record are all of this kind. **Social proof** is a person using others' behavior as a source of information in an uncertain situation. The mechanism is sound at its core: others' choices may carry something they know. This lesson measures where that soundness ends. #### A Visible Count Destroys Independent Information The first computation models a catalog situation. A work has two editions; one is genuinely better (more complete, more legible). Looking on their own, each user can tell the correct edition apart with 68 percent probability — a good but imperfect signal. Forty users arrive in sequence. Two layouts are compared: one where each user sees only their own signal, and one where they also see the borrow counts accumulated so far. ```js // social.mjs — effect of a visible count on information-carrying capacity function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } const SIGNAL = 0.68; // accuracy of the user's own signal const USERS = 40; const RUNS = 4000; // ---- 1) Correct majority decision: independent vote versus visible count ---- // There are two editions, one genuinely better (1). Each user gets a 68% accurate signal. function sequential(visibleCount, rnd) { let n0 = 0, n1 = 0, ownSignal = 0, cascadeStart = null; for (let i = 0; i < USERS; i++) { const signal = rnd() < SIGNAL ? 1 : 0; const diff = n1 - n0; let choice; if (!visibleCount || Math.abs(diff) < 2) { choice = signal; ownSignal++; } else { choice = diff >= 2 ? 1 : 0; if (cascadeStart === null) cascadeStart = i; } if (choice === 1) n1++; else n0++; } return { n0, n1, ownSignal, cascadeStart }; } // Probability that the majority of k independent signals is correct function binomialCoeff(n, k) { let r = 1; for (let i = 1; i <= k; i++) r = (r * (n - k + i)) / i; return r; } function majorityAccuracy(k, q) { let t = 0; for (let i = Math.floor(k / 2) + 1; i <= k; i++) t += binomialCoeff(k, i) * q ** i * (1 - q) ** (k - i); return t; } console.log("layout majority correct used own signal cascade started at"); const measured = {}; for (const visible of [false, true]) { const rnd = generator(20260527); let majorityCorrect = 0, own = 0, start = 0, startCount = 0; for (let k = 0; k < RUNS; k++) { const r = sequential(visible, rnd); if (r.n1 > r.n0) majorityCorrect++; own += r.ownSignal; if (r.cascadeStart !== null) { start += r.cascadeStart; startCount++; } } const rate = majorityCorrect / RUNS; measured[visible ? "visible" : "hidden"] = rate; console.log( `${(visible ? "count visible" : "count hidden").padEnd(15)} ${((rate * 100).toFixed(1) + " %").padStart(16)}` + ` ${(own / RUNS).toFixed(1).padStart(16)} ${(startCount === 0 ? "-" : "user " + (start / startCount).toFixed(1)).padStart(24)}` ); } // How many independent votes the visible-count layout is equivalent to console.log("\nindependent votes probability majority is correct"); for (const k of [1, 3, 5, 11, 21, 41]) { console.log(`${String(k).padStart(18)} ${(majorityAccuracy(k, SIGNAL) * 100).toFixed(1)} %`); } let equiv = 1; while (equiv < 41 && majorityAccuracy(equiv, SIGNAL) < measured.visible) equiv += 2; console.log(`with the count visible, 40 users amount to ${equiv} independent votes`); // ---- 2) The self-reinforcing effect of the "most borrowed" list ---- const RECORDS = 200, ROUNDS = 60, ROUND_USERS = 100, LIST = 10, LIST_BOOST = 6; function rankCorrelation(a, b) { // Spearman rank correlation const rank = (v) => { const s = v.map((x, i) => [x, i]).sort((p, q) => q[0] - p[0]); const r = new Array(v.length); s.forEach(([, i], k) => { r[i] = k + 1; }); return r; }; const ra = rank(a), rb = rank(b), n = a.length, mean = (n + 1) / 2; let num = 0, pa = 0, pb = 0; for (let i = 0; i < n; i++) { num += (ra[i] - mean) * (rb[i] - mean); pa += (ra[i] - mean) ** 2; pb += (rb[i] - mean) ** 2; } return num / Math.sqrt(pa * pb); } function popularity(listShown) { const rnd = generator(20260601); const quality = Array.from({ length: RECORDS }, () => 0.2 + rnd() * 0.8); const counts = new Array(RECORDS).fill(0), firstRound = new Array(RECORDS).fill(0); for (let round = 0; round < ROUNDS; round++) { const topTen = [...counts.keys()].sort((a, b) => counts[b] - counts[a]).slice(0, LIST); const inList = new Set(listShown ? topTen : []); const weight = quality.map((q, i) => q * (inList.has(i) ? LIST_BOOST : 1)); const total = weight.reduce((a, b) => a + b, 0); for (let u = 0; u < ROUND_USERS; u++) { let x = rnd() * total, i = 0; while (x > weight[i] && i < RECORDS - 1) { x -= weight[i]; i++; } counts[i]++; if (round === 0) firstRound[i]++; } } return { quality, counts, firstRound }; } console.log("\nlist quality correlation first-round luck correlation true top 10 in top 10"); for (const listShown of [false, true]) { const { quality, counts, firstRound } = popularity(listShown); const best = new Set([...quality.keys()].sort((a, b) => quality[b] - quality[a]).slice(0, LIST)); const topTen = [...counts.keys()].sort((a, b) => counts[b] - counts[a]).slice(0, LIST); console.log( `${(listShown ? "list shown" : "no list").padEnd(20)} ${rankCorrelation(counts, quality).toFixed(3).padStart(19)}` + ` ${rankCorrelation(counts, firstRound).toFixed(3).padStart(29)} ${(topTen.filter((i) => best.has(i)).length + " / 10").padStart(24)}` ); } // ---- 3) When does an authority signal replace one's own signal ---- console.log("\nauthority accuracy follows authority own signal follows the more reliable 5 independent votes"); const fiveVotes = majorityAccuracy(5, SIGNAL); for (const a of [0.50, 0.60, 0.68, 0.80, 0.92]) { console.log( `${a.toFixed(2).padStart(18)} ${((a * 100).toFixed(1) + " %").padStart(16)} ${((SIGNAL * 100).toFixed(1) + " %").padStart(10)}` + ` ${((Math.max(a, SIGNAL) * 100).toFixed(1) + " %").padStart(28)} ${((fiveVotes * 100).toFixed(1) + " %").padStart(21)}` ); } ``` ``` layout majority correct used own signal cascade started at count hidden 98.4 % 40.0 - count visible 82.1 % 3.5 user 3.5 independent votes probability majority is correct 1 68.0 % 3 75.8 % 5 80.9 % 11 89.7 % 21 95.8 % 41 99.2 % with the count visible, 40 users amount to 7 independent votes list quality correlation first-round luck correlation true top 10 in top 10 no list 0.918 0.177 7 / 10 list shown 0.858 0.290 1 / 10 authority accuracy follows authority own signal follows the more reliable 5 independent votes 0.50 50.0 % 68.0 % 68.0 % 80.9 % 0.60 60.0 % 68.0 % 68.0 % 80.9 % 0.68 68.0 % 68.0 % 68.0 % 80.9 % 0.80 80.0 % 68.0 % 80.0 % 80.9 % 0.92 92.0 % 68.0 % 92.0 % 80.9 % ``` #### The Information Cascade and the Effective Vote Count The first table gives a reversed result. When the count is hidden, the majority of forty users finds the correct edition 98.4 percent of the time. When the count is visible, that rate drops to 82.1 percent. Giving users **more information** makes the collective decision **worse**. The reason is in the second column. When the count is hidden, all forty of the forty users use their own signal. When it is visible, only 3.5 do. On average, after user 3.5, the count difference reaches two votes, and beyond that point no user's own signal is enough to change the decision. Everyone follows the count, the count grows, and the reason to follow it strengthens. This structure is called an **information cascade**. (It should not be confused with cascading in CSS; what cascades here is decisions, not style rules.) The cascade's critical property is this: the information held by the thirty-six people who arrive after the fourth user never enters the system. The count grows, but the information inside it does not. The third block reduces this to a single number. A majority of independent votes gives 80.9 percent accuracy at 5 votes and 99.2 percent at 41 votes. The 82.1 percent accuracy produced by forty users in the visible-count layout is equivalent to **seven independent votes**. Thirty-three of the forty users contribute nothing at all to the collective information. The design consequence is clear: the moment the count is shown matters. A count shown **after** the user has made their own decision does not corrupt information; a count shown **before** the decision suppresses the information of every user who comes after it. #### A Popularity List Feeds Itself The second computation follows borrows in a two-hundred-record catalog over sixty rounds. Each record has a quality. Two layouts are compared: one where the "ten most borrowed records" list is shown and one where it is not. Without the list, the ranking's rank correlation with quality is 0.918, with first-round luck it is 0.177; seven of the resulting top ten records are genuinely among the ten highest-quality records. With the list shown, the quality correlation drops to 0.858, the luck correlation rises to 0.290, and only **one** of the resulting top ten is in the true top ten. This third number exposes what the first two conceal. The overall correlation degrades very little (0.918 to 0.858) because the ranking of the one hundred ninety records outside the list is still determined by quality. What degrades is exactly the list itself: records that got lucky in the first round enter the list, being on the list gives them six times the visibility, that visibility raises their counts, and that keeps them on the list. Records that are high quality but unlucky in the first round never rise again. The result is that the list **does not measure what it claims to**. The "most borrowed" heading promises the user a quality ranking; what it measures is, to a large extent, early visibility. If the user knew how this list was produced, they would not look at it as a quality signal. This does not mean a popularity list is wrong under every condition. What produces the distortion is the **feedback loop**: being on the list makes staying on the list easier. There are design decisions that break the loop — limiting the list to a narrow time window, removing records from the list for a period after they enter it, binding the list to a harder-to-game metric such as completion or post-return rating instead of borrow count. Each of these decouples the visibility gain from the count. #### Authority Is Information Only If It Is More Reliable The last table addresses the authority signal: labels like "on the department reading list," "librarian's pick," "faculty selection." The user's own signal is 68 percent correct. At an authority accuracy of 0.50, following the authority is 50 percent correct — had the user decided on their own, they would have gotten 68 percent. When authority accuracy rises to 0.68, the two options are tied. Only at 0.80 and 0.92 does following the authority pay off. The rule is this: **an authority signal is information to the degree that it is more reliable than the user's own judgment; if it is not, it has merely taken over the decision.** When a label is shown in the interface, the user cannot make this comparison, because they do not know the label's accuracy. What makes a label meaningful is that its basis is visible: "which course's reading list," "who recommended it," "selected by what criterion." An authority label whose basis is not written gives the user not a number but only a direction. The last column ties the table back to the first part of this lesson. A majority of five **independent** users gives 80.9 percent accuracy — better than an authority with 0.80 accuracy. A crowd, as long as it stays independent, is more reliable than most authorities. But the first computation showed that a visible count destroys exactly this independence. What makes social proof valuable and the way it is displayed undo each other. #### Criteria for Recognition and Rejection Whether a social proof or authority signal is legitimate is audited with three questions. **Does the number have a verifiable basis?** "412 users borrowed this record" corresponds to a countable fact. "This record is very popular" corresponds to nothing and cannot be audited. An unauditable claim is not social proof; it is the appearance of social proof. **Is the number shown before or after the user's decision?** A count shown before the decision produces a cascade. Instead of showing the borrow count to the user opening the record, saying "users who borrowed this record also borrowed" after they have borrowed gives the same information without breaking independence. **Is the scarcity real?** The claim "the last remaining copy of this record," if true from the catalog data, is information useful to the user: it changes their waiting decision. If the same claim is shown independent of the actual copy count, it is a fabricated scarcity signal manufactured to push the user into a hasty decision. The two look the same on screen; the distinction lies in the data's source. The common thread of these three questions is the criterion this topic has used from the start. Would the decision change if the user knew how the number they see was produced? If the borrow count comes from a real tally, it would not. If the popularity list comes from a self-reinforcing loop, it would. If the scarcity warning is shown independent of stock, it would — and that means the decision's power comes from the user's ignorance. #### Summary - Social proof is the use of others' behavior as a source of information; its value depends on that behavior arising from independent judgments. - A visible count produces an information cascade: in the simulation, majority accuracy dropped from 98.4 percent to 82.1 percent, and only 3.5 of forty users were able to use their own signal. - Forty users under a visible count produce information equivalent to seven independent votes; the remaining users' information never enters the system. - A popularity list builds a feedback loop: in the computation, only one of the resulting top ten was genuinely among the ten highest-quality records; the list measures early visibility, not quality. - An authority signal carries information only if it is more reliable than the user's own judgment; a majority of five independent users is more accurate than an authority with 0.80 accuracy. - The audit is done with three questions: does the number have a verifiable basis, is it shown before the decision, and is the scarcity real? #### Next Step This lesson addressed the information the interface gives the user **about others**. There is one more kind of information the interface gives, and it concerns the user directly: where they are, how far they have come, how much is left. The next lesson computes the effect of visible progress on flow completion, addresses the thresholds at which feedback delay changes behavior, and separates when a progress indicator is information and when it is decoration. ### Progress and Feedback Source: https://academia.sh/en/courses/user-experience/progress-and-feedback Computing whether a progress indicator correctly reports the work remaining, distinguishing deception that raises the completion rate, and tying feedback delay to indicator classes. The previous lesson addressed the information the interface gives the user about others. There is one more kind of information the interface gives, and it concerns the user directly: where they are in the flow, how far they have come, how much is left. This information feeds a single decision the user makes: continue or leave? The decision is remade at the end of every step, and the only input the user has is their estimate of how much work remains. This lesson measures that estimate, computes how the indicator changes it, and separates when an indicator that raises the completion rate is information and when it is deception. #### Step Count Does Not Measure Time The borrowing flow consists of six steps, but the steps are not equal in length. Opening a record detail takes 0.8 seconds, copy selection takes 5.1 seconds. A progress bar can be filled in two different ways: by step count (three of six steps done, 50 percent) or by time (the ratio of elapsed time to total time). The computation below compares the two bars' estimates of remaining time, then simulates three thousand users' decisions to continue. ```js // progress.mjs — accuracy of the progress indicator and its effect on dropout function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } // The borrowing flow's six steps and their actual durations (s). const STEPS = [ ["search", 1.2], ["result list", 3.4], ["record detail", 0.8], ["copy selection", 5.1], ["confirmation", 1.0], ["receipt", 2.2], ]; const DURATIONS = STEPS.map(([, d]) => d); const TOTAL = DURATIONS.reduce((a, b) => a + b, 0); const elapsed = DURATIONS.map((_, i) => DURATIONS.slice(0, i + 1).reduce((a, b) => a + b, 0)); // Two indicators: a bar that advances by step count, a bar weighted by time. const stepBar = (i) => (i + 1) / STEPS.length; const timeBar = (i) => elapsed[i] / TOTAL; console.log("step time elapsed step bar time bar remaining (actual) step bar est. time bar est."); let stepError = 0, timeError = 0; for (let i = 0; i < STEPS.length - 1; i++) { const remaining = TOTAL - elapsed[i]; const tA = elapsed[i] * (1 - stepBar(i)) / stepBar(i); const tS = elapsed[i] * (1 - timeBar(i)) / timeBar(i); stepError += Math.abs(tA - remaining) / remaining; timeError += Math.abs(tS - remaining) / remaining; console.log( `${STEPS[i][0].padEnd(15)} ${DURATIONS[i].toFixed(1).padStart(4)} ${elapsed[i].toFixed(1).padStart(7)}` + ` ${(stepBar(i) * 100).toFixed(0).padStart(9)} % ${(timeBar(i) * 100).toFixed(0).padStart(9)} %` + ` ${remaining.toFixed(1).padStart(19)} ${tA.toFixed(1).padStart(15)} ${tS.toFixed(1).padStart(15)}` ); } const n = STEPS.length - 1; console.log(`average relative error: step bar ${(stepError / n * 100).toFixed(1)} %, time bar ${(timeError / n * 100).toFixed(1)} %`); // Dropout decision: at the end of each step, the user estimates remaining time and compares it to their patience. // The correct decision is to continue if the ACTUAL remaining time does not exceed patience. const USERS = 3000; const UNCERTAINTY = 2.2; // pessimism multiplier the user applies with no indicator function decision(indicator) { const rnd = generator(20260714); let completing = 0, unnecessaryDropout = 0, unnecessaryContinue = 0; for (let k = 0; k < USERS; k++) { const patience = 5 + rnd() * 40; // accepted remaining time (s) let exited = false; for (let i = 0; i < STEPS.length - 1; i++) { const actualRemaining = TOTAL - elapsed[i]; const correctToContinue = actualRemaining <= patience; let estimate; if (indicator === "none") estimate = elapsed[i] * UNCERTAINTY; else if (indicator === "step bar") estimate = elapsed[i] * (1 - stepBar(i)) / stepBar(i); else estimate = elapsed[i] * (1 - timeBar(i)) / timeBar(i); const keepGoing = estimate <= patience; if (keepGoing && !correctToContinue) unnecessaryContinue++; if (!keepGoing && correctToContinue) unnecessaryDropout++; if (!keepGoing) { exited = true; break; } } if (!exited) completing++; } return { rate: (completing / USERS) * 100, unnecessaryDropout, unnecessaryContinue }; } console.log("\nindicator completion unnecessary dropout unnecessary continuation wrong decisions"); for (const g of ["none", "step bar", "time bar"]) { const t = decision(g); console.log( `${g.padEnd(15)} ${(t.rate.toFixed(1) + " %").padStart(9)} ${String(t.unnecessaryDropout).padStart(21)}` + ` ${String(t.unnecessaryContinue).padStart(25)} ${String(t.unnecessaryDropout + t.unnecessaryContinue).padStart(15)}` ); } console.log(`(${USERS} users, the flow's actual total time is ${TOTAL.toFixed(1)} s)`); // Feedback delay classes and the indicator each class requires const OPERATIONS = [ ["typing in the search field", 0.04], ["applying a filter", 0.18], ["opening a record", 0.7], ["copy status query", 2.4], ["creating the borrow record", 6.5], ["generating the receipt", 14.0], ]; const classify = (t) => t <= 0.1 ? "instant" : t <= 1.0 ? "flow preserved" : t <= 10 ? "attention drifts" : "attention breaks"; const required = (t) => t <= 0.1 ? "no indicator needed" : t <= 1.0 ? "a state change is enough" : t <= 10 ? "an indeterminate wait indicator" : "determinate progress + remaining time"; console.log("\noperation time class required feedback"); for (const [name, t] of OPERATIONS) { console.log(`${name.padEnd(28)} ${(t.toFixed(2) + " s").padStart(7)} ${classify(t).padEnd(17)} ${required(t)}`); } // Artificial delay: the relationship between displayed progress and actual work console.log("\nrelationship between displayed progress and actual work"); const actualWork = [0.10, 0.25, 0.45, 0.70, 0.88, 1.00]; const conditions = { "tied to reality": actualWork, "constant speed (artificial)": [0.17, 0.33, 0.50, 0.67, 0.83, 1.00], "stuck near the end": [0.30, 0.60, 0.85, 0.93, 0.96, 1.00], }; for (const [name, g] of Object.entries(conditions)) { let maxDeviation = 0, total = 0; for (let i = 0; i < g.length; i++) { const s = Math.abs(g[i] - actualWork[i]); total += s; if (s > maxDeviation) maxDeviation = s; } console.log(`${name.padEnd(28)} average deviation ${(total / g.length * 100).toFixed(1)} points, max deviation ${(maxDeviation * 100).toFixed(1)} points`); } ``` ``` step time elapsed step bar time bar remaining (actual) step bar est. time bar est. search 1.2 1.2 17 % 9 % 12.5 6.0 12.5 result list 3.4 4.6 33 % 34 % 9.1 9.2 9.1 record detail 0.8 5.4 50 % 39 % 8.3 5.4 8.3 copy selection 5.1 10.5 67 % 77 % 3.2 5.3 3.2 confirmation 1.0 11.5 83 % 84 % 2.2 2.3 2.2 average relative error: step bar 31.3 %, time bar 0.0 % indicator completion unnecessary dropout unnecessary continuation wrong decisions none 48.7 % 1226 558 1784 step bar 89.2 % 11 476 487 time bar 81.4 % 0 0 0 (3000 users, the flow's actual total time is 13.7 s) operation time class required feedback typing in the search field 0.04 s instant no indicator needed applying a filter 0.18 s flow preserved a state change is enough opening a record 0.70 s flow preserved a state change is enough copy status query 2.40 s attention drifts an indeterminate wait indicator creating the borrow record 6.50 s attention drifts an indeterminate wait indicator generating the receipt 14.00 s attention breaks determinate progress + remaining time relationship between displayed progress and actual work tied to reality average deviation 0.0 points, max deviation 0.0 points constant speed (artificial) average deviation 4.7 points, max deviation 8.0 points stuck near the end average deviation 21.0 points, max deviation 40.0 points ``` #### The Step Bar's Error Has a Direction In the first table, the two bars diverge at the first step. When search finishes, the step bar shows 17 percent, the time bar shows 9 percent. The actual remaining time is 12.5 seconds. A user looking at the step bar estimates 6.0 seconds — less than half the truth. The time bar gives 12.5 seconds. Average relative error is 31.3 percent for the step bar, 0 percent for the time bar. The second number is not an achievement, it is correct by definition: the time bar already shows the ratio of elapsed time, so the estimate derived from it gives the actual remaining time. The meaningful number is the first. The step bar's error is not random, either. Before the long step arrives, the bar produces an optimistic estimate (6.0 instead of 12.5); after the long step has passed, it produces a pessimistic one (5.3 instead of 3.2). That is, the user thinks the remaining work is **less** than it actually is at the start of the flow. #### Completion Rate Is the Wrong Metric The second table is the center of gravity of this lesson. With no indicator, the completion rate is 48.7 percent; with the step bar, 89.2 percent; with the time bar, 81.4 percent. Anyone looking only at completion rate would pick the step bar. The three columns next to it say what that choice actually is. With the time bar, the number of wrong decisions is zero: no user leaves a flow they would have preferred to continue, and none stays in a flow they could not tolerate. With the step bar, 476 users **unnecessarily continue** — because they think the remaining work is less than it is, they stay in a flow beyond what their patience allows. The step bar raising completion rate from 81.4 percent to 89.2 percent looks like a seven-point-eight gain. The source of that gain is the user misestimating the remaining work. This has the same structure as the regret rate in The Power of Defaults lesson: the number on the dashboard rises, and the entirety of that rise comes from being wrong. The no-indicator condition shows a different kind of loss: 1226 **unnecessary dropouts**. These are users who leave because of uncertainty even though the remaining work is within their patience. This is the real justification for a progress indicator — it is put there not to keep the user in the flow but to ground the user's decision in reality. A correct indicator raising the completion rate is a consequence of that decision, not its purpose. #### Feedback Delay Determines the Indicator Class The third table addresses a different question: what should be shown to the user while an operation is running? The answer depends on the operation's duration and is classified with three established thresholds. **Up to 0.1 seconds**, the user does not perceive the delay; the result appears to arrive instantly. Typing in the search field is in this class and needs no indicator. Adding a loading indicator in this class does harm: the indicator appearing and disappearing makes a delay that does not exist visible. **Up to 1 second**, the delay is perceived but does not break the user's train of thought. Applying a filter and opening a record are in this class. Sufficient feedback is the result itself changing — the list refreshes, the page arrives. A separate wait indicator is not required. **Up to 10 seconds**, the user's attention drifts; they start looking at something else. Copy status query and creating the borrow record are in this class. A wait indicator is required here; if how long the work will take is unknown, an indeterminate indicator is sufficient, because the only decision the user makes is not "should I wait" but "is the interface working." **Above 10 seconds**, attention breaks. Generating the receipt is in this class. Here an indeterminate indicator is not enough; for the user to be able to decide whether to leave the task and come back, progress and remaining time must be shown. This is where the second table's computation applies. #### Progress Is Bound to Actual Work The last table is an audit tool: it measures the deviation between displayed progress and the actual proportion of work completed. The indicator tied to reality gives zero deviation. The constant-speed indicator — where the bar advances in equal intervals and does not reflect the actual distribution of work — deviates by 4.7 points on average, 8.0 at most. The indicator stuck near the end — where the bar shoots up to 85 percent and waits there — deviates by 21.0 points on average, 40.0 at most. The sign of the deviation matters. The indicator stuck near the end shows the work as **more** complete than it is; the user waits because it looks "almost done," and their wait grows longer. This is a design that raises completion rate but grounds the user's decision in wrong information. Another form of the same structure is holding a wait indicator up for a fixed duration even though the work has actually finished; this is a delay added to make the work done look heavier than it is. The criterion is the same criterion as in every lesson of this topic. If the user knew the bar was tied to a constant speed rather than actual work, would they keep waiting on the screen? For the indicator stuck near the end, the answer is no: the user keeps waiting because they trust the information "85 percent done." For the indicator tied to reality, the answer is yes; the user is already deciding with correct information. #### Summary - A progress indicator feeds a single decision the user makes: continue or leave. It is useful to the degree that it correctly reports the work remaining. - A bar that advances by step count produces a faulty estimate when the steps are not equal in length; in the computation, average relative error is 31.3 percent, and the error runs optimistic at the start of the flow. - Completion rate is the wrong metric: the step bar raises completion from 81.4 percent to 89.2 percent, but that gain comes from 476 users thinking the remaining work is less than it is. - The correct metric is the number of wrong decisions; with no indicator, 1226 users drop out unnecessarily, and with the time bar the number of wrong decisions is zero. - Feedback delay splits into four classes: up to 0.1 seconds no indicator is needed, up to 1 second a state change is enough, up to 10 seconds an indeterminate wait indicator, above that determinate progress and remaining time are required. - A progress bar is bound to actual work; a constant-speed or stuck-near-the-end indicator grounds the user's wait decision in wrong information. #### Next Step Across the seven lessons of this topic, the same criterion appeared seven times: would the user make the same choice if they knew how the design decision was made? In salience, in the default, in social proof, in the progress bar, the same distinction showed up every time — does the decision's power come from the user's knowledge or their ignorance? The next lesson systematizes this criterion: it ties the patterns of rushing, hidden cost, hard-to-cancel settings, confirmshaming, and false scarcity to countable indicators and builds a way to audit an interface with those indicators. ### Dark Patterns and Ethics Source: https://academia.sh/en/courses/user-experience/dark-patterns-and-ethics Turning a design decision's legitimacy criterion into countable indicators, and recognizing and rejecting the patterns of urgency, hidden costs, the roach motel, the consent trap, and false scarcity. Across the seven lessons of this topic, the same question came up seven times. In the dual-process thinking lesson: would the user pick the first record if they knew why the list was ordered this way? In the defaults lesson: would they leave the setting on if they noticed it came pre-checked? In the social proof lesson: would they trust the number if they knew how it was produced? In the progress lesson: would they keep waiting if they knew the bar was not tied to actual work? The question was the same every time, and every time one of two answers came out. This lesson turns the question into a criterion, and the criterion into auditable indicators. #### The Criterion **A design decision is legitimate if the user would make the same choice even knowing how that decision was made. If the decision's power comes from the user's ignorance, that decision is a dark pattern.** The criterion has two properties. First, it does not by itself treat changing the user's behavior as a flaw — every design decision changes behavior; sorting records by relevance changes it too. The flaw is the change being built on a false belief. Second, it does not ask about the designer's intent. A pattern may have been built by accident; the criterion still gives the same result, and the obligation to fix it is the same. A **dark pattern** is an interface pattern that systematically violates this criterion, turning the outcome toward what the designer wants by relying on the user's wrong information. The five patterns below are this lesson's audit list. **Urgency** is a pattern that narrows the user's thinking time without a real constraint: countdowns that never end or that restart on every session. **Hidden costs** is a pattern that discloses part of the price the user committed to only after the commitment. The **roach motel** is a pattern that makes entering a state easy and exiting it disproportionately hard. The **consent trap** is a pattern in which what the user believes they agreed to diverges from what they actually agreed to: pre-checked boxes, layouts that make the decline option invisible. **False scarcity** is a claim of shortage about stock, time, or demand that is not based on data. None of these is given here as a recipe for implementation; each is defined as a **recognition target**. The computation below ties recognition to measurement. #### Measuring the Indicators ```js // audit.mjs — measuring an interface's dark-pattern indicators // ---- 1) Effort asymmetry: the step gap between entering something and exiting it ---- const FLOWS = [ { name: "borrow / return", entry: 2, exit: 2 }, { name: "due date reminder", entry: 1, exit: 1 }, { name: "recommendations newsletter", entry: 1, exit: 4 }, { name: "membership", entry: 3, exit: 7 }, ]; console.log("flow entry steps exit steps asymmetry threshold (2x)"); for (const f of FLOWS) { const asymmetry = f.exit / f.entry; console.log( `${f.name.padEnd(29)} ${String(f.entry).padStart(11)} ${String(f.exit).padStart(12)}` + ` ${(asymmetry.toFixed(2) + "x").padStart(9)} ${asymmetry > 2 ? "EXCEEDED" : "passed"}` ); } // ---- 2) Hidden cost: the gap between the first disclosed commitment and the final total ---- const COSTS = [ { name: "borrowing (free)", disclosed: 0, final: 0 }, { name: "late return fee", disclosed: 0, final: 12 }, { name: "inter-branch transfer", disclosed: 5, final: 5 }, { name: "reservation + transfer + late fee", disclosed: 5, final: 23 }, ]; console.log("\nitem disclosed upfront final total surprise ratio"); for (const c of COSTS) { const surprise = c.final === 0 ? 0 : (c.final - c.disclosed) / c.final; console.log( `${c.name.padEnd(34)} ${String(c.disclosed).padStart(14)} ${String(c.final).padStart(11)}` + ` ${((surprise * 100).toFixed(0) + " %").padStart(14)} ${surprise > 0.2 ? "EXCEEDED" : "passed"}` ); } // ---- 3) The scarcity claim's link to data ---- // For 12 records: actual shelf status and the "copies left" claim shown in the interface. const ACTUAL = [1, 4, 2, 7, 1, 3, 9, 2, 5, 1, 6, 8]; const CLAIMS = { "derived from data": [1, 4, 2, 7, 1, 3, 9, 2, 5, 1, 6, 8], "fixed warning": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "partially tied": [1, 3, 2, 3, 1, 3, 3, 2, 3, 1, 3, 3], }; function correlation(a, b) { const n = a.length; const oa = a.reduce((x, y) => x + y, 0) / n, ob = b.reduce((x, y) => x + y, 0) / n; let num = 0, pa = 0, pb = 0; for (let i = 0; i < n; i++) { num += (a[i] - oa) * (b[i] - ob); pa += (a[i] - oa) ** 2; pb += (b[i] - ob) ** 2; } return pb === 0 ? 0 : num / Math.sqrt(pa * pb); } console.log("\nclaim form correlation with actual stock threshold (0.9)"); for (const [name, v] of Object.entries(CLAIMS)) { const r = correlation(ACTUAL, v); console.log(`${name.padEnd(21)} ${r.toFixed(3).padStart(30)} ${r >= 0.9 ? "passed" : "EXCEEDED"}`); } // ---- 4) Whether the countdown is tied to a real deadline ---- // The same user's "time remaining" (minutes) seen across consecutive sessions. const SESSIONS = { "real deadline": [58, 41, 27, 12, 3], "resets every session": [10, 10, 10, 10, 10], "resets on page refresh": [10, 10, 9, 10, 10], }; console.log("\ncountdown sessions decreasing"); for (const [name, v] of Object.entries(SESSIONS)) { let decreasing = true; for (let i = 1; i < v.length; i++) if (v[i] >= v[i - 1]) decreasing = false; console.log(`${name.padEnd(32)} ${v.join(", ").padEnd(20)} ${decreasing ? "yes" : "NO"}`); } // ---- 5) Five indicators together: auditing a design decision ---- // Criterion: would the user make the same choice if they knew how the decision was made? // Indicators are measured individually; if even one is exceeded, the decision counts as a dark pattern. const DECISIONS = [ { name: "due date reminder opt-out default", regret: 0.128, asymmetry: 1.0, surprise: 0.0, basis: 1.00, countdown: null }, { name: "recommendations newsletter opt-out default", regret: 0.188, asymmetry: 4.0, surprise: 0.0, basis: 1.00, countdown: null }, { name: "last copy warning (from stock)", regret: 0.000, asymmetry: 1.0, surprise: 0.0, basis: 1.00, countdown: null }, { name: "last copy warning (fixed)", regret: 0.000, asymmetry: 1.0, surprise: 0.0, basis: 0.00, countdown: null }, { name: "reservation countdown", regret: 0.000, asymmetry: 1.0, surprise: 0.0, basis: 1.00, countdown: false }, { name: "late fee at the last step", regret: 0.000, asymmetry: 1.0, surprise: 0.52, basis: 1.00, countdown: null }, ]; const THRESHOLDS = { regret: 0.15, asymmetry: 2.0, surprise: 0.20, basis: 0.90 }; console.log("\ndecision exceeded indicator result"); for (const d of DECISIONS) { const exceeded = []; if (d.regret > THRESHOLDS.regret) exceeded.push("regret"); if (d.asymmetry > THRESHOLDS.asymmetry) exceeded.push("effort asymmetry"); if (d.surprise > THRESHOLDS.surprise) exceeded.push("hidden cost"); if (d.basis < THRESHOLDS.basis) exceeded.push("unsubstantiated claim"); if (d.countdown === false) exceeded.push("false urgency"); console.log( `${d.name.padEnd(45)} ${(exceeded.length === 0 ? "-" : exceeded.join(", ")).padEnd(29)}` + ` ${exceeded.length === 0 ? "legitimate" : "DARK PATTERN"}` ); } ``` ``` flow entry steps exit steps asymmetry threshold (2x) borrow / return 2 2 1.00x passed due date reminder 1 1 1.00x passed recommendations newsletter 1 4 4.00x EXCEEDED membership 3 7 2.33x EXCEEDED item disclosed upfront final total surprise ratio borrowing (free) 0 0 0 % passed late return fee 0 12 100 % EXCEEDED inter-branch transfer 5 5 0 % passed reservation + transfer + late fee 5 23 78 % EXCEEDED claim form correlation with actual stock threshold (0.9) derived from data 1.000 passed fixed warning 0.000 EXCEEDED partially tied 0.807 EXCEEDED countdown sessions decreasing real deadline 58, 41, 27, 12, 3 yes resets every session 10, 10, 10, 10, 10 NO resets on page refresh 10, 10, 9, 10, 10 NO decision exceeded indicator result due date reminder opt-out default - legitimate recommendations newsletter opt-out default regret, effort asymmetry DARK PATTERN last copy warning (from stock) - legitimate last copy warning (fixed) unsubstantiated claim DARK PATTERN reservation countdown false urgency DARK PATTERN late fee at the last step hidden cost DARK PATTERN ``` #### Reading the Four Indicators **Effort asymmetry** is the ratio of the number of steps required to enter a state to the number required to exit it. Borrowing and returning are both two steps; the ratio is 1.00. The recommendations newsletter turns on in one step and off in four; the ratio is 4.00. This measurement is taken not from logs but from the interface itself: by counting the two flows. Because counting requires no assumption, it is the indicator audited with the least effort. **Surprise ratio** is the share of the final total cost that was not disclosed at the moment of commitment. The inter-branch transfer fee is disclosed from the start, so its ratio is zero. The late fee is never disclosed, so its ratio is 100 percent. The fee being conditional is not a justification: if the user does not know what happens in case of a late return when they borrow, they have not seen part of their commitment. **The scarcity claim's link to data** is the correlation between the displayed shortage claim and the actual stock. The claim derived from data gives 1.000. A fixed warning that writes "last copy" on every record gives 0.000 — the claim corresponds to nothing. The third row measures an in-between case: the claim is partially related to stock (0.807) but stays fixed once stock rises above three. A partially true claim does not pass the audit; when the user looks at the number, they assume it is a measurement. **Whether the countdown decreases** shows whether the urgency is tied to a real deadline. With a real deadline, the remaining time seen across consecutive sessions decreases. A countdown that starts from the same value every session does not decrease. The third row shows a form that is easy to miss: the values wobble a little but do not trend in one direction; the time is not actually running out, it is only being shown as if it were. #### The Audit's Result and Its Limits The last table applies five indicators to six decisions together. Two of the decisions in the borrowing flow pass every threshold; four exceed at least one. Because the audit also records which indicator was exceeded, it produces a direct correction list: the newsletter's default is turned off and its cancel step is reduced to one; the fixed warning is tied to stock data or removed; the countdown is tied to a real deadline or removed; the late fee is written on the borrow confirmation screen. The audit's limits must be stated clearly. **The thresholds are a convention.** A factor of two for effort asymmetry, 20 percent for surprise ratio, 0.90 for correlation — these are chosen values, not computed ones. Their function is to turn debate into measurement: once a threshold is written down, whether a decision exceeds it is not debated, it is measured. The threshold itself can and should be debated. **A decision that passes every threshold can still be flawed.** The audit catches patterns that leave a measurable trace. A decline label that shames the user ("no thanks, I do not want to save money") does not disturb any number; this form of the consent trap can only be seen by reading the text. The audit is a screening tool, not proof. **No indicator can be offset by a rise in conversion.** A pattern raising the number of borrows does not mean it passes the audit; as measured in The Power of Defaults lesson and the Progress and Feedback lesson, the source of that rise is the user being mistaken. The size of a number built on being mistaken cannot be a decision's justification. #### Rejecting This entire topic was meant to show how much power the designer holds over the user's decision. The conclusion is that this power is already in use in every design decision: there is no neutral default, no neutral ordering, no neutral emphasis. This is why the professional stance cannot be "avoid influencing"; avoiding it is not possible. The stance is this: **What indicator a decision raises is written down.** When a change is proposed, what it will increase must be stated openly. Is the rising number borrows, or is it consent given unintentionally — the discussion cannot proceed without making this distinction. **The criterion is applied on the user's behalf.** The user is not in the room; it is the designer who applies the criterion in their place. If the answer to "would the user make the same choice even knowing this" is no, the decision is rejected. **A rejection is given with a reason.** The five indicators above turn the grounds for rejection from a personal preference into an auditable finding. Instead of "I do not want to do this," it is said "this decision's effort asymmetry is four, the threshold is two, and the cancel step can be reduced to one." **A legitimate alternative is proposed.** Every decision that fails the audit mostly has a counterpart that serves the same purpose without violating the criterion: real stock information instead of false scarcity, a symmetric flow instead of the roach motel, the full cost disclosed at the moment of commitment instead of a hidden cost. #### Summary - The legitimacy criterion is single: a decision is legitimate if the user would make the same choice even knowing how it was made; if the decision's power comes from ignorance, it is a dark pattern. - Five patterns are defined as recognition targets: urgency, hidden costs, the roach motel, the consent trap, false scarcity. - Four indicators leave a measurable trace: effort asymmetry, surprise ratio, the scarcity claim's correlation with data, and whether the countdown genuinely decreases. - Because the audit records which indicator was exceeded, it produces a direct correction list; the thresholds are a convention, written down to turn debate into measurement. - The audit is a screening tool: a decision that passes every threshold can still be flawed, and a rising conversion number cannot offset any indicator. - The designer's stance is not to avoid influencing but to write down what number a decision raises, apply the criterion on the user's behalf, ground rejection in a reason, and propose a legitimate alternative. #### Next Step Throughout this topic, applying the criterion continually required one thing: knowing the user's informed preference, the real task duration, the useful-visit rate. None of these is obtained by guessing. The next topic is devoted to measurement and begins with the most basic question: how is it shown that an interface is good? How are task success, duration, and error rate measured, how are these three numbers reduced to a single metric, and where does the average become misleading? ## Measurement Producing a composite metric from task success, duration, and error rate; locating drop-off points in the conversion funnel; designing A/B and multivariate tests; and combining quantitative findings with qualitative observation. ### Experience Metrics Source: https://academia.sh/en/courses/user-experience/experience-metrics Reporting task success together with its confidence interval, how duration-distribution skew misleads the arithmetic mean, and producing a weighted composite metric from three metrics. Throughout the Behavior Design topic, applying the criterion continually required one thing: knowing the user's informed preference, the real task duration, the useful-visit rate. None of these is obtained by guessing. This topic is devoted to measurement and begins with the most basic question: how is it shown that an interface is good? The raw record from a usability test carries three columns — whether the task was completed, how long it took, how many errors were made. This lesson takes up how those three columns are read and when they mislead. #### Three Basic Metrics **Task success** is the proportion of participants who complete the task. For the definition to work, "completed" must be written down in advance. If the task in the catalog interface is "find a specific volume and borrow it," the success criterion is that the correct volume enters the borrow record; a participant who borrows the wrong volume has not completed the task. **Time on task** is the time between starting the task and finishing it. The duration of failed sessions is not pooled together with that of successful sessions: a failed session is cut off at the time limit, and its real duration is unknown. **Error rate** is the number of erroneous actions taken during the task. An error is any action that moves the participant away from the task: applying the wrong filter, opening the wrong record, having to go back. The following computation processes a record of fourteen sessions using these three metrics. ```js // metrics.mjs — composite metric from task success, duration, and error rate // Raw record of 14 sessions: was the task completed, duration (s), error count. // Task: find a specific volume in the catalog and borrow it. const SESSIONS = [ ["P01", true, 42, 0], ["P02", true, 58, 1], ["P03", false, 180, 3], ["P04", true, 37, 0], ["P05", true, 91, 2], ["P06", true, 46, 0], ["P07", false, 180, 4], ["P08", true, 63, 1], ["P09", true, 39, 0], ["P10", true, 154, 2], ["P11", true, 51, 1], ["P12", false, 180, 2], ["P13", true, 44, 0], ["P14", true, 72, 1], ]; const n = SESSIONS.length; const succeeded = SESSIONS.filter(([, b]) => b); const successCount = succeeded.length; // ---- 1) Task success and the Wilson confidence interval ---- function wilson(k, N, z = 1.96) { const p = k / N, d = 1 + z * z / N; const center = (p + z * z / (2 * N)) / d; const half = (z * Math.sqrt(p * (1 - p) / N + z * z / (4 * N * N))) / d; return [center - half, center + half]; } const [lower, upper] = wilson(successCount, n); console.log(`task success: ${successCount}/${n} = ${(successCount / n * 100).toFixed(1)} %`); console.log(`Wilson 95% confidence interval: [${(lower * 100).toFixed(1)} %, ${(upper * 100).toFixed(1)} %]`); // ---- 2) Duration: arithmetic mean vs. geometric mean ---- const durations = succeeded.map(([, , s]) => s); const arithmetic = durations.reduce((a, b) => a + b, 0) / durations.length; const geometric = Math.exp(durations.reduce((a, s) => a + Math.log(s), 0) / durations.length); const sorted = [...durations].sort((a, b) => a - b); const median = sorted.length % 2 ? sorted[(sorted.length - 1) / 2] : (sorted[sorted.length / 2 - 1] + sorted[sorted.length / 2]) / 2; console.log(`\nduration of successful sessions (${durations.length} sessions): ${sorted.join(", ")}`); console.log(`arithmetic mean ${arithmetic.toFixed(1)} s, geometric mean ${geometric.toFixed(1)} s, median ${median.toFixed(1)} s`); console.log(`sessions above the mean: ${durations.filter((s) => s > arithmetic).length} / ${durations.length}`); // ---- 3) Error rate ---- const totalErrors = SESSIONS.reduce((a, [, , , h]) => a + h, 0); const errorFree = SESSIONS.filter(([, , , h]) => h === 0).length; console.log(`\ntotal errors ${totalErrors}, per session ${(totalErrors / n).toFixed(2)}, error-free sessions ${errorFree}/${n}`); // ---- 4) Reducing three metrics to a single number ---- // Each metric is first mapped to the 0-1 range; duration and errors are inverted. const DURATION_TARGET = 40, DURATION_CAP = 180, ERROR_CAP = 4; function sessionScore([, b, s, h]) { const success = b ? 1 : 0; const durationScore = b ? Math.max(0, Math.min(1, (DURATION_CAP - s) / (DURATION_CAP - DURATION_TARGET))) : 0; const errorScore = Math.max(0, 1 - h / ERROR_CAP); return { success, durationScore, errorScore, composite: 0.5 * success + 0.3 * durationScore + 0.2 * errorScore }; } console.log("\nsession success duration errors duration score error score composite"); for (const o of SESSIONS) { const s = sessionScore(o); console.log( `${o[0].padEnd(8)} ${(o[1] ? "yes" : "no").padEnd(8)} ${String(o[2]).padStart(9)} ${String(o[3]).padStart(7)}` + ` ${s.durationScore.toFixed(2).padStart(14)} ${s.errorScore.toFixed(2).padStart(12)} ${s.composite.toFixed(3).padStart(10)}` ); } const composites = SESSIONS.map((o) => sessionScore(o).composite); const compositeMean = composites.reduce((a, b) => a + b, 0) / n; const compositeStdDev = Math.sqrt(composites.reduce((a, b) => a + (b - compositeMean) ** 2, 0) / (n - 1)); console.log(`composite metric: mean ${compositeMean.toFixed(3)}, standard deviation ${compositeStdDev.toFixed(3)}, 95% interval +-${(1.96 * compositeStdDev / Math.sqrt(n)).toFixed(3)}`); // ---- 5) Effect of weights on the outcome ---- console.log("\nweight (success/duration/error) composite mean lowest session"); const WEIGHTS = [[0.5, 0.3, 0.2], [1.0, 0.0, 0.0], [0.34, 0.33, 0.33], [0.2, 0.6, 0.2]]; for (const [wSuccess, wDuration, wError] of WEIGHTS) { const scores = SESSIONS.map((o) => { const s = sessionScore(o); return wSuccess * s.success + wDuration * s.durationScore + wError * s.errorScore; }); const avg = scores.reduce((a, b) => a + b, 0) / n; const lowest = SESSIONS[scores.indexOf(Math.min(...scores))][0]; console.log(`${`${wSuccess} / ${wDuration} / ${wError}`.padEnd(26)} ${avg.toFixed(3).padStart(16)} ${lowest.padStart(15)}`); } ``` ``` task success: 11/14 = 78.6 % Wilson 95% confidence interval: [52.4 %, 92.4 %] duration of successful sessions (11 sessions): 37, 39, 42, 44, 46, 51, 58, 63, 72, 91, 154 arithmetic mean 63.4 s, geometric mean 57.6 s, median 51.0 s sessions above the mean: 3 / 11 total errors 17, per session 1.21, error-free sessions 5/14 session success duration errors duration score error score composite P01 yes 42 0 0.99 1.00 0.996 P02 yes 58 1 0.87 0.75 0.911 P03 no 180 3 0.00 0.25 0.050 P04 yes 37 0 1.00 1.00 1.000 P05 yes 91 2 0.64 0.50 0.791 P06 yes 46 0 0.96 1.00 0.987 P07 no 180 4 0.00 0.00 0.000 P08 yes 63 1 0.84 0.75 0.901 P09 yes 39 0 1.00 1.00 1.000 P10 yes 154 2 0.19 0.50 0.656 P11 yes 51 1 0.92 0.75 0.926 P12 no 180 2 0.00 0.50 0.100 P13 yes 44 0 0.97 1.00 0.991 P14 yes 72 1 0.77 0.75 0.881 composite metric: mean 0.728, standard deviation 0.380, 95% interval +-0.199 weight (success/duration/error) composite mean lowest session 0.5 / 0.3 / 0.2 0.728 P07 1 / 0 / 0 0.786 P03 0.34 / 0.33 / 0.33 0.712 P07 0.2 / 0.6 / 0.2 0.688 P07 ``` #### A Ratio Alone Misleads Task success is 11/14, that is, 78.6 percent. When this number is written alone in a report, it gives an impression of precision. The Wilson confidence interval tells the truth: between 52.4 and 92.4 percent. The interval shows what fourteen sessions do and do not determine. The true success rate could be 55 percent, or it could be 90. An interval this wide does not answer the question "is the interface good?" The question it does answer is different: does the interface work for more than half of users? To that, it says yes. Two rules follow from this. First, **a ratio is always written with its confidence interval**; a ratio without an interval hides the sample size. Second, the job of a small-sample test is not to estimate a ratio but to find problems. A usability test surfaces problems with a small number of participants; the number of participants needed to measure a ratio precisely is much larger. Using the Wilson method to compute the interval is not a minor detail. The simple interval based on the ratio's standard error can spill outside the 0–100 percent range at small sample sizes and ratios near the extremes; the Wilson interval does not, and it is more reliable at small sample sizes. #### The Duration Distribution Is Not Symmetric The durations of the eleven successful sessions range from 37 to 154 seconds. The arithmetic mean is 63.4 seconds. Only **three sessions are above** this mean; eight are below. The reason is the skew of the distribution. Task duration is bounded from below — no session can be shorter than zero seconds — but unbounded from above. If a user gets stuck, the duration doubles or triples. In distributions of this shape, the arithmetic mean is pulled upward by a few long sessions and does not represent the "typical" user. The geometric mean is 57.6, the median 51.0 seconds. The median is the duration below which half the sessions fall, and it is not affected by a single long session. The geometric mean is the average of the logarithms of the durations; it captures the center of skewed distributions better than the arithmetic mean, and unlike the mean, it behaves consistently with the ratios between session durations. Rule: **when duration is reported, the median or the geometric mean is written**; if the arithmetic mean is to be reported, the extremes of the distribution are given as well. In the catalog interface, the 154-second session is not an outlier; it is the trace of a real snag in the borrow flow, and it is not discarded — it is examined separately. #### Composite Metric and the Test of Weights When the three metrics are reported separately, comparison becomes difficult: if a design raises success but lengthens duration, which one is better? The composite metric is built to answer this question. Building it takes three steps. Each metric is first mapped to the 0–1 range; duration and errors run in the "higher is worse" direction, so they are inverted; then they are summed with weights. For the duration score, a target of 40 seconds and a cap of 180 seconds were chosen — these numbers come from a decision, not a measurement, and they must be written in the report. The fourth table gives the composite metric session by session. Session P10 completed the task (success 1) but took 154 seconds; its composite value is 0.656. P12 did not complete the task but made few errors; 0.100. A single number puts the information carried by the three columns into an order. The last table is the composite metric's most important test: does the outcome change if the weights change? The mean composite value moves between 0.688 and 0.786 — a range of 14 percentage points. That is, **the absolute level of the composite metric depends on the weights and carries no meaning on its own.** One thing does not change in the same table: the lowest session is P07 in three of the four weightings. The ranking is more robust against the weights. This is the correct use of the composite metric: **it is used to rank two designs or two sessions, not to measure against a threshold.** A target of the form "the composite metric should be above 0.75" means the person who chose the weights is the one setting the target. The standard deviation of the composite value is 0.380, and its 95 percent interval across the fourteen sessions is ±0.199. This width says that a composite difference smaller than 0.2 between two designs cannot be distinguished with fourteen sessions. #### Reporting the Metrics Together **The raw numbers are given together with the composite.** The composite metric hides which component changed; a change that lowers success but shortens duration can show up as an improvement in the composite. **Weights and thresholds are written down.** The duration target, the duration cap, the error cap, and the three weights are part of the report. Without them, the composite value cannot be audited. **Failed sessions do not enter the duration pool.** The duration of a session cut off at the time limit has not been measured; including it in the pool both pulls the duration distribution upward and produces an artificial cap. **A list of problems does not replace the metrics.** This lesson dealt with the numbers; the numbers do not say which step is bad. That question is the subject of the next lesson. #### Summary - The task success ratio is written with its confidence interval; the 78.6 percent success rate across fourteen sessions corresponds to a range of 52.4–92.4 percent, and the ratio alone is not enough to estimate the true rate. - The task duration distribution is right-skewed; only three of the eleven sessions fell above the arithmetic mean, so the median or the geometric mean is reported. - The duration of failed sessions is cut off at the time limit and does not enter the duration pool. - The composite metric maps the three metrics to the 0–1 range and sums them with weights; the target, cap, and weight values come from a decision, not a measurement, and are written in the report. - The absolute level of the composite metric is sensitive to the weights (0.688–0.786), while its ranking is more robust; it is therefore used for comparison, not for a threshold. - The uncertainty of the metrics is measured: the composite value's 95 percent interval is ±0.199, meaning differences smaller than this cannot be distinguished with this sample. #### Next Step This lesson measured a task as a whole: whether it was completed, how long it took, how many errors were made. What it did not measure is **where** an uncompleted task got stuck. Did all three failed sessions get stuck at the same step, or at three different places? The next lesson breaks the borrow flow into steps, measures the drop-off at each step separately, finds which step accounts for the largest loss, and shows that the highest drop-off rate and the largest user loss may not be the same step. ### Conversion Funnel Source: https://academia.sh/en/courses/user-experience/conversion-funnel Step-by-step drop-off analysis of the borrow flow, how the highest drop-off rate and the largest user loss occur at different steps, and how an aggregated funnel hides differences between segments. The previous lesson measured a task as a whole: whether it was completed, how long it took, how many errors were made. Three of the fourteen sessions failed to complete the task. What it did not measure is **where** those three sessions got stuck. This question is answered not by a small-sample test but by the behavior record of all users. The **conversion funnel** is an analysis that breaks a task into sequential steps and counts how many users pass through each one. In the catalog interface, the funnel's steps are the borrow flow: search, results list, record detail, copy selection, borrow confirmation, completion. #### The Funnel's Three Columns Each step has three separate numbers, and they get confused with one another. **Step conversion** is the proportion of those entering a step who move on to the next one. **Drop-off rate** is the proportion of those who leave at the same step; it is the complement of the step conversion. **Cumulative conversion** is the proportion of those entering the first step who reach a given step. Overall conversion is the **product** of the step conversions — this is the funnel's most important structural property. ```js // funnel.mjs — step-by-step drop-off analysis of the borrow flow const FUNNEL = [ ["search performed", 10000], ["results list viewed", 8200], ["record detail opened", 5330], ["copy selected", 4530], ["borrow confirmation opened", 2500], ["borrow completed", 2350], ]; function wilson(k, N, z = 1.96) { const p = k / N, d = 1 + z * z / N; const m = (p + z * z / (2 * N)) / d; const y = (z * Math.sqrt(p * (1 - p) / N + z * z / (4 * N * N))) / d; return [m - y, m + y]; } console.log("step entering advancing step conversion drop-off rate lost cumulative"); for (let i = 1; i < FUNNEL.length; i++) { const entering = FUNNEL[i - 1][1], advancing = FUNNEL[i][1]; const rate = advancing / entering; const [a, u] = wilson(advancing, entering); console.log( `${FUNNEL[i][0].padEnd(28)} ${String(entering).padStart(6)} ${String(advancing).padStart(8)}` + ` ${((rate * 100).toFixed(1) + " %").padStart(15)} ${(((1 - rate) * 100).toFixed(1) + " %").padStart(13)}` + ` ${String(entering - advancing).padStart(6)} ${((advancing / FUNNEL[0][1] * 100).toFixed(1) + " %").padStart(11)}` + ` [${(a * 100).toFixed(1)}, ${(u * 100).toFixed(1)}]` ); } const overall = FUNNEL[FUNNEL.length - 1][1] / FUNNEL[0][1]; console.log(`overall conversion: ${(overall * 100).toFixed(2)} %`); // Is the highest drop-off rate the same step as the largest user loss? let lowestRate = 1, lowestRateStep = "", highestLoss = 0, highestLossStep = ""; for (let i = 1; i < FUNNEL.length; i++) { const rate = FUNNEL[i][1] / FUNNEL[i - 1][1], loss = FUNNEL[i - 1][1] - FUNNEL[i][1]; if (rate < lowestRate) { lowestRate = rate; lowestRateStep = FUNNEL[i][0]; } if (loss > highestLoss) { highestLoss = loss; highestLossStep = FUNNEL[i][0]; } } console.log(`\nhighest drop-off rate: ${lowestRateStep} (${((1 - lowestRate) * 100).toFixed(1)} %)`); console.log(`largest user loss: ${highestLossStep} (${highestLoss} users)`); // Effect on overall conversion of raising every step to 0.98 (the funnel is multiplicative) console.log("\nstep current rate raised to 0.98: overall conversion relative gain"); for (let i = 1; i < FUNNEL.length; i++) { const rate = FUNNEL[i][1] / FUNNEL[i - 1][1]; const updated = overall * (0.98 / rate); console.log( `${FUNNEL[i][0].padEnd(28)} ${((rate * 100).toFixed(1) + " %").padStart(13)}` + ` ${((updated * 100).toFixed(2) + " %").padStart(32)} ${("x" + (updated / overall).toFixed(2)).padStart(13)}` ); } // An aggregated funnel can hide two segments const SEGMENTS = { "first-time visitor": [6000, 4800, 2900, 2400, 900, 820], "registered user": [4000, 3400, 2430, 2130, 1600, 1530], }; console.log("\nsegment n step conversions overall"); for (const [name, v] of Object.entries(SEGMENTS)) { const rates = []; for (let i = 1; i < v.length; i++) rates.push((v[i] / v[i - 1] * 100).toFixed(1) + "%"); console.log(`${name.padEnd(20)} ${String(v[0]).padStart(5)} ${rates.join(" ").padEnd(40)} ${(v[v.length - 1] / v[0] * 100).toFixed(2)} %`); } const combined = FUNNEL.map((_, i) => Object.values(SEGMENTS).reduce((a, v) => a + v[i], 0)); const combinedRates = []; for (let i = 1; i < combined.length; i++) combinedRates.push((combined[i] / combined[i - 1] * 100).toFixed(1) + "%"); console.log(`${"aggregated".padEnd(20)} ${String(combined[0]).padStart(5)} ${combinedRates.join(" ").padEnd(40)} ${(combined[5] / combined[0] * 100).toFixed(2)} %`); // Is the difference between the two segments at the fourth step real? const [a1, u1] = wilson(SEGMENTS["first-time visitor"][4], SEGMENTS["first-time visitor"][3]); const [a2, u2] = wilson(SEGMENTS["registered user"][4], SEGMENTS["registered user"][3]); console.log(`\ntransition to borrow confirmation first-time [${(a1 * 100).toFixed(1)}, ${(u1 * 100).toFixed(1)}] registered [${(a2 * 100).toFixed(1)}, ${(u2 * 100).toFixed(1)}] intervals ${u1 < a2 || u2 < a1 ? "disjoint" : "overlapping"}`); ``` ``` step entering advancing step conversion drop-off rate lost cumulative results list viewed 10000 8200 82.0 % 18.0 % 1800 82.0 % [81.2, 82.7] record detail opened 8200 5330 65.0 % 35.0 % 2870 53.3 % [64.0, 66.0] copy selected 5330 4530 85.0 % 15.0 % 800 45.3 % [84.0, 85.9] borrow confirmation opened 4530 2500 55.2 % 44.8 % 2030 25.0 % [53.7, 56.6] borrow completed 2500 2350 94.0 % 6.0 % 150 23.5 % [93.0, 94.9] overall conversion: 23.50 % highest drop-off rate: borrow confirmation opened (44.8 %) largest user loss: record detail opened (2870 users) step current rate raised to 0.98: overall conversion relative gain results list viewed 82.0 % 28.09 % x1.20 record detail opened 65.0 % 35.43 % x1.51 copy selected 85.0 % 27.10 % x1.15 borrow confirmation opened 55.2 % 41.73 % x1.78 borrow completed 94.0 % 24.50 % x1.04 segment n step conversions overall first-time visitor 6000 80.0% 60.4% 82.8% 37.5% 91.1% 13.67 % registered user 4000 85.0% 71.5% 87.7% 75.1% 95.6% 38.25 % aggregated 10000 82.0% 65.0% 85.0% 55.2% 94.0% 23.50 % transition to borrow confirmation first-time [35.6, 39.5] registered [73.2, 76.9] intervals disjoint ``` #### The Highest Drop-off Rate and the Largest Loss Are Not the Same Step The first table gives two different "worst steps." The step with the highest drop-off rate is the transition to borrow confirmation: 44.8 percent. The step that loses the most users is opening the record detail: 2870 users. The difference comes from the steps working with audiences of different sizes. 8200 users enter the record detail step; a 35 percent drop-off amounts to 2870 people. 4530 users enter the borrow confirmation step; the higher 44.8 percent drop-off amounts to 2030 people. The two numbers answer two different questions. The drop-off rate says **how bad the step itself is**; the number of users lost says **that step's contribution to the total loss**. If a report writes only one of them, the reader assumes the other. #### The Multiplicative Funnel and the Share of Improvement The second table determines which step to intervene on, and the answer gives a third ranking. Because overall conversion is the product of the step rates, raising a step's rate from $r$ to $r'$ multiplies overall conversion by a factor of $r'/r$. This factor depends only on that step's **current rate**; it does not depend on how many users enter the step. The table computes what happens if every step's rate is raised to 0.98. The largest relative gain comes from the borrow confirmation step: x1.78, that is, overall conversion goes from 23.50 percent to 41.73. The record detail step — the one that loses the most users — gives x1.51. The completion step, whose rate is already 94 percent, gives only x1.04. Rule: **the share of improvement lies with the lowest-rate step**, not the step that loses the most users. Intuition runs the other way; the place where the most people leave looks like the most urgent problem. The computation carries two assumptions that must be stated. First, the steps are treated as independent of one another: improving one step does not change the rates of the following steps. In reality it can — carrying more undecided users into the next step can lower that step's rate. Second, the 0.98 target was chosen as a common ceiling, not as a reachability calculation; not every step can be raised to it. The table gives a priority ranking, not a promise. #### The Aggregated Funnel Hides Segments The third table shows the funnel's most common mistake. The aggregated rate for the transition to borrow confirmation is 55.2 percent. The same step is 37.5 percent for first-time visitors and 75.1 percent for registered users. 55.2 percent describes no user segment at all; it is a number born of the mixing ratio of two different behaviors. An intervention on this step should be designed around the first-time visitor; someone looking at the aggregated number underestimates the size of the problem by half. The last line tests whether the difference is real. The 95 percent confidence interval for the first-time segment is [35.6, 39.5], and for the registered segment [73.2, 76.9]. The intervals are disjoint; the difference cannot be explained by sample fluctuation. This split connects directly to the framework from the behavior design topic. A first-time visitor meets membership registration, pickup-branch selection, and duration decisions on the borrow confirmation screen; a registered user has already made these decisions. A difference in ability is not a difference in motivation, and an intervention aimed at raising motivation will not work at this step. #### Rules for Building a Funnel **Steps are defined around the user's task.** "Page viewed" is not a step; "copy selected" is a step. A funnel built around the interface's internal events shows not where the user got stuck but what the system logged. **Drop-off rate and loss count are written together.** One gives the step's quality, the other its contribution. **Every step's rate is given with a confidence interval.** In a funnel of ten thousand, the user count shrinks quickly through the last steps; the rates at the later steps are more uncertain. **The funnel is broken out along at least one dimension.** First-time visitor versus registered user, narrow screen versus wide screen, arriving by search versus by navigation. A "worst step" found without breaking out the segments is often a mixing effect. **The funnel does not say why.** At the borrow confirmation step, 62.5 percent of first-time visitors leave; why they leave is not in this table. The funnel says which step to look at. #### Summary - The conversion funnel breaks a task into sequential steps; overall conversion is the product of the step rates. - The highest drop-off rate and the largest user loss can occur at different steps: in this computation, the borrow confirmation step has the worst rate at 44.8 percent drop-off, while the record detail step has the largest loss at 2870 users. - The share of improvement depends on the step's current rate, not its user count; raising the lowest-rate step to 0.98 moves overall conversion by a factor of x1.78, while raising the step with the largest loss moves it by x1.51. - The computation assumes the steps are independent and takes 0.98 as a common ceiling; its output is a priority ranking, not a reachable target. - The aggregated funnel hides segments: the 55.2 percent step rate is a mixture of two distinct behaviors ranging between 37.5 and 75.1 percent, and their confidence intervals are disjoint. - The funnel says which step to look at; it does not say what happens at that step. #### Next Step When the funnel shows that a step is bad, the next task is to try a change. But the measured number coming out different after the change does not mean the change worked; funnel rates move on their own from week to week. The next lesson builds this distinction: how many users are needed to test a change, when a measured difference is real, why stopping an experiment early inflates the wrong result, and what correction is required when testing several changes at once. ### A/B and Multivariate Testing Source: https://academia.sh/en/courses/user-experience/ab-and-multivariate-testing How sample size relates to the detectable difference, the confidence interval of a rate difference, how early stopping inflates the false-positive rate, and the multiple-comparison correction in a multivariate test. The previous lesson found the weakest step in the borrow flow: the transition to borrow confirmation, at 55.2 percent. The next task is to try a change. But the measured rate coming out different after the change does not mean the change worked — funnel rates move from week to week even when nothing changes. An **A/B test** is an experiment that randomly splits users into two groups, shows one the current interface and the other the changed interface, and measures the difference. This lesson takes up four questions about the experiment: how many users are needed, when a measured difference is real, why the experiment is not stopped early, and how several changes are tested at the same time? #### Four Computations ```js // experiment.mjs — sample size, confidence interval, early stopping, and multiple comparisons function generator(seed) { let s = seed >>> 0; return () => { s = (s * 1664525 + 1013904223) >>> 0; return s / 4294967296; }; } // Normal distribution functions (Abramowitz-Stegun approximation + bisection for the inverse) function normalCdf(x) { const t = 1 / (1 + 0.2316419 * Math.abs(x)); const d = 0.3989422804014327 * Math.exp(-x * x / 2); const p = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429)))); return x >= 0 ? 1 - p : p; } function z(p) { let low = -10, high = 10; for (let i = 0; i < 200; i++) { const mid = (low + high) / 2; if (normalCdf(mid) < p) low = mid; else high = mid; } return (low + high) / 2; } // ---- 1) Sample size: to detect a difference between two ratios ---- // From the funnel lesson: the current rate of the borrow confirmation step is 0.552. const BASELINE = 0.552; function sampleSize(p1, p2, alpha = 0.05, power = 0.80) { const zAlpha = z(1 - alpha / 2), zBeta = z(power); const mean = (p1 + p2) / 2; const numerator = zAlpha * Math.sqrt(2 * mean * (1 - mean)) + zBeta * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)); return Math.ceil((numerator * numerator) / ((p1 - p2) ** 2)); } console.log("detectable difference target rate users per group total"); for (const diff of [0.01, 0.02, 0.03, 0.05, 0.08]) { const n = sampleSize(BASELINE, BASELINE + diff); console.log( `${((diff * 100).toFixed(0) + " pts").padStart(22)} ${((BASELINE + diff).toFixed(3)).padStart(11)}` + ` ${String(n).padStart(22)} ${String(2 * n).padStart(7)}` ); } console.log(`(baseline rate ${BASELINE}, alpha 0.05 two-tailed, power 0.80)`); // ---- 2) Confidence interval of the rate difference ---- console.log("\ngroup A group B diff 95% confidence interval result"); const EXPERIMENTS = [ { nA: 2000, kA: 1104, nB: 2000, kB: 1160 }, { nA: 2000, kA: 1104, nB: 2000, kB: 1260 }, { nA: 8000, kA: 4416, nB: 8000, kB: 4640 }, ]; for (const d of EXPERIMENTS) { const pA = d.kA / d.nA, pB = d.kB / d.nB, diff = pB - pA; const se = Math.sqrt(pA * (1 - pA) / d.nA + pB * (1 - pB) / d.nB); const [low, high] = [diff - 1.96 * se, diff + 1.96 * se]; console.log( `${(d.kA + "/" + d.nA).padEnd(15)} ${(d.kB + "/" + d.nB).padEnd(15)} ${((diff * 100).toFixed(2) + " pts").padStart(8)}` + ` [${(low * 100).toFixed(2)}, ${(high * 100).toFixed(2)}] pts ${low > 0 || high < 0 ? "difference found" : "inconclusive"}` ); } // ---- 3) Early stopping: how often is "difference found" said when there is truly no difference ---- const RUNS = 2000, GROUP_SIZE = 8000, CHECK_INTERVAL = 500; function aaTrial(rnd, peeking) { let kA = 0, kB = 0, signaled = false; for (let i = 1; i <= GROUP_SIZE; i++) { if (rnd() < BASELINE) kA++; if (rnd() < BASELINE) kB++; if (peeking && i % CHECK_INTERVAL === 0 && i >= CHECK_INTERVAL * 2) { if (Math.abs(zValue(kA, i, kB, i)) > 1.96) { signaled = true; break; } } } if (!peeking) signaled = Math.abs(zValue(kA, GROUP_SIZE, kB, GROUP_SIZE)) > 1.96; return signaled; } function zValue(kA, nA, kB, nB) { const pA = kA / nA, pB = kB / nB, mean = (kA + kB) / (nA + nB); const se = Math.sqrt(mean * (1 - mean) * (1 / nA + 1 / nB)); return se === 0 ? 0 : (pB - pA) / se; } console.log("\nA/A experiment (both groups see the same interface, true difference is zero)"); for (const peeking of [false, true]) { const rnd = generator(20260909); let flagged = 0; for (let k = 0; k < RUNS; k++) if (aaTrial(rnd, peeking)) flagged++; console.log( `${(peeking ? `peeking every ${CHECK_INTERVAL} users` : "looking only at the end").padEnd(32)}` + ` false "difference found": ${(flagged / RUNS * 100).toFixed(1)} % (${flagged}/${RUNS})` ); } // ---- 4) Multiple comparisons: family-wise error in a multivariate test ---- console.log("\ncomparisons uncorrected family error Bonferroni alpha Sidak alpha"); for (const k of [1, 3, 7, 15]) { const familyError = 1 - Math.pow(0.95, k); console.log( `${String(k).padStart(11)} ${((familyError * 100).toFixed(1) + " %").padStart(25)}` + ` ${(0.05 / k).toFixed(4).padStart(16)} ${(1 - Math.pow(0.95, 1 / k)).toFixed(4).padStart(11)}` ); } // 2x2x2 multivariate test: 8 combinations, 7 comparisons against the control const K = 7, CORRECTED_ALPHA = 0.05 / K; console.log(`\n2x2x2 multivariate test: 8 combinations, ${K} comparisons`); console.log(`uncorrected family error ${((1 - Math.pow(0.95, K)) * 100).toFixed(1)} %, Bonferroni alpha ${CORRECTED_ALPHA.toFixed(4)}`); console.log(`users per group to detect a 2-point difference with the corrected alpha: ${sampleSize(BASELINE, BASELINE + 0.02, CORRECTED_ALPHA)}`); console.log(`total for 8 combinations: ${8 * sampleSize(BASELINE, BASELINE + 0.02, CORRECTED_ALPHA)} users`); console.log(`total in an uncorrected, single-comparison A/B test: ${2 * sampleSize(BASELINE, BASELINE + 0.02)} users`); ``` ``` detectable difference target rate users per group total 1 pts 0.562 38734 77468 2 pts 0.572 9660 19320 3 pts 0.582 4282 8564 5 pts 0.602 1532 3064 8 pts 0.632 592 1184 (baseline rate 0.552, alpha 0.05 two-tailed, power 0.80) group A group B diff 95% confidence interval result 1104/2000 1160/2000 2.80 pts [-0.27, 5.87] pts inconclusive 1104/2000 1260/2000 7.80 pts [4.76, 10.84] pts difference found 4416/8000 4640/8000 2.80 pts [1.26, 4.34] pts difference found A/A experiment (both groups see the same interface, true difference is zero) looking only at the end false "difference found": 5.7 % (114/2000) peeking every 500 users false "difference found": 20.4 % (408/2000) comparisons uncorrected family error Bonferroni alpha Sidak alpha 1 5.0 % 0.0500 0.0500 3 14.3 % 0.0167 0.0170 7 30.2 % 0.0071 0.0073 15 53.7 % 0.0033 0.0034 2x2x2 multivariate test: 8 combinations, 7 comparisons uncorrected family error 30.2 %, Bonferroni alpha 0.0071 users per group to detect a 2-point difference with the corrected alpha: 15351 total for 8 combinations: 122808 users total in an uncorrected, single-comparison A/B test: 19320 users ``` #### Sample Size Is Computed Before the Experiment The first table gives the experiment's most frequently skipped step: how many users are needed is computed **before the users arrive**. The computation needs four inputs. The baseline rate (0.552, from the funnel lesson), the smallest difference to be detected, the false-positive rate $\alpha$ (0.05), and the power (0.80 — the probability of catching a real difference if one exists). The real information the table carries is a scaling relationship: the required sample size is **inversely proportional to the square** of the detectable difference. Catching an eight-point difference needs 592 users per group; catching a one-point difference needs 38734. Cutting the difference to an eighth multiplies the sample by sixty-five. The practical consequence is that the experiment is expensive for small improvements. If the catalog interface sees five thousand borrow confirmations a week, measuring a two-point difference takes about four weeks; measuring a one-point difference takes fifteen. The experiment's duration depends on the size of the change being tested, and if that duration is not available, the experiment is not run — the change is defended on some other basis, or not at all. #### The Same Difference, Two Different Outcomes The second table shows what the question "is there a difference" depends on. The measured difference in the first and third rows is **exactly the same**: 2.80 points. In the first row, with groups of 2000, the confidence interval is [-0.27, 5.87] and includes zero; the result is inconclusive. In the third row, with groups of 8000, the interval is [1.26, 4.34] and does not include zero; there is a difference. The only difference between them is the sample size. The sentence "we measured a 2.8 percent improvement" says nothing without the sample size written alongside it. The second row gives the opposite case: a large difference of 7.80 points stays outside the interval even with groups of 2000. Large effects can be detected with a small sample; the same relationship as in the first table. The width of the confidence interval also carries decision information. In the third row, the true difference lies between 1.26 and 4.34 points; if the change's cost is only covered by a 4-point gain, the interval is not enough to make that call. "Statistically significant" and "large enough to change the decision" are not the same thing. #### Early Stopping Inflates the False-Positive Rate The third computation sets up an A/A experiment: both groups see the same interface, and the true difference is zero. A correctly working test should say "difference found" by mistake in 5 percent of cases here. Looking only at the end of the experiment, the rate comes out at 5.7 percent — close to the expected 5 percent; the difference is the simulation fluctuation of two thousand runs. When the test peeks every 500 users and stops as soon as the boundary is crossed, the rate rises to **20.4 percent**. With no true difference at all, one in five experiments says "the change worked." The reason is that the test carries a 5 percent margin of error at every look. Fifteen separate looks are fifteen separate chances to be wrong. A randomly fluctuating difference sooner or later crosses the boundary once, and if the test stops at that moment, the value that crossed it is recorded as the result. Rule: **the experiment is not stopped, and no decision is made, until it reaches the computed sample size.** Looking at the numbers while the experiment is running is not forbidden; looking and then stopping is. In situations where an interim decision is genuinely needed, sequential-testing methods that tighten the boundary according to the number of interim looks are used; an interim decision is not made with the ordinary boundary. #### The Cost of a Multivariate Test The fourth table gives another form of the same problem. When more than one comparison is made in an experiment, the probability of being wrong in at least one of them — the **family-wise error** — rises together with the number of comparisons. 30.2 percent at seven comparisons, 53.7 percent at fifteen. A **multivariate test** changes more than one element in the same experiment and compares their combinations. Three elements with two values each produce $2^3 = 8$ combinations, and seven comparisons against the control. Without correction, this means the experiment has a 30.2 percent probability of reporting at least one false "winner." Correction tightens each comparison's boundary. The Bonferroni correction divides $\alpha$ by the number of comparisons (0.0071); the Šidák correction fixes the family error directly at 0.05 (0.0073) and is slightly less strict. The last block gives the cost. Detecting a two-point difference with the corrected boundary needs 15351 users per group; 122808 users for eight combinations. Measuring the same difference with a single A/B test needs 19320 users. The multivariate test demands more than six times the traffic. This determines when to choose a multivariate test. If an interaction is expected between the elements — if one's effect depends on the other's value — separate A/B tests cannot see that interaction at all, and a multivariate test is required. If no interaction is expected, separate, sequential A/B tests are both cheaper and easier to interpret. #### Rules for the Experiment **The hypothesis and the sample size are written before the experiment.** Which metric, in which direction, and by at least how much it must change is decided in advance. A metric chosen after the fact is the hidden form of the multiple-comparison problem from the fourth table. **No decision is made before the sample target is reached.** The third computation gives the cost of not following this. **The difference is reported with its confidence interval.** A point estimate alone hides the experiment's uncertainty. **How many comparisons were made is written down.** Results broken out by segment are comparisons too; a finding of "significant on narrow screens" means as many comparisons were made as there are segments. **The experiment does not change the legitimacy criterion.** Showing that a change raises conversion does not show that the change passes the audit from the Dark Patterns and Ethics lesson. The experiment answers "does it work"; it does not answer "should it be done." #### Summary - The required sample size is inversely proportional to the square of the detectable difference: 592 users per group for an 8-point difference, 38734 for a 1-point difference. - The same measured difference (2.80 points) is inconclusive with groups of 2000 and significant with groups of 8000; a difference cannot be reported without the sample size. - The width of the confidence interval is a separate piece of decision information; being significant does not mean being large enough to change the decision. - Early stopping inflates the false-positive rate: with the true difference at zero, the test that looked only at the end said "difference found" 5.7 percent of the time, and the test that peeked every 500 users said so 20.4 percent of the time. - In a multivariate test, the family-wise error rises with the number of comparisons (30.2 percent at 7 comparisons); the Bonferroni or Šidák correction tightens the boundary and raises the required traffic to more than six times in this example. - The experiment shows whether a change works; whether it should be made is decided by a separate criterion. #### Next Step This lesson taught how to tell whether a difference is real; it did not say **why** the difference exists. If the change on the borrow confirmation screen raised the rate by 2.8 points, what users did differently does not show up in the number. The course's last lesson closes this gap: the drop-off rate at the same funnel step is placed side by side with the codes from interviews conducted at that step, it is worked out which qualitative observation matches the quantitative finding, and what to do when the two sources contradict each other is addressed. ### Combining Quantitative and Qualitative Data Source: https://academia.sh/en/courses/user-experience/combining-quantitative-and-qualitative-data Combining the two sources along the funnel-step axis, narrowing the question that arises when the rankings diverge, reporting coverage, and reading a qualitative rate with its confidence interval. The previous lesson taught how to tell whether a difference is real; it did not say why the difference exists. If the drop-off rate on the borrow form is above fifty-four percent, what users **did** on that screen does not show up in the number. The reverse is also true: in a twelve-session observation study, five participants getting stuck at the same place shows that a problem exists but does not say how many people it affects. This lesson brings the two sources together on the same axis. The axis is the funnel step: the quantitative record reports how many sessions drop off at each step, and the qualitative record reports which problem was seen in how many participants at the same step. Without a meeting point, the two sources cannot be placed side by side; matching by step is this lesson's first rule. #### Bringing the Two Records Onto the Same Axis The quantitative side consists of the entering and completing counts for the funnel steps. The qualitative side is the events coded during observation sessions: each record reports a problem one participant experienced at one step. The coding work was set up in the User Research Methods lesson; here, **which step** each code is attached to matters. The common field between the two records is the step, not the user. The participant in an observation session and the session in the field data are not the same person, and cannot be; so the combination is done at the step level, not the person level. The following computation builds this combination and compares the rankings of the two sources. ```js // combine.mjs — placing the quantitative drop-off rate side by side with qualitative observation codes at the same step // 1. Quantitative source: funnel steps and the number of sessions entering / completing each step. const FUNNEL = [ { step: "search-results", entering: 4820, completing: 3140 }, { step: "record-detail", entering: 3140, completing: 2610 }, { step: "borrow-form", entering: 2610, completing: 1180 }, { step: "authentication", entering: 1180, completing: 1044 }, { step: "confirmation", entering: 1044, completing: 1002 }, ]; // 2. Qualitative source: events coded across 12 observation sessions. // Each record reports a problem one participant experienced at one step. const CODES = [ { session: "P01", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P02", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P03", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P05", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P07", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P02", step: "borrow-form", code: "date-format-rejected" }, { session: "P06", step: "borrow-form", code: "date-format-rejected" }, { session: "P09", step: "borrow-form", code: "date-format-rejected" }, { session: "P04", step: "search-results", code: "filter-emptied-results" }, { session: "P08", step: "search-results", code: "filter-emptied-results" }, { session: "P10", step: "search-results", code: "sort-order-unexpected" }, { session: "P11", step: "record-detail", code: "shelf-code-unclear" }, { session: "P12", step: "authentication", code: "verification-code-missing" }, ]; const SESSION_COUNT = 12; // 3. Quantitative side: drop-off rate and absolute loss per step. const quant = FUNNEL.map((h) => ({ step: h.step, dropRate: (h.entering - h.completing) / h.entering, loss: h.entering - h.completing, })); console.log("step entering completing drop-off rate absolute loss"); for (const h of FUNNEL) { const n = quant.find((x) => x.step === h.step); console.log( h.step.padEnd(20) + String(h.entering).padStart(8) + String(h.completing).padStart(12) + (100 * n.dropRate).toFixed(1).padStart(12) + "%" + String(n.loss).padStart(14) ); } // 4. Qualitative side: code diversity per step and the number of participants who show each code. const codeSummary = new Map(); for (const k of CODES) { const key = k.step + "|" + k.code; if (!codeSummary.has(key)) codeSummary.set(key, new Set()); codeSummary.get(key).add(k.session); } console.log("\nstep code participants rate"); for (const [key, sessions] of [...codeSummary].sort((a, b) => b[1].size - a[1].size)) { const [step, code] = key.split("|"); console.log( step.padEnd(20) + code.padEnd(28) + String(sessions.size).padStart(6) + (" " + (100 * sessions.size / SESSION_COUNT).toFixed(0) + "%").padStart(9) ); } // 5. The two sources' rankings, side by side. const quantRank = [...quant].sort((a, b) => b.loss - a.loss).map((x) => x.step); const qualCount = new Map(FUNNEL.map((h) => [h.step, 0])); for (const k of CODES) qualCount.set(k.step, qualCount.get(k.step) + 1); const qualRank = [...qualCount].sort((a, b) => b[1] - a[1]).map(([step]) => step); console.log("\nstep quant rank qual rank diff observation count"); let matching = 0; for (const h of FUNNEL) { const n = quantRank.indexOf(h.step) + 1; const q = qualRank.indexOf(h.step) + 1; if (n === q) matching++; console.log( h.step.padEnd(20) + String(n).padStart(10) + String(q).padStart(12) + String(n - q).padStart(6) + String(qualCount.get(h.step)).padStart(15) ); } console.log(`steps landing on the same rank: ${matching} / ${FUNNEL.length}`); // 6. Coverage: how much of the largest loss can be tied to a coded cause? const largest = [...quant].sort((a, b) => b.loss - a.loss)[0]; const stepCodes = [...codeSummary].filter(([a]) => a.startsWith(largest.step + "|")); const codedParticipants = new Set(CODES.filter((k) => k.step === largest.step).map((k) => k.session)); console.log( `\nlargest loss: ${largest.step} (${largest.loss} sessions, ${(100 * largest.dropRate).toFixed(1)}%)` ); console.log(` distinct causes coded at that step: ${stepCodes.length}`); console.log(` participants who experienced the problem: ${codedParticipants.size} / ${SESSION_COUNT}`); // 7. Steps where the two sources diverge: why do the rankings not match? console.log("\nsteps where the two sources diverge"); for (const h of FUNNEL) { const n = quantRank.indexOf(h.step) + 1, q = qualRank.indexOf(h.step) + 1; if (n === q) continue; const measure = quant.find((x) => x.step === h.step); const participants = new Set(CODES.filter((k) => k.step === h.step).map((k) => k.session)).size; console.log( ` ${h.step.padEnd(18)} quant #${n} / qual #${q} ` + `drop-off ${(100 * measure.dropRate).toFixed(1)}%, loss ${measure.loss}, ` + `${participants}/${SESSION_COUNT} participants in the observation` ); } // 8. The limit of turning a qualitative count into a rate: confidence interval in a set of 12 sessions. function rateInterval(successes, n) { const z = 1.96, p = successes / n; const denom = 1 + (z * z) / n; const center = (p + (z * z) / (2 * n)) / denom; const spread = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / denom; return [center - spread, center + spread]; } console.log("\nthe limit of reading a qualitative rate as a quantitative one (12 sessions)"); for (const [key, sessions] of codeSummary) { if (sessions.size < 3) continue; const [low, high] = rateInterval(sessions.size, SESSION_COUNT); const [step, code] = key.split("|"); console.log( ` ${code.padEnd(28)} ${sessions.size}/12 = ${(100 * sessions.size / SESSION_COUNT).toFixed(0)}%` + ` 95% interval: ${(100 * low).toFixed(0)}% - ${(100 * high).toFixed(0)}%` ); } ``` ``` step entering completing drop-off rate absolute loss search-results 4820 3140 34.9% 1680 record-detail 3140 2610 16.9% 530 borrow-form 2610 1180 54.8% 1430 authentication 1180 1044 11.5% 136 confirmation 1044 1002 4.0% 42 step code participants rate borrow-form branch-selection-unclear 5 42% borrow-form date-format-rejected 3 25% search-results filter-emptied-results 2 17% search-results sort-order-unexpected 1 8% record-detail shelf-code-unclear 1 8% authentication verification-code-missing 1 8% step quant rank qual rank diff observation count search-results 1 2 -1 3 record-detail 3 3 0 1 borrow-form 2 1 1 8 authentication 4 4 0 1 confirmation 5 5 0 0 steps landing on the same rank: 3 / 5 largest loss: search-results (1680 sessions, 34.9%) distinct causes coded at that step: 2 participants who experienced the problem: 3 / 12 steps where the two sources diverge search-results quant #1 / qual #2 drop-off 34.9%, loss 1680, 3/12 participants in the observation borrow-form quant #2 / qual #1 drop-off 54.8%, loss 1430, 7/12 participants in the observation the limit of reading a qualitative rate as a quantitative one (12 sessions) branch-selection-unclear 5/12 = 42% 95% interval: 19% - 68% date-format-rejected 3/12 = 25% 95% interval: 9% - 53% ``` #### When the Rankings Diverge The third section of the output places the two rankings side by side: three of the five steps land on the same rank, two swap places. These two steps are this lesson's real subject. The search results screen is **first in absolute loss**: one thousand six hundred eighty sessions drop off there. But in the observation, only three participants experienced a problem there. The borrow form is second in absolute loss and first in the observation: seven participants got stuck there, and two distinct causes were coded. This divergence has two explanations, and both can be tested with data. The first is that the drop at the search results screen may **not be a problem**: a user who is browsing leaving the results list is expected behavior — the funnel counts it as a drop-off, but it is not a task failure. The second is that the observation study fell short at that step **because of how the task was defined**: if a participant was handed the book to search for, the difficulty of searching never arises. This table cannot decide which explanation is correct. What it can do is narrow the question: a new round of observation is needed for the search results screen, with the task definition left open. Combining quantitative and qualitative data does not produce a conclusion; it produces **the subject of the next study**. #### Coverage: How Much of the Loss Can Be Explained The fourth section counts the largest loss and the causes coded at that step. Two distinct causes were coded at the search results step, and they were seen in three of the twelve participants. This number is a coverage indicator: it measures the distance between the size of the loss and the strength of the observation that explains it. When the distance is large, two wrong decisions become possible. The first is generalizing the cause found in a small number of observations to the entire loss — the two separate problems three participants reported get counted as the cause of one thousand six hundred eighty sessions. The second is not using the observation at all and looking only at the number; then what needs fixing stays undetermined. The correct reading reports the coverage itself: "the explainable portion of the loss at this step is limited to the two causes reported by three participants." #### Reading a Qualitative Rate as a Quantitative One The last section shows a common mistake with a number. Saying "forty-two percent of users" for a problem experienced by five of twelve participants hides the sample size. The same observation's ninety-five percent confidence interval runs from nineteen to sixty-eight percent. An interval this wide can justify a design decision but cannot, on its own, set a priority order. The rule is this: a qualitative study reports **the existence and shape of a problem**, not its prevalence. The prevalence question is answered with field data, or with a measurement where the problem is turned into a metric. Five participants failing to understand branch selection is sufficient grounds for redesigning that field; the sentence "forty-two percent of users," however, should not be written. #### When the Two Sources Contradict Each Other If the drop-off rate at a step is low but more than one problem was coded in the observation, either the problem is not preventing task completion, or the field data is measuring that step incorrectly. In the opposite case — high drop-off, no observation — either the observation tasks never use that path, or the drop is not a failure. In both cases, the order to follow is the same: first the measurement itself is audited (the step definition, the moment the event is logged, the session definition), then the task definition is audited, and design is changed last. A design change made before a measurement error is fixed makes the outcome impossible to interpret. #### Summary - Quantitative and qualitative records are combined at the **step level**, not the person level; the observation participant and the session in the field data are not the same person. - When the two sources' rankings diverge, the table produces a question, not a conclusion: two of the five steps swap places, and each has two explanations that can be tested with data. - Coverage is reported: the observation explaining the largest loss comes from three of twelve participants; this distance guards against both overgeneralizing and ignoring the observation. - A qualitative study reports the existence and shape of a problem, not its prevalence: the confidence interval of a 5/12 observation runs between nineteen and sixty-eight percent. - When the two sources contradict each other, the order is to audit the measurement, audit the task definition, and change the design last. #### Course Wrap-Up This course built the chain that turns the user into an input to design. The Research and Definition topic addressed the choice of method together with participant consent and data limits, tied personas to real data, and made writing the problem statement before the solution a rule. The Information Architecture and Flow topic derived structure from content, computed navigation models through the depth-versus-breadth trade-off, and built usability testing around the limit of what a small sample can say. The Behavior Design topic modeled how a decision gets made and tied every mechanism to a single criterion: a decision is legitimate if the user would make the same choice knowing how it was made, and it is a dark pattern if its power comes from the user's ignorance. The Measurement topic taught how to report the outcome of these decisions through task success, duration, and error rate, how to read a funnel, how to build an experiment correctly, and how to combine two sources the way this lesson did. Up to this point, every decision was verified one at a time. The next question belongs to scale: how is it ensured that the same decisions stay consistent across dozens of screens, multiple teams, and time? The next course, **Design Systems**, takes up this question. It translates the design language into tokens, manages the component catalog with documentation and versioning discipline, measures the system's adoption, and builds its governance. The decisions made in this course become contracts there.