--- title: 'Combining Quantitative and Qualitative Data' source: 'https://academia.sh/en/courses/user-experience/combining-quantitative-and-qualitative-data' course: 'User Experience and Behavior Design' language: en updated: '2026-08-19T05:19:59+00:00' license: 'CC BY-SA 4.0' --- # Combining Quantitative and Qualitative Data Combining the two sources along the funnel-step axis, narrowing the question that arises when the rankings diverge, reporting coverage, and reading a qualitative rate with its confidence interval. The previous lesson taught how to tell whether a difference is real; it did not say why the difference exists. If the drop-off rate on the borrow form is above fifty-four percent, what users **did** on that screen does not show up in the number. The reverse is also true: in a twelve-session observation study, five participants getting stuck at the same place shows that a problem exists but does not say how many people it affects. This lesson brings the two sources together on the same axis. The axis is the funnel step: the quantitative record reports how many sessions drop off at each step, and the qualitative record reports which problem was seen in how many participants at the same step. Without a meeting point, the two sources cannot be placed side by side; matching by step is this lesson's first rule. ## Bringing the Two Records Onto the Same Axis The quantitative side consists of the entering and completing counts for the funnel steps. The qualitative side is the events coded during observation sessions: each record reports a problem one participant experienced at one step. The coding work was set up in the User Research Methods lesson; here, **which step** each code is attached to matters. The common field between the two records is the step, not the user. The participant in an observation session and the session in the field data are not the same person, and cannot be; so the combination is done at the step level, not the person level. The following computation builds this combination and compares the rankings of the two sources. ```js // combine.mjs — placing the quantitative drop-off rate side by side with qualitative observation codes at the same step // 1. Quantitative source: funnel steps and the number of sessions entering / completing each step. const FUNNEL = [ { step: "search-results", entering: 4820, completing: 3140 }, { step: "record-detail", entering: 3140, completing: 2610 }, { step: "borrow-form", entering: 2610, completing: 1180 }, { step: "authentication", entering: 1180, completing: 1044 }, { step: "confirmation", entering: 1044, completing: 1002 }, ]; // 2. Qualitative source: events coded across 12 observation sessions. // Each record reports a problem one participant experienced at one step. const CODES = [ { session: "P01", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P02", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P03", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P05", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P07", step: "borrow-form", code: "branch-selection-unclear" }, { session: "P02", step: "borrow-form", code: "date-format-rejected" }, { session: "P06", step: "borrow-form", code: "date-format-rejected" }, { session: "P09", step: "borrow-form", code: "date-format-rejected" }, { session: "P04", step: "search-results", code: "filter-emptied-results" }, { session: "P08", step: "search-results", code: "filter-emptied-results" }, { session: "P10", step: "search-results", code: "sort-order-unexpected" }, { session: "P11", step: "record-detail", code: "shelf-code-unclear" }, { session: "P12", step: "authentication", code: "verification-code-missing" }, ]; const SESSION_COUNT = 12; // 3. Quantitative side: drop-off rate and absolute loss per step. const quant = FUNNEL.map((h) => ({ step: h.step, dropRate: (h.entering - h.completing) / h.entering, loss: h.entering - h.completing, })); console.log("step entering completing drop-off rate absolute loss"); for (const h of FUNNEL) { const n = quant.find((x) => x.step === h.step); console.log( h.step.padEnd(20) + String(h.entering).padStart(8) + String(h.completing).padStart(12) + (100 * n.dropRate).toFixed(1).padStart(12) + "%" + String(n.loss).padStart(14) ); } // 4. Qualitative side: code diversity per step and the number of participants who show each code. const codeSummary = new Map(); for (const k of CODES) { const key = k.step + "|" + k.code; if (!codeSummary.has(key)) codeSummary.set(key, new Set()); codeSummary.get(key).add(k.session); } console.log("\nstep code participants rate"); for (const [key, sessions] of [...codeSummary].sort((a, b) => b[1].size - a[1].size)) { const [step, code] = key.split("|"); console.log( step.padEnd(20) + code.padEnd(28) + String(sessions.size).padStart(6) + (" " + (100 * sessions.size / SESSION_COUNT).toFixed(0) + "%").padStart(9) ); } // 5. The two sources' rankings, side by side. const quantRank = [...quant].sort((a, b) => b.loss - a.loss).map((x) => x.step); const qualCount = new Map(FUNNEL.map((h) => [h.step, 0])); for (const k of CODES) qualCount.set(k.step, qualCount.get(k.step) + 1); const qualRank = [...qualCount].sort((a, b) => b[1] - a[1]).map(([step]) => step); console.log("\nstep quant rank qual rank diff observation count"); let matching = 0; for (const h of FUNNEL) { const n = quantRank.indexOf(h.step) + 1; const q = qualRank.indexOf(h.step) + 1; if (n === q) matching++; console.log( h.step.padEnd(20) + String(n).padStart(10) + String(q).padStart(12) + String(n - q).padStart(6) + String(qualCount.get(h.step)).padStart(15) ); } console.log(`steps landing on the same rank: ${matching} / ${FUNNEL.length}`); // 6. Coverage: how much of the largest loss can be tied to a coded cause? const largest = [...quant].sort((a, b) => b.loss - a.loss)[0]; const stepCodes = [...codeSummary].filter(([a]) => a.startsWith(largest.step + "|")); const codedParticipants = new Set(CODES.filter((k) => k.step === largest.step).map((k) => k.session)); console.log( `\nlargest loss: ${largest.step} (${largest.loss} sessions, ${(100 * largest.dropRate).toFixed(1)}%)` ); console.log(` distinct causes coded at that step: ${stepCodes.length}`); console.log(` participants who experienced the problem: ${codedParticipants.size} / ${SESSION_COUNT}`); // 7. Steps where the two sources diverge: why do the rankings not match? console.log("\nsteps where the two sources diverge"); for (const h of FUNNEL) { const n = quantRank.indexOf(h.step) + 1, q = qualRank.indexOf(h.step) + 1; if (n === q) continue; const measure = quant.find((x) => x.step === h.step); const participants = new Set(CODES.filter((k) => k.step === h.step).map((k) => k.session)).size; console.log( ` ${h.step.padEnd(18)} quant #${n} / qual #${q} ` + `drop-off ${(100 * measure.dropRate).toFixed(1)}%, loss ${measure.loss}, ` + `${participants}/${SESSION_COUNT} participants in the observation` ); } // 8. The limit of turning a qualitative count into a rate: confidence interval in a set of 12 sessions. function rateInterval(successes, n) { const z = 1.96, p = successes / n; const denom = 1 + (z * z) / n; const center = (p + (z * z) / (2 * n)) / denom; const spread = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / denom; return [center - spread, center + spread]; } console.log("\nthe limit of reading a qualitative rate as a quantitative one (12 sessions)"); for (const [key, sessions] of codeSummary) { if (sessions.size < 3) continue; const [low, high] = rateInterval(sessions.size, SESSION_COUNT); const [step, code] = key.split("|"); console.log( ` ${code.padEnd(28)} ${sessions.size}/12 = ${(100 * sessions.size / SESSION_COUNT).toFixed(0)}%` + ` 95% interval: ${(100 * low).toFixed(0)}% - ${(100 * high).toFixed(0)}%` ); } ``` ``` step entering completing drop-off rate absolute loss search-results 4820 3140 34.9% 1680 record-detail 3140 2610 16.9% 530 borrow-form 2610 1180 54.8% 1430 authentication 1180 1044 11.5% 136 confirmation 1044 1002 4.0% 42 step code participants rate borrow-form branch-selection-unclear 5 42% borrow-form date-format-rejected 3 25% search-results filter-emptied-results 2 17% search-results sort-order-unexpected 1 8% record-detail shelf-code-unclear 1 8% authentication verification-code-missing 1 8% step quant rank qual rank diff observation count search-results 1 2 -1 3 record-detail 3 3 0 1 borrow-form 2 1 1 8 authentication 4 4 0 1 confirmation 5 5 0 0 steps landing on the same rank: 3 / 5 largest loss: search-results (1680 sessions, 34.9%) distinct causes coded at that step: 2 participants who experienced the problem: 3 / 12 steps where the two sources diverge search-results quant #1 / qual #2 drop-off 34.9%, loss 1680, 3/12 participants in the observation borrow-form quant #2 / qual #1 drop-off 54.8%, loss 1430, 7/12 participants in the observation the limit of reading a qualitative rate as a quantitative one (12 sessions) branch-selection-unclear 5/12 = 42% 95% interval: 19% - 68% date-format-rejected 3/12 = 25% 95% interval: 9% - 53% ``` ## When the Rankings Diverge The third section of the output places the two rankings side by side: three of the five steps land on the same rank, two swap places. These two steps are this lesson's real subject. The search results screen is **first in absolute loss**: one thousand six hundred eighty sessions drop off there. But in the observation, only three participants experienced a problem there. The borrow form is second in absolute loss and first in the observation: seven participants got stuck there, and two distinct causes were coded. This divergence has two explanations, and both can be tested with data. The first is that the drop at the search results screen may **not be a problem**: a user who is browsing leaving the results list is expected behavior — the funnel counts it as a drop-off, but it is not a task failure. The second is that the observation study fell short at that step **because of how the task was defined**: if a participant was handed the book to search for, the difficulty of searching never arises. This table cannot decide which explanation is correct. What it can do is narrow the question: a new round of observation is needed for the search results screen, with the task definition left open. Combining quantitative and qualitative data does not produce a conclusion; it produces **the subject of the next study**. ## Coverage: How Much of the Loss Can Be Explained The fourth section counts the largest loss and the causes coded at that step. Two distinct causes were coded at the search results step, and they were seen in three of the twelve participants. This number is a coverage indicator: it measures the distance between the size of the loss and the strength of the observation that explains it. When the distance is large, two wrong decisions become possible. The first is generalizing the cause found in a small number of observations to the entire loss — three participants' filter problem gets counted as the cause of one thousand six hundred eighty sessions. The second is not using the observation at all and looking only at the number; then what needs fixing stays undetermined. The correct reading reports the coverage itself: "the explainable portion of the loss at this step is limited to the two causes reported by three participants." ## Reading a Qualitative Rate as a Quantitative One The last section shows a common mistake with a number. Saying "forty-two percent of users" for a problem experienced by five of twelve participants hides the sample size. The same observation's ninety-five percent confidence interval runs from nineteen to sixty-eight percent. An interval this wide can justify a design decision but cannot, on its own, set a priority order. The rule is this: a qualitative study reports **the existence and shape of a problem**, not its prevalence. The prevalence question is answered with field data, or with a measurement where the problem is turned into a metric. Five participants failing to understand branch selection is sufficient grounds for redesigning that field; the sentence "forty-two percent of users," however, should not be written. ## When the Two Sources Contradict Each Other If the drop-off rate at a step is low but more than one problem was coded in the observation, either the problem is not preventing task completion, or the field data is measuring that step incorrectly. In the opposite case — high drop-off, no observation — either the observation tasks never use that path, or the drop is not a failure. In both cases, the order to follow is the same: first the measurement itself is audited (the step definition, the moment the event is logged, the session definition), then the task definition is audited, and design is changed last. A design change made before a measurement error is fixed makes the outcome impossible to interpret. ## Summary - Quantitative and qualitative records are combined at the **step level**, not the person level; the observation participant and the session in the field data are not the same person. - When the two sources' rankings diverge, the table produces a question, not a conclusion: two of the five steps swap places, and each has two explanations that can be tested with data. - Coverage is reported: the observation explaining the largest loss comes from three of twelve participants; this distance guards against both overgeneralizing and ignoring the observation. - A qualitative study reports the existence and shape of a problem, not its prevalence: the confidence interval of a 5/12 observation runs between nineteen and sixty-eight percent. - When the two sources contradict each other, the order is to audit the measurement, audit the task definition, and change the design last. ## Course Wrap-Up This course built the chain that turns the user into an input to design. The Research and Definition topic addressed the choice of method together with participant consent and data limits, tied personas to real data, and made writing the problem statement before the solution a rule. The Information Architecture and Flow topic derived structure from content, computed navigation models through the depth-versus-breadth trade-off, and built usability testing around the limit of what a small sample can say. The Behavior Design topic modeled how a decision gets made and tied every mechanism to a single criterion: a decision is legitimate if the user would make the same choice knowing how it was made, and it is a dark pattern if its power comes from the user's ignorance. The Measurement topic taught how to report the outcome of these decisions through task success, duration, and error rate, how to read a funnel, how to build an experiment correctly, and how to combine two sources the way this lesson did. Up to this point, every decision was verified one at a time. The next question belongs to scale: how is it ensured that the same decisions stay consistent across dozens of screens, multiple teams, and time? The next course, **Design Systems**, takes up this question. It translates the design language into tokens, manages the component catalog with documentation and versioning discipline, measures the system's adoption, and builds its governance. The decisions made in this course become contracts there.