Experience Metrics

Throughout the Behavior Design topic, applying the criterion continually required one thing: knowing the user’s informed preference, the real task duration, the useful-visit rate. None of these is obtained by guessing.

This topic is devoted to measurement and begins with the most basic question: how is it shown that an interface is good? The raw record from a usability test carries three columns — whether the task was completed, how long it took, how many errors were made. This lesson takes up how those three columns are read and when they mislead.

Three Basic Metrics

Task success is the proportion of participants who complete the task. For the definition to work, “completed” must be written down in advance. If the task in the catalog interface is “find a specific volume and borrow it,” the success criterion is that the correct volume enters the borrow record; a participant who borrows the wrong volume has not completed the task.

Time on task is the time between starting the task and finishing it. The duration of failed sessions is not pooled together with that of successful sessions: a failed session is cut off at the time limit, and its real duration is unknown.

Error rate is the number of erroneous actions taken during the task. An error is any action that moves the participant away from the task: applying the wrong filter, opening the wrong record, having to go back.

The following computation processes a record of fourteen sessions using these three metrics.

// metrics.mjs — composite metric from task success, duration, and error rate

// Raw record of 14 sessions: was the task completed, duration (s), error count.
// Task: find a specific volume in the catalog and borrow it.
const SESSIONS = [
  ["P01", true, 42, 0], ["P02", true, 58, 1], ["P03", false, 180, 3],
  ["P04", true, 37, 0], ["P05", true, 91, 2], ["P06", true, 46, 0],
  ["P07", false, 180, 4], ["P08", true, 63, 1], ["P09", true, 39, 0],
  ["P10", true, 154, 2], ["P11", true, 51, 1], ["P12", false, 180, 2],
  ["P13", true, 44, 0], ["P14", true, 72, 1],
];

const n = SESSIONS.length;
const succeeded = SESSIONS.filter(([, b]) => b);
const successCount = succeeded.length;

// ---- 1) Task success and the Wilson confidence interval ----
function wilson(k, N, z = 1.96) {
  const p = k / N, d = 1 + z * z / N;
  const center = (p + z * z / (2 * N)) / d;
  const half = (z * Math.sqrt(p * (1 - p) / N + z * z / (4 * N * N))) / d;
  return [center - half, center + half];
}
const [lower, upper] = wilson(successCount, n);
console.log(`task success: ${successCount}/${n} = ${(successCount / n * 100).toFixed(1)} %`);
console.log(`Wilson 95% confidence interval: [${(lower * 100).toFixed(1)} %, ${(upper * 100).toFixed(1)} %]`);

// ---- 2) Duration: arithmetic mean vs. geometric mean ----
const durations = succeeded.map(([, , s]) => s);
const arithmetic = durations.reduce((a, b) => a + b, 0) / durations.length;
const geometric = Math.exp(durations.reduce((a, s) => a + Math.log(s), 0) / durations.length);
const sorted = [...durations].sort((a, b) => a - b);
const median = sorted.length % 2 ? sorted[(sorted.length - 1) / 2]
  : (sorted[sorted.length / 2 - 1] + sorted[sorted.length / 2]) / 2;
console.log(`\nduration of successful sessions (${durations.length} sessions): ${sorted.join(", ")}`);
console.log(`arithmetic mean ${arithmetic.toFixed(1)} s, geometric mean ${geometric.toFixed(1)} s, median ${median.toFixed(1)} s`);
console.log(`sessions above the mean: ${durations.filter((s) => s > arithmetic).length} / ${durations.length}`);

// ---- 3) Error rate ----
const totalErrors = SESSIONS.reduce((a, [, , , h]) => a + h, 0);
const errorFree = SESSIONS.filter(([, , , h]) => h === 0).length;
console.log(`\ntotal errors ${totalErrors}, per session ${(totalErrors / n).toFixed(2)}, error-free sessions ${errorFree}/${n}`);

// ---- 4) Reducing three metrics to a single number ----
// Each metric is first mapped to the 0-1 range; duration and errors are inverted.
const DURATION_TARGET = 40, DURATION_CAP = 180, ERROR_CAP = 4;
function sessionScore([, b, s, h]) {
  const success = b ? 1 : 0;
  const durationScore = b ? Math.max(0, Math.min(1, (DURATION_CAP - s) / (DURATION_CAP - DURATION_TARGET))) : 0;
  const errorScore = Math.max(0, 1 - h / ERROR_CAP);
  return { success, durationScore, errorScore, composite: 0.5 * success + 0.3 * durationScore + 0.2 * errorScore };
}
console.log("\nsession  success  duration  errors  duration score  error score  composite");
for (const o of SESSIONS) {
  const s = sessionScore(o);
  console.log(
    `${o[0].padEnd(8)} ${(o[1] ? "yes" : "no").padEnd(8)} ${String(o[2]).padStart(9)} ${String(o[3]).padStart(7)}` +
    ` ${s.durationScore.toFixed(2).padStart(14)} ${s.errorScore.toFixed(2).padStart(12)} ${s.composite.toFixed(3).padStart(10)}`
  );
}
const composites = SESSIONS.map((o) => sessionScore(o).composite);
const compositeMean = composites.reduce((a, b) => a + b, 0) / n;
const compositeStdDev = Math.sqrt(composites.reduce((a, b) => a + (b - compositeMean) ** 2, 0) / (n - 1));
console.log(`composite metric: mean ${compositeMean.toFixed(3)}, standard deviation ${compositeStdDev.toFixed(3)}, 95% interval +-${(1.96 * compositeStdDev / Math.sqrt(n)).toFixed(3)}`);

// ---- 5) Effect of weights on the outcome ----
console.log("\nweight (success/duration/error)  composite mean  lowest session");
const WEIGHTS = [[0.5, 0.3, 0.2], [1.0, 0.0, 0.0], [0.34, 0.33, 0.33], [0.2, 0.6, 0.2]];
for (const [wSuccess, wDuration, wError] of WEIGHTS) {
  const scores = SESSIONS.map((o) => {
    const s = sessionScore(o);
    return wSuccess * s.success + wDuration * s.durationScore + wError * s.errorScore;
  });
  const avg = scores.reduce((a, b) => a + b, 0) / n;
  const lowest = SESSIONS[scores.indexOf(Math.min(...scores))][0];
  console.log(`${`${wSuccess} / ${wDuration} / ${wError}`.padEnd(26)} ${avg.toFixed(3).padStart(16)} ${lowest.padStart(15)}`);
}

task success: 11/14 = 78.6 %
Wilson 95% confidence interval: [52.4 %, 92.4 %]

duration of successful sessions (11 sessions): 37, 39, 42, 44, 46, 51, 58, 63, 72, 91, 154
arithmetic mean 63.4 s, geometric mean 57.6 s, median 51.0 s
sessions above the mean: 3 / 11

total errors 17, per session 1.21, error-free sessions 5/14

session  success  duration  errors  duration score  error score  composite
P01      yes             42       0           0.99         1.00      0.996
P02      yes             58       1           0.87         0.75      0.911
P03      no             180       3           0.00         0.25      0.050
P04      yes             37       0           1.00         1.00      1.000
P05      yes             91       2           0.64         0.50      0.791
P06      yes             46       0           0.96         1.00      0.987
P07      no             180       4           0.00         0.00      0.000
P08      yes             63       1           0.84         0.75      0.901
P09      yes             39       0           1.00         1.00      1.000
P10      yes            154       2           0.19         0.50      0.656
P11      yes             51       1           0.92         0.75      0.926
P12      no             180       2           0.00         0.50      0.100
P13      yes             44       0           0.97         1.00      0.991
P14      yes             72       1           0.77         0.75      0.881
composite metric: mean 0.728, standard deviation 0.380, 95% interval +-0.199

weight (success/duration/error)  composite mean  lowest session
0.5 / 0.3 / 0.2                       0.728             P07
1 / 0 / 0                             0.786             P03
0.34 / 0.33 / 0.33                    0.712             P07
0.2 / 0.6 / 0.2                       0.688             P07

A Ratio Alone Misleads

Task success is 11/14, that is, 78.6 percent. When this number is written alone in a report, it gives an impression of precision. The Wilson confidence interval tells the truth: between 52.4 and 92.4 percent.

The interval shows what fourteen sessions do and do not determine. The true success rate could be 55 percent, or it could be 90. An interval this wide does not answer the question “is the interface good?” The question it does answer is different: does the interface work for more than half of users? To that, it says yes.

Two rules follow from this. First, a ratio is always written with its confidence interval; a ratio without an interval hides the sample size. Second, the job of a small-sample test is not to estimate a ratio but to find problems. A usability test surfaces problems with a small number of participants; the number of participants needed to measure a ratio precisely is much larger.

Using the Wilson method to compute the interval is not a minor detail. The simple interval based on the ratio’s standard error can spill outside the 0–100 percent range at small sample sizes and ratios near the extremes; the Wilson interval does not, and it is more reliable at small sample sizes.

The Duration Distribution Is Not Symmetric

The durations of the eleven successful sessions range from 37 to 154 seconds. The arithmetic mean is 63.4 seconds. Only three sessions are above this mean; eight are below.

The reason is the skew of the distribution. Task duration is bounded from below — no session can be shorter than zero seconds — but unbounded from above. If a user gets stuck, the duration doubles or triples. In distributions of this shape, the arithmetic mean is pulled upward by a few long sessions and does not represent the “typical” user.

The geometric mean is 57.6, the median 51.0 seconds. The median is the duration below which half the sessions fall, and it is not affected by a single long session. The geometric mean is the average of the logarithms of the durations; it captures the center of skewed distributions better than the arithmetic mean, and unlike the mean, it behaves consistently with the ratios between session durations.

Rule: when duration is reported, the median or the geometric mean is written; if the arithmetic mean is to be reported, the extremes of the distribution are given as well. In the catalog interface, the 154-second session is not an outlier; it is the trace of a real snag in the borrow flow, and it is not discarded — it is examined separately.

Composite Metric and the Test of Weights

When the three metrics are reported separately, comparison becomes difficult: if a design raises success but lengthens duration, which one is better? The composite metric is built to answer this question.

Building it takes three steps. Each metric is first mapped to the 0–1 range; duration and errors run in the “higher is worse” direction, so they are inverted; then they are summed with weights. For the duration score, a target of 40 seconds and a cap of 180 seconds were chosen — these numbers come from a decision, not a measurement, and they must be written in the report.

The fourth table gives the composite metric session by session. Session P10 completed the task (success 1) but took 154 seconds; its composite value is 0.656. P12 did not complete the task but made few errors; 0.100. A single number puts the information carried by the three columns into an order.

The last table is the composite metric’s most important test: does the outcome change if the weights change? The mean composite value moves between 0.688 and 0.786 — a range of 14 percentage points. That is, the absolute level of the composite metric depends on the weights and carries no meaning on its own.

One thing does not change in the same table: the lowest session is P07 in three of the four weightings. The ranking is more robust against the weights. This is the correct use of the composite metric: it is used to rank two designs or two sessions, not to measure against a threshold. A target of the form “the composite metric should be above 0.75” means the person who chose the weights is the one setting the target.

The standard deviation of the composite value is 0.380, and its 95 percent interval across the fourteen sessions is ±0.199. This width says that a composite difference smaller than 0.2 between two designs cannot be distinguished with fourteen sessions.

Reporting the Metrics Together

The raw numbers are given together with the composite. The composite metric hides which component changed; a change that lowers success but shortens duration can show up as an improvement in the composite.

Weights and thresholds are written down. The duration target, the duration cap, the error cap, and the three weights are part of the report. Without them, the composite value cannot be audited.

Failed sessions do not enter the duration pool. The duration of a session cut off at the time limit has not been measured; including it in the pool both pulls the duration distribution upward and produces an artificial cap.

A list of problems does not replace the metrics. This lesson dealt with the numbers; the numbers do not say which step is bad. That question is the subject of the next lesson.

Summary

• The task success ratio is written with its confidence interval; the 78.6 percent success rate across fourteen sessions corresponds to a range of 52.4–92.4 percent, and the ratio alone is not enough to estimate the true rate.
• The task duration distribution is right-skewed; only three of the eleven sessions fell above the arithmetic mean, so the median or the geometric mean is reported.
• The duration of failed sessions is cut off at the time limit and does not enter the duration pool.
• The composite metric maps the three metrics to the 0–1 range and sums them with weights; the target, cap, and weight values come from a decision, not a measurement, and are written in the report.
• The absolute level of the composite metric is sensitive to the weights (0.688–0.786), while its ranking is more robust; it is therefore used for comparison, not for a threshold.
• The uncertainty of the metrics is measured: the composite value’s 95 percent interval is ±0.199, meaning differences smaller than this cannot be distinguished with this sample.

Next Step

This lesson measured a task as a whole: whether it was completed, how long it took, how many errors were made. What it did not measure is where an uncompleted task got stuck. Did all three failed sessions get stuck at the same step, or at three different places? The next lesson breaks the borrow flow into steps, measures the drop-off at each step separately, finds which step accounts for the largest loss, and shows that the highest drop-off rate and the largest user loss may not be the same step.