Visual Verification

Up to this point, every claim was about a value: a status code, a text, a date. The interface’s real output, though, is an image, and a button’s shadow disappearing, a drop in contrast, or a card shifting by a pixel shows up in no value claim. Visual regression testing wants to fill that gap: a screenshot is compared against an accepted baseline image.

The question looks like an equality question, and it is not. Two screenshots are never made of the same pixels; font antialiasing, subpixel placement, and regions of the screen that change produce a difference on every run. So the comparison is tied to a threshold, and the threshold choice directly sets two rates: a false alarm (turning red when there is no regression) and a miss (staying green when there is one).

This lesson does not take screenshots. The comparison runs over a real pixel array; the screen is a small model of the loan card.

The Pixel Model of the Screen

Forty columns, twenty rows, a single tone channel. The card draws three things: the title bar, the book’s status text, and the borrow button. There is also a region at the top right that changes on every run — the remaining-time indicator.

// image.mjs — small pixel model of the card screen and a comparison
export const WIDTH = 40;
export const HEIGHT = 20;

const rect = (p, x0, y0, x1, y1, tone) => {
  for (let y = y0; y <= y1; y += 1) for (let x = x0; x <= x1; x += 1) p[y * WIDTH + x] = tone;
};

export function draw(d) {
  const p = new Uint8Array(WIDTH * HEIGHT).fill(240);
  rect(p, 2, 2, 30 + (d.titleShift ?? 0), 4, 90);
  rect(p, 2, 7, 1 + (d.statusWidth ?? 12), 8, 120);
  if (d.buttonVisible !== false) {
    const k = d.shift ?? 0;
    rect(p, 2 + k, 12, 16 + k, 16, d.buttonTone ?? 30);
  }
  if (d.roundedCorners) for (const [x, y] of [[2, 12], [16, 12], [2, 16], [16, 16]]) p[y * WIDTH + x] = 240;
  rect(p, 32, 0, 39, 1, d.clockTone ?? 40);
  return p;
}

export const excludedRegion = () => {
  const m = new Uint8Array(WIDTH * HEIGHT);
  for (let y = 0; y <= 1; y += 1) for (let x = 32; x < WIDTH; x += 1) m[y * WIDTH + x] = 1;
  return m;
};

export const compare = (a, b, pixelThreshold = 12, excluded = null) => {
  let different = 0;
  let counted = 0;
  for (let i = 0; i < a.length; i += 1) {
    if (excluded && excluded[i] === 1) continue;
    counted += 1;
    if (Math.abs(a[i] - b[i]) > pixelThreshold) different += 1;
  }
  return { different, rate: (different / counted) * 100 };
};

The comparison has two thresholds, and they must not be confused. The pixel threshold says how much tone difference in a single pixel counts as negligible; here, twelve. The ratio threshold says how many differing pixels count as a regression, and that is what actually decides the outcome. The scan below holds the pixel threshold fixed and scans the ratio threshold.

Threshold Scan

The scan produces two sets. The first set has no regression: the same accepted state is redrawn with a changing time indicator and, with a one-in-ten probability, a title shifted by a pixel. The second set has five real regressions. For every threshold, how many of the first set trigger an alarm and how many of the second set are missed are counted. The variable region at the top right is left in the comparison once and excluded from it once, giving two measurements.

// threshold.mjs — threshold scan: false-alarm and miss rates, seed 31
import { draw, compare, excludedRegion } from './image.mjs';

function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const ACCEPTED = { buttonVisible: true, buttonTone: 30, statusWidth: 12, clockTone: 40 };
const baseline = draw(ACCEPTED);

const regressions = {
  'no button': { buttonVisible: false },
  'low contrast': { buttonTone: 200 },
  'button shift': { shift: 1 },
  'status shortened': { statusWidth: 8 },
  'rounded corners': { roundedCorners: true },
};

const thresholds = [0, 0.25, 0.5, 1, 2, 5, 10];
const RUNS = 200;

const equivalent = (random) => draw({
  ...ACCEPTED,
  clockTone: 60 + Math.floor(random() * 8) * 20,
  titleShift: random() < 0.1 ? 1 : 0,
});

const measure = (excluded) => {
  const falseAlarm = thresholds.map(() => 0);
  const missed = thresholds.map(() => 0);
  const random = createRng(31);
  for (let i = 0; i < RUNS; i += 1) {
    const rate = compare(baseline, equivalent(random), 12, excluded).rate;
    thresholds.forEach((e, j) => { if (rate > e) falseAlarm[j] += 1; });
  }
  const regressionNames = Object.keys(regressions);
  for (const name of regressionNames) {
    for (let i = 0; i < RUNS / regressionNames.length; i += 1) {
      const broken = draw({
        ...ACCEPTED, ...regressions[name], clockTone: 60 + Math.floor(random() * 8) * 20,
      });
      const rate = compare(baseline, broken, 12, excluded).rate;
      thresholds.forEach((e, j) => { if (rate <= e) missed[j] += 1; });
    }
  }
  return { falseAlarm, missed };
};

const unexcluded = measure(null);
const excluded = measure(excludedRegion());
const s = (n, g) => String(n).padStart(g);
const pct = (n) => `%${((n / RUNS) * 100).toFixed(1)}`;
console.log(`${'ratio threshold'.padEnd(16)}${s('false alarm', 14)}${s('missed', 9)}`
  + `${s('excl. alarm', 17)}${s('excl. missed', 19)}`);
thresholds.forEach((e, j) => {
  console.log(`${`%${e.toFixed(2)}`.padEnd(16)}${s(pct(unexcluded.falseAlarm[j]), 14)}`
    + `${s(pct(unexcluded.missed[j]), 9)}${s(pct(excluded.falseAlarm[j]), 17)}`
    + `${s(pct(excluded.missed[j]), 19)}`);
});
for (const [name, diff] of Object.entries(regressions)) {
  const rate = compare(baseline, draw({ ...ACCEPTED, ...diff }), 12, excludedRegion()).rate;
  console.log(`${name.padEnd(18)} excluded diff rate %${rate.toFixed(2)}`);
}

ratio threshold    false alarm   missed      excl. alarm       excl. missed
%0.00                   %100.0     %0.0             %5.5               %0.0
%0.25                   %100.0     %0.0             %5.5               %0.0
%0.50                   %100.0     %0.0             %0.0               %0.0
%1.00                   %100.0     %0.0             %0.0              %20.0
%2.00                     %5.5     %0.0             %0.0              %60.0
%5.00                     %0.0    %60.0             %0.0              %60.0
%10.00                    %0.0    %60.0             %0.0             %100.0
no button          excluded diff rate %9.57
low contrast       excluded diff rate %9.57
button shift       excluded diff rate %1.28
status shortened   excluded diff rate %1.02
rounded corners    excluded diff rate %0.51

The first two columns show that no threshold works while the variable region stays in. The time indicator alone is sixteen pixels, or two percent of the screen; so every threshold below two percent alarms on every run, and every threshold above it swallows every small regression. A two-percent threshold happens to look good by chance — zero misses, 5.5 percent alarms — but that is the threshold settling onto the size of the noise: once the indicator grows by one digit, the threshold has to be retuned.

Once the variable region is excluded from the comparison, the table becomes readable. At the half-percent threshold, false alarms drop to zero and all five regressions are caught. As the threshold rises, misses open up in order: at one percent, rounded corners; at two percent, the status text and the button shift; at ten percent, even the button disappearing entirely. The last lines give the reason for this order — the difference rate each regression produces is fixed, and the moment the threshold passes it, the defect becomes invisible.

The rule that follows is this: the threshold is kept below the smallest real regression that could be missed; if the noise exceeds that threshold, the threshold is not raised — the source of the noise is removed from the comparison. Excluding variable regions is the cost of keeping the threshold low, and every excluded region also means that any regression that could occur there goes unseen.

The Defect Caught and the Defect Missed

The class visual verification catches is a defect that shows up in no value claim. The button tone lightening at the style layer is one of these: the tree is the same, the selector works, the flow completes, and the button is too faint to read.

// style.mjs — version 1: button tone lightened
export const style = { buttonTone: 200 };

// visual.test.mjs — card screen compared against the baseline image
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { draw, compare, excludedRegion } from './image.mjs';
import { style } from './style.mjs';

const ACCEPTED = { buttonVisible: true, buttonTone: 30, statusWidth: 12, clockTone: 40 };
const baseline = draw(ACCEPTED);
const excluded = excludedRegion();
const THRESHOLD = 0.5;

test('the card screen matches the baseline image', () => {
  const now = draw({ ...ACCEPTED, buttonTone: style.buttonTone, clockTone: 120 });

  const { different, rate } = compare(baseline, now, 12, excluded);

  assert.ok(rate <= THRESHOLD, `${different} pixels different, rate %${rate.toFixed(2)}`);
});

test('an inactive button produces no difference in the image', () => {
  const active = draw({ ...ACCEPTED, buttonTone: style.buttonTone, active: true, clockTone: 120 });
  const inactive = draw({ ...ACCEPTED, buttonTone: style.buttonTone, active: false, clockTone: 200 });

  assert.equal(compare(active, inactive, 12, excluded).different, 0);
});

node --test --test-reporter=tap visual.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

not ok 1 - the card screen matches the baseline image
ok 2 - an inactive button produces no difference in the image
# tests 2
# pass 1
# fail 1

The second test sets up the class that is missed, and it is green in both runs: the button being active or inactive produces the same pixels, because inactivity has no visual counterpart in the model. Visual verification asks whether what is seen is correct; it does not ask what happens when it is touched. If the borrow button looks exactly right and sends no request at all, this test stays green, and that defect is caught only by the flow tests from the earlier lessons.

The fix is to return the tone to the baseline image’s value.

// style.mjs — version 2: button tone restored to the baseline value
export const style = { buttonTone: 30 };

node --test --test-reporter=tap visual.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the card screen matches the baseline image
ok 2 - an inactive button produces no difference in the image
# tests 2
# pass 2
# fail 0

The run-dependent side of the cost is small here: the two tests took about 37 ms in this run, the threshold scan ran eight hundred comparisons, and each comparison read 784 pixels. The run-independent cost is the number of baseline images. If the borrowing flow’s four screens are kept across the previous lesson’s six cells selected by coverage, that stores twenty-four baseline images. A design decision that deliberately changes the button’s tone requires re-approving all twenty-four of them — visual verification’s real cost is not the run but this approval work.

Summary

• Visual verification compares a screenshot against an accepted baseline image, and the decision is tied to two thresholds: the pixel tone threshold and the differing-pixel ratio threshold.
• While the variable region stayed in the comparison, every threshold below two percent gave a 100 percent false-alarm rate, and every threshold above it missed the small regressions.
• Once the variable region was excluded, the half-percent threshold caught all five regressions with zero false alarms.
• As the threshold rises, misses open up in the order of the difference rate each regression produces: rounded corners at one percent, a one-pixel shift at two percent, the button disappearing at ten percent.
• Visual verification asks whether what is seen is correct; an inactive button produces the same pixels, so it passes this test green.

Next Step

The last three lessons saw the same fact from separate angles: wait policy shifted the drop rate, device selection shifted defect coverage, threshold choice shifted the false-alarm rate. All three are pieces of a single problem — end-to-end tests carry an instability that the lower-level tests do not carry. The next lesson takes on that instability in aggregate: what proportion the causes of drops are distributed across in an end-to-end team, and how much cleaning the signal with quarantine raises the rate of escaped defects.