---
title: 'Visual Regression Testing'
source: 'https://academia.sh/en/courses/frontend-quality/visual-regression-testing'
course: 'Frontend Quality'
language: en
updated: '2026-08-17T18:11:06+00:00'
license: 'CC BY-SA 4.0'
---

# Visual Regression Testing

Pixel-level comparison of the baseline image against the candidate image, the separate roles of pixel tolerance and ratio threshold, the masking decision, and the determinism conditions that make the comparison meaningful.

A flow test verifies that the measurement row appears; it does not verify how it appears.
When a style rule is applied to the wrong container, a spacing value doubles, or the save
button becomes the same color as the background, the tests from the previous two lessons
keep passing. The element is in the tree, its role is in place, it can be clicked — but it
is wrong on screen.

Visual regression testing fills this gap: it compares a screenshot against an accepted
baseline and counts the difference.

## What Gets Compared

There are three images. The **baseline image** is the version previously reviewed and
approved, and it is kept in the repository. The **candidate image** is the version produced
from the current code. The **diff image** marks the pixels where the two disagree.

Comparison can be done at three scales. Full-page comparison gives the widest coverage but
is the noisiest: a change anywhere on the page fails the test. Component comparison captures
only the frame of the relevant component and ties the defect to a narrower area.
Element-level comparison is the narrowest scope.

A separate method from this is the **snapshot test**: the rendered tree is dumped into text
form and saved. It catches a change in structure, not a change in appearance — color,
spacing, and font decisions are not visible in the tree. The two cover different risks.

## The Diff Computation

The core of the comparison is a short computation: walk the corresponding pixels of two
grids, compare the difference against a tolerance, and count the ones that disagree.

```js
// image-diff.mjs — compares two grayscale grids
export function diff(baseline, candidate, { tolerance = 0, mask = [] } = {}) {
  if (baseline.width !== candidate.width || baseline.height !== candidate.height) {
    return { sizeMismatch: true, diffPixels: NaN, ratio: NaN };
  }
  const masked = (x, y) =>
    mask.some((m) => x >= m.x && x < m.x + m.w && y >= m.y && y < m.y + m.h);
  let diffCount = 0;
  let counted = 0;
  for (let y = 0; y < baseline.height; y += 1) {
    for (let x = 0; x < baseline.width; x += 1) {
      if (masked(x, y)) continue;
      counted += 1;
      const i = y * baseline.width + x;
      if (Math.abs(baseline.pixels[i] - candidate.pixels[i]) > tolerance) diffCount += 1;
    }
  }
  return { sizeMismatch: false, diffPixels: diffCount, ratio: diffCount / counted };
}

export function decision(result, threshold) {
  if (result.sizeMismatch) return 'failed (size mismatch)';
  return result.ratio > threshold ? 'failed' : 'passed';
}
```

Testing this does not need a real screenshot; a forty-column grid that deterministically
draws the measurement table screen is enough. The following module places the header bar,
the clock field, four measurement rows, and the save button in shades of gray.

```js
// screen.mjs — deterministic grayscale rendering of the measurement table screen
const WIDTH = 40;
const HEIGHT = 20;

function blank() {
  return { width: WIDTH, height: HEIGHT, pixels: new Uint8Array(WIDTH * HEIGHT).fill(255) };
}

function box(g, x, y, w, h, tone) {
  for (let py = y; py < y + h; py += 1) {
    for (let px = x; px < x + w; px += 1) {
      if (px >= 0 && px < g.width && py >= 0 && py < g.height) g.pixels[py * g.width + px] = tone;
    }
  }
}

export function screen({ buttonTone = 60, clockPattern = 0, shift = 0 } = {}) {
  const g = blank();
  box(g, 0, 0 + shift, 40, 2, 200);               // header bar
  box(g, 30, 0 + shift, 8, 1, 120 + clockPattern); // clock field
  for (let s = 0; s < 4; s += 1) {                 // measurement rows
    box(g, 2, 4 + s * 3 + shift, 34, 2, 230);
    box(g, 3, 5 + s * 3 + shift, 10, 1, 90);
  }
  box(g, 26, 17 + shift, 12, 2, buttonTone);       // save button
  return g;
}

// a small deviation resembling the rendering engine's anti-aliasing
export function addNoise(g, seed = 12345, amplitude = 3) {
  const copy = { ...g, pixels: Uint8Array.from(g.pixels) };
  let s = seed;
  for (let i = 0; i < copy.pixels.length; i += 1) {
    s = (s * 1103515245 + 12345) % 2147483648;
    const deviation = (s % (2 * amplitude + 1)) - amplitude;
    copy.pixels[i] = Math.min(255, Math.max(0, copy.pixels[i] + deviation));
  }
  return copy;
}
```

```js
// report.mjs — comparing candidates against the baseline
import { diff, decision } from './image-diff.mjs';
import { screen, addNoise } from './screen.mjs';

const baseline = screen();
const THRESHOLD = 0.005;
const CLOCK_MASK = [{ x: 30, y: 0, w: 8, h: 1 }];

const candidates = [
  ['identical rendering', screen(), {}],
  ['anti-aliasing deviation', addNoise(screen()), {}],
  ['same deviation, tolerance 5', addNoise(screen()), { tolerance: 5 }],
  ['clock field changed', screen({ clockPattern: 40 }), {}],
  ['clock field masked', screen({ clockPattern: 40 }), { mask: CLOCK_MASK }],
  ['button color changed', screen({ buttonTone: 30 }), {}],
  ['content shifted 1 pixel', screen({ shift: 1 }), {}],
];

console.log('status                          diff  ratio  decision');
for (const [label, candidate, options] of candidates) {
  const result = diff(baseline, candidate, options);
  const ratio = (result.ratio * 100).toFixed(2).padStart(6);
  console.log(`${label.padEnd(29)} ${String(result.diffPixels).padStart(6)} ${ratio}%  ${decision(result, THRESHOLD)}`);
}
```

```sh
node report.mjs
```

```
status                          diff  ratio  decision
identical rendering                0   0.00%  passed
anti-aliasing deviation          502  62.75%  failed
same deviation, tolerance 5        0   0.00%  passed
clock field changed                8   1.00%  failed
clock field masked                 0   0.00%  passed
button color changed              24   3.00%  failed
content shifted 1 pixel          424  53.00%  failed
```

Seven rows contain every decision point of the method.

## Tolerance, Threshold, and Mask

The second row shows why zero tolerance cannot be used. When a deviation of at most three
units was added to every pixel, five hundred two of the eight hundred pixels were counted as
"different": 62.75%. A deviation invisible to the eye rendered the entire comparison
meaningless. In the third row, the same image was compared with a tolerance of five units,
and the difference dropped to zero.

Two separate clamps follow from this, and the two should not be confused. **Pixel tolerance**
says how far a single pixel is allowed to deviate; it absorbs the rendering engine's
anti-aliasing noise. **Ratio threshold** says how many pixels are allowed to deviate; it is
the total accepted amount of change. Tolerance guards against noise, threshold against small
and inconsequential shifts.

The fourth and fifth rows show dynamic content. The clock field in the header carries a
different value on every run and changes eight pixels: one percent. This is a difference
unrelated to the code, and it is excluded from the comparison with a mask. The mask
rectangle drops out of the count entirely; the ratio is computed over the remaining pixels.

The sixth row is a real regression: the save button's tone changed, twenty-four pixels
differ, three percent. Because it is above the threshold, the test fails.

The seventh row shows the method's most important limitation. When the content shifted one
pixel down, four hundred twenty-four pixels changed — 53%. Visual comparison is **not
local**: a one-unit change in a container's margin shifts everything beneath it and spreads
the difference across the whole page. This is why the diff ratio does not measure the size
of the defect; it only says "something changed". Finding the location of the defect requires
looking at the diff image.

The last detail is the size check. If the images have different dimensions, pixel comparison
is undefined; the comparison is invalidated before it even begins.

## Putting It Into a Test

Once the decision criterion is settled, it turns into a test.

```js
// visual.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { diff, decision } from './image-diff.mjs';
import { screen, addNoise } from './screen.mjs';

const THRESHOLD = 0.005;
const CRITERIA = { tolerance: 5, mask: [{ x: 30, y: 0, w: 8, h: 1 }] };

test('rendering deviation and clock field are not counted as a regression', () => {
  const candidate = addNoise(screen({ clockPattern: 40 }));
  assert.equal(decision(diff(screen(), candidate, CRITERIA), THRESHOLD), 'passed');
});

test('a change in the button tone is reported as a regression', () => {
  const result = diff(screen(), screen({ buttonTone: 30 }), CRITERIA);
  assert.equal(decision(result, THRESHOLD), 'failed');
  assert.equal(result.diffPixels, 24);
});

test('a size mismatch invalidates the comparison', () => {
  const narrow = { width: 39, height: 20, pixels: new Uint8Array(39 * 20) };
  assert.equal(decision(diff(screen(), narrow, CRITERIA), THRESHOLD), 'failed (size mismatch)');
});
```

```sh
node --test visual.test.mjs
```

```
✔ rendering deviation and clock field are not counted as a regression (0.825291ms)
✔ a change in the button tone is reported as a regression (0.121292ms)
✔ a size mismatch invalidates the comparison (0.04775ms)
ℹ tests 3
ℹ suites 0
ℹ pass 3
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 30.034792
```

## Determinism Conditions

Visual comparison only works when the candidate image is produced deterministically. Five
conditions must be met.

The viewport and pixel ratio are fixed; a different width produces a different layout.
Animation and transitions are stopped, otherwise the image depends on which frame it was
captured at — the reduced motion preference from the Layout Systems and Responsive Design
course can also be used for this purpose. Waiting happens until the font has loaded; a frame
drawn with the fallback font changes the width of every line. The clock, random identifiers,
and live data are fixed or masked. The scroll position is reset.

There is one more condition, and it comes from outside: the rendering engine and the
operating system. The same code renders anti-aliasing differently in a different
environment. This is why baseline images are tied to the environment that produced them, and
are produced and compared in a single environment.

The baselines themselves are an asset that must be managed. They sit in the repository as
binary files, and, as described in the Introduction to Version Control course, a binary
file's diff cannot be read. Every accepted change grows the repository. This is why a
baseline update is never automatic: a person sees the change, approves it, and the new
baseline is committed with that approval. An unapproved update turns the regression test
into a mechanism that justifies itself.

## Summary

- Visual regression testing compares the baseline and candidate images at the pixel level; a
  snapshot test, by contrast, captures structure, not appearance.
- Pixel tolerance limits a single pixel's deviation, ratio threshold limits the total amount
  of change; the two guard against different problems and should not be confused.
- Dynamic areas are masked; a masked region falls entirely outside the count.
- The diff ratio is not local: a one-pixel shift made half the page appear different, which
  is why the ratio does not measure the size of the defect.
- The comparison is meaningful only once the viewport, animation, fonts, clock, and scroll
  are fixed; baselines are tied to the environment that produced them, and their updates
  require approval.

## Next Step

All the tests built across four lessons rested on one assumption: the same code gives the
same result. In practice, this assumption breaks often. A test passes in nineteen of twenty
runs and fails in one; no one changed anything. The team first tries rerunning it, then
starts ignoring that test, and the signal the suite gives becomes worthless. The next lesson
makes the causes of this behavior — timing, order dependence, and shared state — countable,
and shows the fix for each.