Browser Automation

The previous lesson’s end-to-end tests drove the HTTP interface: a request was sent, a status code was read. The surface the member actually sees, though, is a page. A test that drives a page has to make two decisions, and neither is visible while writing the code — both show up only at run time: how to say which element to touch, and how to wait for that element to be ready.

This lesson does not drive a real browser. What it drives is a small tree model of the page; both decisions are actually computed over that model. What is measured is not a tool’s behavior but the two decisions themselves — which selector type breaks under which change, and how much each wait policy shifts the drop rate.

Selector here is the same word as the selector in the Visual Presentation with CSS course, but the job is different: there it says which elements a rule applies to; here it says which element a test drives.

The Page Model

The loan page carries a card: a title, the book’s status, and a borrow button. The button has three separate handles — its position, its class name, and its test id.

// page.mjs — small tree model of the loan page
export const element = (tag, attributes = {}, children = [], text = '') =>
  ({ tag, attributes, children, text });

export function buildPage(state) {
  const button = element('button', { class: 'primary', id: 'borrow-book' }, [],
    state.buttonText ?? 'Borrow');
  const card = element('div', { class: 'card', id: 'book-card' }, [
    element('h2', { class: 'title' }, [], 'Lost Books'),
    element('span', { class: 'status', id: 'book-status' }, [], state.book),
    button,
  ]);
  return { root: element('main', {}, [element('nav', {}, [], 'Catalog'), card]), button, card };
}

Four selector types search for the same button four separate ways: by position in the tree, by class name, by visible text, and by test id. The number of nodes visited is counted; this is the selector’s running cost.

// selector.mjs — four selector strategies and a visited-node counter
export const counter = { node: 0 };

function* walk(node) {
  counter.node += 1;
  yield node;
  for (const child of node.children) yield* walk(child);
}

const first = (root, predicate) => {
  for (const node of walk(root)) if (predicate(node)) return node;
  return null;
};

const path = (root, indices) => {
  let node = root;
  for (const i of indices) {
    counter.node += 1;
    node = node.children[i];
    if (node === undefined) return null;
  }
  return node;
};

export const selectors = {
  position: (root) => path(root, [1, 2]),
  class: (root) => first(root, (d) => d.attributes.class === 'primary'),
  text: (root) => first(root, (d) => d.text === 'Borrow'),
  id: (root) => first(root, (d) => d.attributes.id === 'borrow-book'),
};

Resilience to Structural Change

An interface changes as it lives, and most of those changes do not change behavior: the card gets wrapped in a layout container, the class name gets renewed, the button’s label gets corrected, the order inside the card changes, an attribute gets added for measurement. Five changes are applied one at a time, each to a page built fresh from scratch, and the element each selector finds is compared against the actual button.

// resilience.mjs — selector types against five structural changes
import { element, buildPage } from './page.mjs';
import { selectors, counter } from './selector.mjs';

const changes = {
  wrapping: (p) => { p.root.children[1] = element('div', { class: 'layout' }, [p.card]); },
  'class name': (p) => { p.button.attributes.class = 'action-primary'; },
  'label text': (p) => { p.button.text = 'Borrow Now'; },
  'card order': (p) => { p.card.children = [p.card.children[0], p.button, p.card.children[1]]; },
  'tracking attribute': (p) => { p.button.attributes.tracking = 'card-primary'; },
};

const s = (n, g) => String(n).padStart(g);
console.log(`${'selector'.padEnd(8)}${s('ok', 7)}${s('not found', 12)}${s('wrong element', 15)}${s('node', 8)}`);

for (const [name, selector] of Object.entries(selectors)) {
  const result = { ok: 0, notFound: 0, wrong: 0 };
  counter.node = 0;
  for (const apply of Object.values(changes)) {
    const page = buildPage({ book: 'shelved' });
    apply(page);
    const found = selector(page.root);
    if (found === page.button) result.ok += 1;
    else if (found === null) result.notFound += 1;
    else result.wrong += 1;
  }
  console.log(`${name.padEnd(8)}${s(result.ok, 7)}${s(result.notFound, 12)}${s(result.wrong, 15)}${s(counter.node, 8)}`);
}

selector     ok   not found  wrong element    node
position      3           1              1      10
class         4           1              0      30
text          4           1              0      30
id            5           0              0      30

Three results can be read off. The first is the ranking: the position-based selector broke on two of the five changes, the class and text selectors on one each, the test id on none. The id selector’s resilience is not a superiority but a contract — that attribute exists only for the test, and design changes never touch it.

The second is the two separate kinds of breakage. not found turns the test red and the reason is read immediately. wrong element is silent: when the order inside the card changes, the position-based selector finds the status element instead of the button, the test touches it, and nothing happens. This is the most expensive form a flaky test can take; it is as likely to stay green as it is to turn red and point at the wrong place.

The third is cost. The position-based selector touched only ten nodes across five searches, the other three touched thirty. That gap widens as the tree grows — but a thirty-node traversal is not comparable to the cost of a test that silently touches the wrong element.

Wait Policy

A page cannot be driven before it is ready, but “ready” is not a single moment: an element first enters the tree, then becomes clickable. Three policies build three separate relationships with these two moments. Sleep waits a fixed duration and tries once. Implicit wait polls for the element’s presence and returns the moment it appears. Explicit wait polls the expected condition itself — here, clickability.

The run below produces two thousand page loads from its own seeded generator; two percent of the pages load heavily, and forty percent have a gap between the element appearing and becoming clickable.

// wait.mjs — drop rate of three wait policies, seed 20
function createRng(seed) {
  let state = (seed * 2654435761) % 2147483647;
  return () => {
    state = (state * 48271) % 2147483647;
    return state / 2147483647;
  };
}

const POLL_INTERVAL = 25;
const TIMEOUT = 1000;
const FIXED_SLEEP = 250;

const generateEvent = (random) => {
  const visibleAt = random() < 0.02 ? 900 + random() * 600 : 20 + random() * 180;
  const gap = random() < 0.6 ? 0 : 10 + random() * 200;
  return { visibleAt, clickableAt: visibleAt + gap };
};

const poll = (predicate) => {
  let t = 0;
  let polls = 0;
  while (t <= TIMEOUT) {
    polls += 1;
    if (predicate(t)) return { t, polls, reached: true };
    t += POLL_INTERVAL;
  }
  return { t, polls, reached: false };
};

const policies = {
  sleep: (event) => ({ success: event.clickableAt <= FIXED_SLEEP, wait: FIXED_SLEEP, polls: 1 }),
  'implicit wait': (event) => {
    const r = poll((t) => event.visibleAt <= t);
    return { success: r.reached && event.clickableAt <= r.t, wait: r.t, polls: r.polls };
  },
  'explicit wait': (event) => {
    const r = poll((t) => event.clickableAt <= t);
    return { success: r.reached, wait: r.t, polls: r.polls };
  },
};

const RUNS = 2000;
const s = (n, g) => String(n).padStart(g);
console.log(`${'policy'.padEnd(15)}${s('drop rate', 12)}${s('avg. wait', 14)}${s('polls', 9)}`);

for (const [name, policy] of Object.entries(policies)) {
  const random = createRng(20);
  let dropped = 0;
  let wait = 0;
  let polls = 0;
  for (let i = 0; i < RUNS; i += 1) {
    const result = policy(generateEvent(random));
    if (result.success === false) dropped += 1;
    wait += result.wait;
    polls += result.polls;
  }
  console.log(`${name.padEnd(15)}${s(`%${((dropped / RUNS) * 100).toFixed(1)}`, 12)}`
    + `${s(`${(wait / RUNS).toFixed(0)} ms`, 14)}${s(polls, 9)}`);
}

policy            drop rate     avg. wait    polls
sleep                 %16.0        250 ms     2000
implicit wait         %40.9        136 ms    12878
explicit wait          %1.4        181 ms    16477

It is no accident that implicit wait gives the lowest average wait while also producing the highest drop rate: it polls the wrong condition. It returns the moment the element enters the tree, while the test wanted the element to be clickable; every time that gap opens, it produces a drop. Sleep sits at the opposite end — it pays two hundred fifty milliseconds on every run and still prevents only a sixth of the drops; wasted on a fast-loading page, insufficient on a heavily loaded one. Explicit wait returns early on a fast page, so on average it comes in cheaper than sleep, and it brings the drop rate down to a twentieth.

Explicit wait’s rate is not zero — it is 1.4 percent. The remaining drops are heavy loads that exceed the timeout. A wait policy manages flakiness, it does not eliminate it; the rising poll count is the cost of that.

The Defect Caught and the Defect Missed

What defect class does a page-driving test catch? The card’s status element not updating even though the loan request succeeded is one of them — the server returns the right response, the previous lesson’s end-to-end test stays green, and the member still sees “shelved” on screen.

// update.mjs — version 1: page state after the click
export const afterClick = (state) => ({ ...state, loaned: true });

// flow.test.mjs — loan flow driven through the page model
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { buildPage } from './page.mjs';
import { selectors } from './selector.mjs';
import { afterClick } from './update.mjs';

const textById = (root, id) => {
  if (root.attributes.id === id) return root.text;
  for (const child of root.children) {
    const found = textById(child, id);
    if (found !== null) return found;
  }
  return null;
};

test('the id selector finds the button even if the class name changes', () => {
  const page = buildPage({ book: 'shelved' });
  page.button.attributes.class = 'action-primary';

  assert.equal(selectors.id(page.root), page.button);
});

test('the card status updates after the loan is taken', () => {
  const state = { book: 'shelved' };
  const button = selectors.id(buildPage(state).root);
  assert.equal(button.tag, 'button');

  const next = buildPage(afterClick(state));

  assert.equal(textById(next.root, 'book-status'), 'loaned');
});

test('a button with the wrong label is still found and clicked', () => {
  const page = buildPage({ book: 'shelved', buttonText: 'Return' });

  const button = selectors.id(page.root);

  assert.equal(button.tag, 'button');
  assert.equal(button.attributes.id, 'borrow-book');
});

node --test --test-reporter=tap flow.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the id selector finds the button even if the class name changes
not ok 2 - the card status updates after the loan is taken
ok 3 - a button with the wrong label is still found and clicked
# tests 3
# pass 2
# fail 1

The fix is for the post-click state to also carry the book’s status.

// update.mjs — version 2: book status is also updated
export const afterClick = (state) => ({ ...state, loaned: true, book: 'loaned' });

node --test --test-reporter=tap flow.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the id selector finds the button even if the class name changes
ok 2 - the card status updates after the loan is taken
ok 3 - a button with the wrong label is still found and clicked
# tests 3
# pass 3
# fail 0

The third test shows the class that is missed, and it is green in both runs: even if the button’s label is “Return”, the test finds it, touches it, and the flow completes. A selector bound to the test id never looks at the label at all. So the cost of the selector’s resilience is that a defect in the label text falls outside this test’s scope; unless that check is written separately, a mislabeled button ships with a green run.

On the cost side, these three tests took about 33 ms in this run; duration depends on the machine and grows by an order of magnitude once a real browser is driven. The run-independent numbers stand in the measurements themselves: the tree driven has six nodes, the id selector visited thirty nodes across five searches, and the two-thousand-load run with explicit wait makes 16,477 polls.

Summary

• A test that drives a page makes two decisions: which handle to find the element by, and how to wait for it to be ready.
• Across five structural changes, the position-based selector broke twice, the class and text selectors once each; the test id never broke.
• The two kinds of breakage carry separate costs: not-found turns the test red, a wrong-element match silently touches the wrong place.
• Across two thousand loads, sleep gave a 16.0 percent drop rate, implicit wait 40.9 percent, explicit wait 1.4 percent; implicit wait’s cheapness comes from polling the wrong condition.
• The page test caught the card-not-updating defect and missed the mislabeled button; a selector bound to the id never looks at the label.

Next Step

All the measurements here were over a single page model: one tree, one timeline. The question changes once the same interface runs on a handful of devices, several operating system versions, and different screen widths. The question is no longer which selector is resilient but which devices the test runs on, because the product of device and version grows the number of runs far faster than the number of pages. The next lesson measures how fast that product grows, what share of users the selected subset covers, and the risk of a defect falling outside that coverage.