Mobile Automation

All the previous lesson’s measurements were over a single page model: one tree, one timeline. The borrowing flow’s mobile interface, though, does not run in one place. The same screen opens on a handful of device classes and several operating system versions, and the test’s question is no longer “which selector is resilient” but “which devices do I run on.”

This lesson does not set up a real device or emulator. What is measured is not the device itself but the selection decision: how fast the matrix grows, how many users the selected subset represents, and what the defect class left out of coverage is. All three are countable.

The Matrix and Two Selection Criteria

The model’s input is written out explicitly. MB1 — six device classes and five operating system versions; the shares are taken as measured from member sessions, and each axis’s shares sum to one hundred; the two axes are assumed independent. The independence assumption does not hold in real deployments (old versions concentrate on old devices); there, the cell share is measured directly and the computation below works the same way.

// matrix.mjs — device-version matrix and two subset selections
export const devices = [
  ['small', 22], ['medium', 26], ['large', 18], ['foldable', 6], ['tablet', 12], ['legacy', 16],
];
export const versions = [['S-6', 4], ['S-7', 9], ['S-8', 17], ['S-9', 31], ['S-10', 39]];

export const cells = devices.flatMap(([device, dp]) =>
  versions.map(([version, vp]) => ({ device, version, share: (dp * vp) / 100 })));

export const byShare = (k) => [...cells].sort((a, b) => b.share - a.share).slice(0, k);

export function byCoverage(k) {
  const selected = [];
  const seen = { device: new Set(), version: new Set() };
  while (selected.length < k) {
    const remaining = cells.filter((h) => selected.includes(h) === false);
    const gain = (h) => (seen.device.has(h.device) ? 0 : 1) + (seen.version.has(h.version) ? 0 : 1);
    remaining.sort((a, b) => gain(b) - gain(a) || b.share - a.share);
    const pick = remaining[0];
    seen.device.add(pick.device);
    seen.version.add(pick.version);
    selected.push(pick);
  }
  return selected;
}

export const subsets = { byShare: byShare(6), byCoverage: byCoverage(6) };

The two criteria optimize different things. Selection by share takes the cells with the most users. Selection by coverage prefers, at every step, the cell that gains a device class or version not yet seen, and falls back to share on a tie. Six cells are selected each way, and the catch is counted for three defect classes: a version-dependent defect is caught if that version is in the subset, a device-class-dependent defect if that class is in it, and a defect specific to a single cell only if that exact cell was selected.

// coverage.mjs — user share and defect-catch rate of two selections
import { devices, versions, cells, subsets } from './matrix.mjs';

const rate = (selected) => {
  const device = new Set(selected.map((h) => h.device));
  const version = new Set(selected.map((h) => h.version));
  return {
    share: selected.reduce((t, h) => t + h.share, 0),
    versionDefect: `${version.size}/${versions.length}`,
    deviceDefect: `${device.size}/${devices.length}`,
    cellDefect: `${selected.length}/${cells.length}`,
  };
};

const s = (n, g) => String(n).padStart(g);
console.log(`${'selection'.padEnd(16)}${s('cell', 6)}${s('user share', 16)}${s('version defect', 16)}${s('device defect', 15)}${s('cell defect', 13)}`);
for (const [name, selected] of [['by share', subsets.byShare], ['by coverage', subsets.byCoverage]]) {
  const r = rate(selected);
  console.log(`${name.padEnd(16)}${s(selected.length, 6)}${s(`%${r.share.toFixed(1)}`, 16)}${s(r.versionDefect, 16)}${s(r.deviceDefect, 15)}${s(r.cellDefect, 13)}`);
}
console.log(`matrix ${devices.length} devices x ${versions.length} versions = ${cells.length} cells;`
  + ` adding a version gives ${cells.length + devices.length}, adding a device gives ${cells.length + versions.length}`);
console.log('selected by share:', subsets.byShare.map((h) => `${h.device}/${h.version}`).join(' '));
console.log('selected by coverage:', subsets.byCoverage.map((h) => `${h.device}/${h.version}`).join(' '));

selection         cell      user share  version defect  device defect  cell defect
by share             6           %46.9             2/5            4/6         6/30
by coverage          6           %24.3             5/5            6/6         6/30
matrix 6 devices x 5 versions = 30 cells; adding a version gives 36, adding a device gives 35
selected by share: medium/S-10 small/S-10 medium/S-9 large/S-10 small/S-9 legacy/S-10
selected by coverage: medium/S-10 small/S-9 large/S-8 legacy/S-7 tablet/S-6 foldable/S-10

The numbers expose the trade-off. Selection by share represents 46.9 percent of users but touches only two of the five versions and four of the six device classes; three of its six selected cells are on the same version. Selection by coverage cuts the represented share in half, but in exchange it enters every version and every device class once. On the cell-specific defect the two are equal, and both are weak: six of thirty cells, one in five.

The matrix’s growth reopens this decision every year. When a new version ships, the matrix goes from thirty to thirty-six; when a new device class arrives, to thirty-five; if the subset stays at six cells, cell coverage drops from a fifth to a sixth. The run budget grows linearly while the matrix grows multiplicatively, and that is why the question for a mobile run can never be “run everything.”

The Dimension an Emulator Carries

The coverage table carries an assumption: that the selected cell really runs on that device class. An emulator meets only half of that assumption. An emulator genuinely runs the operating system version — it carries version-dependent defects. What it does not carry is anything tied to the device class: the touch surface’s real size, the memory limit, network interruption, the camera and sensors, the manufacturer’s interface layer.

This arithmetic feeds straight into the table. If all six cells selected by coverage run on an emulator, the device-class-dependent catch drops from six of six to zero; the version-dependent catch stays at five of five. If two cells move to real hardware in a device farm, the device catch rises to two of six. So the decision is not “emulator or real device”; it is which defect class to look for in which cell.

The Defect It Catches: Version-Dependent Parsing

At the end of the borrowing flow, the member is shown a due date. The application is the side that produces the date text; the platform’s date parser is the side that reads it, and that parser’s behavior varies by version: older versions accept only the ordered format.

// platform.mjs — date-parsing behavior that varies by version
const WIDE_PARSER_VERSIONS = new Set(['S-9', 'S-10']);

export function parseDate(version, text) {
  if (/^\d{4}-\d{2}-\d{2}$/.test(text)) return text;
  if (WIDE_PARSER_VERSIONS.has(version) && /^\d{2}\.\d{2}\.\d{4}$/.test(text)) {
    const [day, month, year] = text.split('.');
    return `${year}-${month}-${day}`;
  }
  return null;
}

// duedate.mjs — version 1: due date produced in a local format
export function dueDate(loanDate, days = 14) {
  const t = new Date(`${loanDate}T00:00:00Z`);
  t.setUTCDate(t.getUTCDate() + days);
  const two = (n) => String(n).padStart(2, '0');
  return `${two(t.getUTCDate())}.${two(t.getUTCMonth() + 1)}.${t.getUTCFullYear()}`;
}

The same flow is run over both subsets; for each cell, whether the produced date is readable on that version is tested.

// device.test.mjs — due-date flow runs across the selected subsets
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { subsets } from './matrix.mjs';
import { parseDate } from './platform.mjs';
import { dueDate } from './duedate.mjs';

const run = (selected) => {
  for (const cell of selected) {
    const text = dueDate('2026-03-04');
    assert.equal(parseDate(cell.version, text), '2026-03-18',
      `${cell.device}/${cell.version} could not read the due date: ${text}`);
  }
};

test('the due date is read across the subset selected by share', () => run(subsets.byShare));

test('the due date is read across the subset selected by coverage', () => run(subsets.byCoverage));

node --test --test-reporter=tap device.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the due date is read across the subset selected by share
not ok 2 - the due date is read across the subset selected by coverage
# tests 2
# pass 1
# fail 1

The defect is one, the subsets are two, and the result splits in two. The subset selected by coverage turned red because it carries S-6, S-7, and S-8 cells. The subset selected by share stayed green with the same defect: it holds only S-9 and S-10, and both accept the local format. The class missed here is not one a test form misses but one a selection misses, and the missed users can be counted too: the three versions from S-6 through S-8 are thirty percent of the version axis.

The fix is to produce the date text in a platform-independent format.

// duedate.mjs — version 2: due date produced in a platform-independent format
export function dueDate(loanDate, days = 14) {
  const t = new Date(`${loanDate}T00:00:00Z`);
  t.setUTCDate(t.getUTCDate() + days);
  return t.toISOString().slice(0, 10);
}

node --test --test-reporter=tap device.test.mjs | grep -E '^ *(ok|not ok|# (tests|pass|fail))'

ok 1 - the due date is read across the subset selected by share
ok 2 - the due date is read across the subset selected by coverage
# tests 2
# pass 2
# fail 0

There is also a class both subsets miss: a defect seen in only a single cell — the interface layer of a specific device class on a specific version. Its counterpart in the table is six of thirty; four-fifths of cell-specific defects fall outside coverage under both selections, and that ratio does not improve without enlarging the subset.

The run-dependent side of the cost is small here — the two tests took about 33 ms, because what runs here is a model. In a real run, the number that sets the cost is in the table: twelve cell runs, six device sessions for each of the two subsets. Each session means one setup, one app install, and one teardown; if the subset is expanded to all thirty cells, that number multiplies by five, and duration grows in the same proportion.

Summary

• The device and version axes multiply: six devices and five versions make thirty cells; a new version adds six, a new device class adds five.
• The six cells selected by share represented 46.9 percent of users but touched two of five versions and four of six device classes.
• The six cells selected by coverage cut the share to 24.3 percent, but in exchange entered every version and every device class.
• Only the subset selected by coverage caught the version-dependent date-parsing defect; the subset selected by share stayed green with the same defect.
• An emulator carries the version dimension, not the device-class dimension; if all six cells run on an emulator, device defect catches drop from six of six to zero.

Next Step

Up to this point, every claim was about a value: a status code, a text, a date. The interface’s real output, though, is an image, and asking “are they equal” about it suddenly becomes hard — two screenshots are never made of exactly the same pixels. The next lesson ties this comparison to a threshold and measures it by scanning that threshold: how many false alarms it produces as the threshold drops, and how many real regressions it lets through as the threshold rises.