Compatibility Testing

The previous audit ran on a single document model and gave a single result. Yet the same lending screen does not produce the same tree for every reader: different rendering engines support the same declaration differently, different screen widths build different layouts. Every measurement so far was made in a single environment, and the user share that environment represents was never asked.

Compatibility testing opens the product of these axes into a matrix and asks a single question: which subset of this matrix will run? The Integration, Contract and End-to-End Testing course’s device–version matrix built the same arithmetic over device and operating system version; the axis of the matrix here is not the device but the feature support profile, and the selection criterion derives not from the user share but from how distinct the profiles are.

The Matrix and Support Profiles

Three rendering engines, two platforms, and two screen classes give twelve environments. The support for six features in each environment is a vector; environments carrying the same vector are indistinguishable from a feature standpoint.

NF14 (assumption): the environment shares have been measured and the matrix is complete. An environment absent from the list is counted as nonexistent; every coverage ratio is relative to these twelve rows.

// matrix.mjs — the compatibility matrix, feature support profiles, and known defects
export const FEATURE = ["persistent storage", "transition animation", "grid layout",
  "date formatter", "observer", "input suggestion"];

const ENGINE = { a: [1, 1, 1, 1, 1, 1], b: [1, 0, 0, 1, 1, 0], c: [1, 1, 1, 1, 0, 1] };

// On the mobile platform, the date formatter (feature 3) is absent in every engine.
export const profile = (o) =>
  ENGINE[o.engine].map((v, i) => (i === 3 && o.platform === "mobile" ? 0 : v)).join("");

// Twelve environments with a measured user share.
export const ENVIRONMENT = [
  { engine: "a", platform: "desktop", screen: "wide", share: 42 },
  { engine: "a", platform: "mobile", screen: "narrow", share: 26 },
  { engine: "b", platform: "desktop", screen: "wide", share: 14 },
  { engine: "c", platform: "mobile", screen: "narrow", share: 9 },
  { engine: "b", platform: "mobile", screen: "narrow", share: 3 },
  { engine: "c", platform: "desktop", screen: "wide", share: 2 },
  { engine: "a", platform: "desktop", screen: "narrow", share: 1.5 },
  { engine: "a", platform: "mobile", screen: "wide", share: 1 },
  { engine: "b", platform: "desktop", screen: "narrow", share: 0.7 },
  { engine: "c", platform: "desktop", screen: "narrow", share: 0.4 },
  { engine: "b", platform: "mobile", screen: "wide", share: 0.3 },
  { engine: "c", platform: "mobile", screen: "wide", share: 0.1 },
].map((o) => ({ ...o, name: `${o.engine}-${o.platform}-${o.screen}` }));

// Manually verified defects and the environment condition they appear under.
export const DEFECT = [
  ["K1 grid layout does not render", (o) => o.engine === "b"],
  ["K2 date format returns empty", (o) => o.platform === "mobile"],
  ["K3 list does not refresh without the observer", (o) => o.engine === "c"],
  ["K4 desktop layout overflows on a narrow screen", (o) => o.platform === "desktop" && o.screen === "narrow"],
  ["K5 queue client cannot connect on mobile", (o) => o.engine === "b" && o.platform === "mobile"],
  ["K6 mobile layout scatters on a wide screen", (o) => o.platform === "mobile" && o.screen === "wide"],
];

// Robust behaviors; strict verification would turn these red too.
export const ROBUST = [
  ["Y1 transition animation is skipped", (o) => o.engine === "b"],
  ["Y2 menu collapses on a narrow screen", (o) => o.screen === "narrow"],
];

export const found = (selection, list) => list.filter(([, k]) => selection.some(k)).length;
export const red = (selection, list) => selection.filter((o) => list.some(([, k]) => k(o))).length;

Three Selection Criteria

// selection.mjs — the matrix's size, distinguishing profiles, and three selection criteria
import { ENVIRONMENT, FEATURE, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs";

const profiles = [...new Set(ENVIRONMENT.map(profile))];
console.log(`${ENVIRONMENT.length} environments, ${FEATURE.length} features, ${profiles.length} distinguishing feature support profiles`);
for (const p of profiles) {
  const matching = ENVIRONMENT.filter((o) => profile(o) === p);
  console.log(`  ${p}  ${String(matching.reduce((t, o) => t + o.share, 0).toFixed(1)).padStart(5)}%  ${matching.map((o) => o.name).join(", ")}`);
}

// (a) By share: from largest to smallest, until the cumulative share reaches the threshold.
const byShare = (threshold) => {
  const s = [];
  for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); }
  return s;
};
// (b) By profile: the environment with the largest share from each support profile.
const byProfile = profiles.map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]);
// (c) Profile and layout: the missing platform-screen pairs are added to (b).
const pair = (o) => `${o.platform}-${o.screen}`;
const profileAndLayout = [...byProfile];
for (const o of ENVIRONMENT) if (!profileAndLayout.some((x) => pair(x) === pair(o))) profileAndLayout.push(o);

console.log(`\n${"selection".padEnd(20)}${"runs".padStart(6)}${"share".padStart(7)}${"profile".padStart(9)}${"defects".padStart(9)}${"red runs".padStart(15)}`);
for (const [name, s] of [["by share 90%", byShare(90)], ["by profile", byProfile],
  ["profile and layout", profileAndLayout], ["full matrix", ENVIRONMENT]]) {
  console.log(`${name.padEnd(20)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(7)}` +
    `${`${new Set(s.map(profile)).size}/${profiles.length}`.padStart(9)}${`${found(s, DEFECT)}/${DEFECT.length}`.padStart(9)}` +
    `${String(red(s, ROBUST)).padStart(15)}`);
}

const missed = DEFECT.filter(([, k]) => !byShare(90).some(k));
console.log(`\nwhat the by-share selection misses:`);
for (const [name] of missed) console.log(`  ${name}`);

12 environments, 6 features, 6 distinguishing feature support profiles
  111111   43.5%  a-desktop-wide, a-desktop-narrow
  111011   27.0%  a-mobile-narrow, a-mobile-wide
  100110   14.7%  b-desktop-wide, b-desktop-narrow
  111001    9.1%  c-mobile-narrow, c-mobile-wide
  100010    3.3%  b-mobile-narrow, b-mobile-wide
  111101    2.4%  c-desktop-wide, c-desktop-narrow

selection             runs  share  profile  defects       red runs
by share 90%             4  91.0%      4/6      3/6              3
by profile               6  96.0%      6/6      4/6              4
profile and layout       8  98.5%      6/6      6/6              5
full matrix             12 100.0%      6/6      6/6              8

what the by-share selection misses:
  K4 desktop layout overflows on a narrow screen
  K5 queue client cannot connect on mobile
  K6 mobile layout scatters on a wide screen

Twelve environments come down to six profiles: the matrix is half the size from a feature-support standpoint. Selecting by share covers 91% of users with four runs, but it sees only four of the six profiles and misses three of the six defects. What the missed ones share is telling: all three arise from the combination of axes. The desktop layout overflowing on a narrow screen has nothing to do with any engine’s support; it has to do with a value the two axes take together.

Selecting by profile is therefore not enough: it covers every support profile with six runs, but it still cannot see the layout defects, because the support vector never carries the screen class. The third selection, eight runs, adds the missing platform–screen pairs and finds all six of the six defects — with as many runs as two-thirds of the full matrix.

Threshold Scanning

The common rule is written as “cover this much of the users.” What this threshold actually buys can be scanned.

// threshold.mjs — scanning the covered-share threshold; counts false pass and false fail
import { ENVIRONMENT, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs";

const byShare = (threshold) => {
  const s = [];
  for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); }
  return s;
};
const profiles = new Set(ENVIRONMENT.map(profile)).size;

console.log(`${"share threshold".padEnd(17)}${"runs".padStart(6)}${"covered".padStart(10)}${"profile".padStart(9)}` +
  `${"false pass".padStart(14)}${"false fail".padStart(14)}${"cost w=5".padStart(12)}`);
for (const e of [80, 90, 95, 99, 100]) {
  const s = byShare(e);
  const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST);
  console.log(`${`${e}%`.padEnd(17)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(10)}` +
    `${`${new Set(s.map(profile)).size}/${profiles}`.padStart(9)}${String(pass).padStart(14)}${String(fail).padStart(14)}` +
    `${String(fail + 5 * pass).padStart(12)}`);
}

// The set selected by the profile-and-layout criterion is placed in the same table.
const pair = (o) => `${o.platform}-${o.screen}`;
const s = [...new Set(ENVIRONMENT.map(profile))].map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]);
for (const o of ENVIRONMENT) if (!s.some((x) => pair(x) === pair(o))) s.push(o);
const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST);
console.log(`\nprofile-and-layout criterion: ${s.length} runs, false pass ${pass}, false fail ${fail},` +
  ` cost ${fail + 5 * pass}`);
console.log(`runs needed to catch the same cost by share: ${byShare(99).length}`);

share threshold    runs   covered  profile    false pass    false fail    cost w=5
80%                   3     82.0%      3/6             4             2          22
90%                   4     91.0%      4/6             3             3          18
95%                   6     96.0%      6/6             2             4          14
99%                   9     99.2%      6/6             0             6           6
100%                 12    100.0%      6/6             0             8           8

profile-and-layout criterion: 8 runs, false pass 0, false fail 5, cost 5
runs needed to catch the same cost by share: 9

False pass is a real defect sitting in an environment left outside the selection; false fail is strict verification turning red in an environment that is robust but behaves differently. The two move in opposite directions: as environments are added, missed defects decrease, and runs that turn red for nothing increase. Even the full matrix is not the best option — two of the eight red runs are unavoidable noise anyway.

The real finding is in the last two rows. Selecting by share can find every defect only with nine runs; the profile-and-layout criterion gives the same result with eight runs, at a lower cost. The threshold’s source here is not a measurement but a misconception: covered user share is an intuitive number, but it has nothing to do with how defects are distributed. Defects are distributed by environment class, not by user share, and the selection criterion turns out more decisive than the threshold.

Who owns the decision: every environment left outside the matrix is a known gap. The release decision is tied not to the sentence “all runs are green” but to “this many profiles and this many layout pairs were run”; the classes that were not run go on record.

The Cost of Testing

The run-independent cost lives in three numbers. The first is the run count: eight environments, each demanding its own setup, its own data, and its own time; this number grows multiplicatively as axes are added — adding one value to one axis does not add to the matrix, it multiplies it.

The second is maintaining the matrix: the support vectors are written by hand, and the profile count changes once an engine starts supporting a feature. A support table that is not updated either runs two environments it treats as distinguishable for nothing, or never runs one of two environments it fails to treat as distinct.

The third is sifting through red runs: five of the eight runs turned red, and two of them were not real defects. This is compatibility testing’s permanent burden — not every environment difference is a defect, and what separates a difference from a defect is a decision, not a rule.

Summary

• The compatibility matrix is the product of its axes; twelve environments came down to six distinguishing support profiles over six features, and the matrix shrank by half from a feature standpoint.
• Selecting by share covered 91% of users with four runs but missed three of the six defects; all three of the misses arose not from a single axis but from the combination of axes.
• Selecting by profile was not enough either, because the support profile does not carry the screen class; the profile-and-layout criterion found all six of the six defects with eight runs — as many as two-thirds of the full matrix.
• Threshold scanning moved the two columns in opposite directions: as environments were added, false pass decreased and false fail increased. Even the full matrix was not the lowest-cost option.
• Defects are distributed by environment class, not by user share; the selection criterion is more decisive than the covered-share threshold.
• The cost is multiplicative: every new axis value grows the matrix by a factor, the support table demands manual upkeep, and a portion of red runs gets sifted through again on every run.

Next Step

This topic’s six lessons tested the same system with six separate methods, and all six shared a common assumption: the system’s parts are working. Static scanning looked at the code’s text, dynamic testing sent requests to the running service, dependency scanning matched installed versions against advisories, penetration testing questioned the combination of rules, the accessibility audit read the rendered tree, compatibility testing ran the same screen in different environments. Both load and attack were pressure coming from outside, and in both cases the system itself stayed up. Yet what happens when the database the lending service depends on stops responding, when the queue client loses a message, or when the catalog service returns a half response, was tested in no lesson. The next topic moves the pressure from outside to inside: a fault is deliberately triggered in one of the system’s own parts, and how the system meets that fault is measured.