--- title: 'Compatibility Testing' source: 'https://academia.sh/en/courses/non-functional-testing/compatibility-testing' course: 'Non-Functional Testing' language: en updated: '2026-08-23T14:25:17+00:00' license: 'CC BY-SA 4.0' --- # Compatibility Testing Measuring a matrix that grows with the product of the rendering engine, platform, and screen axes: reducing feature support profiles to distinct classes, the defect gap between selecting by share and selecting by profile, and how the selection criterion turns out more decisive than the threshold. The previous audit ran on a single document model and gave a single result. Yet the same lending screen does not produce the same tree for every reader: different rendering engines support the same declaration differently, different screen widths build different layouts. Every measurement so far was made in a single environment, and the user share that environment represents was never asked. **Compatibility testing** opens the product of these axes into a matrix and asks a single question: which subset of this matrix will run? The Integration, Contract and End-to-End Testing course's device–version matrix built the same arithmetic over device and operating system version; the axis of the matrix here is not the device but the **feature support profile**, and the selection criterion derives not from the user share but from how distinct the profiles are. ## The Matrix and Support Profiles Three rendering engines, two platforms, and two screen classes give twelve environments. The support for six features in each environment is a vector; environments carrying the same vector are **indistinguishable** from a feature standpoint. **NF14 (assumption):** the environment shares have been measured and the matrix is complete. An environment absent from the list is counted as nonexistent; every coverage ratio is relative to these twelve rows. ```js // matrix.mjs — the compatibility matrix, feature support profiles, and known defects export const FEATURE = ["persistent storage", "transition animation", "grid layout", "date formatter", "observer", "input suggestion"]; const ENGINE = { a: [1, 1, 1, 1, 1, 1], b: [1, 0, 0, 1, 1, 0], c: [1, 1, 1, 1, 0, 1] }; // On the mobile platform, the date formatter (feature 3) is absent in every engine. export const profile = (o) => ENGINE[o.engine].map((v, i) => (i === 3 && o.platform === "mobile" ? 0 : v)).join(""); // Twelve environments with a measured user share. export const ENVIRONMENT = [ { engine: "a", platform: "desktop", screen: "wide", share: 42 }, { engine: "a", platform: "mobile", screen: "narrow", share: 26 }, { engine: "b", platform: "desktop", screen: "wide", share: 14 }, { engine: "c", platform: "mobile", screen: "narrow", share: 9 }, { engine: "b", platform: "mobile", screen: "narrow", share: 3 }, { engine: "c", platform: "desktop", screen: "wide", share: 2 }, { engine: "a", platform: "desktop", screen: "narrow", share: 1.5 }, { engine: "a", platform: "mobile", screen: "wide", share: 1 }, { engine: "b", platform: "desktop", screen: "narrow", share: 0.7 }, { engine: "c", platform: "desktop", screen: "narrow", share: 0.4 }, { engine: "b", platform: "mobile", screen: "wide", share: 0.3 }, { engine: "c", platform: "mobile", screen: "wide", share: 0.1 }, ].map((o) => ({ ...o, name: `${o.engine}-${o.platform}-${o.screen}` })); // Manually verified defects and the environment condition they appear under. export const DEFECT = [ ["K1 grid layout does not render", (o) => o.engine === "b"], ["K2 date format returns empty", (o) => o.platform === "mobile"], ["K3 list does not refresh without the observer", (o) => o.engine === "c"], ["K4 desktop layout overflows on a narrow screen", (o) => o.platform === "desktop" && o.screen === "narrow"], ["K5 queue client cannot connect on mobile", (o) => o.engine === "b" && o.platform === "mobile"], ["K6 mobile layout scatters on a wide screen", (o) => o.platform === "mobile" && o.screen === "wide"], ]; // Robust behaviors; strict verification would turn these red too. export const ROBUST = [ ["Y1 transition animation is skipped", (o) => o.engine === "b"], ["Y2 menu collapses on a narrow screen", (o) => o.screen === "narrow"], ]; export const found = (selection, list) => list.filter(([, k]) => selection.some(k)).length; export const red = (selection, list) => selection.filter((o) => list.some(([, k]) => k(o))).length; ``` ## Three Selection Criteria ```js // selection.mjs — the matrix's size, distinguishing profiles, and three selection criteria import { ENVIRONMENT, FEATURE, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs"; const profiles = [...new Set(ENVIRONMENT.map(profile))]; console.log(`${ENVIRONMENT.length} environments, ${FEATURE.length} features, ${profiles.length} distinguishing feature support profiles`); for (const p of profiles) { const matching = ENVIRONMENT.filter((o) => profile(o) === p); console.log(` ${p} ${String(matching.reduce((t, o) => t + o.share, 0).toFixed(1)).padStart(5)}% ${matching.map((o) => o.name).join(", ")}`); } // (a) By share: from largest to smallest, until the cumulative share reaches the threshold. const byShare = (threshold) => { const s = []; for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); } return s; }; // (b) By profile: the environment with the largest share from each support profile. const byProfile = profiles.map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]); // (c) Profile and layout: the missing platform-screen pairs are added to (b). const pair = (o) => `${o.platform}-${o.screen}`; const profileAndLayout = [...byProfile]; for (const o of ENVIRONMENT) if (!profileAndLayout.some((x) => pair(x) === pair(o))) profileAndLayout.push(o); console.log(`\n${"selection".padEnd(20)}${"runs".padStart(6)}${"share".padStart(7)}${"profile".padStart(9)}${"defects".padStart(9)}${"red runs".padStart(15)}`); for (const [name, s] of [["by share 90%", byShare(90)], ["by profile", byProfile], ["profile and layout", profileAndLayout], ["full matrix", ENVIRONMENT]]) { console.log(`${name.padEnd(20)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(7)}` + `${`${new Set(s.map(profile)).size}/${profiles.length}`.padStart(9)}${`${found(s, DEFECT)}/${DEFECT.length}`.padStart(9)}` + `${String(red(s, ROBUST)).padStart(15)}`); } const missed = DEFECT.filter(([, k]) => !byShare(90).some(k)); console.log(`\nwhat the by-share selection misses:`); for (const [name] of missed) console.log(` ${name}`); ``` ``` 12 environments, 6 features, 6 distinguishing feature support profiles 111111 43.5% a-desktop-wide, a-desktop-narrow 111011 27.0% a-mobile-narrow, a-mobile-wide 100110 14.7% b-desktop-wide, b-desktop-narrow 111001 9.1% c-mobile-narrow, c-mobile-wide 100010 3.3% b-mobile-narrow, b-mobile-wide 111101 2.4% c-desktop-wide, c-desktop-narrow selection runs share profile defects red runs by share 90% 4 91.0% 4/6 3/6 3 by profile 6 96.0% 6/6 4/6 4 profile and layout 8 98.5% 6/6 6/6 5 full matrix 12 100.0% 6/6 6/6 8 what the by-share selection misses: K4 desktop layout overflows on a narrow screen K5 queue client cannot connect on mobile K6 mobile layout scatters on a wide screen ``` Twelve environments come down to six profiles: the matrix is half the size from a feature-support standpoint. Selecting by share covers 91% of users with four runs, but it sees only four of the six profiles and misses three of the six defects. What the missed ones share is telling: all three arise from **the combination of axes**. The desktop layout overflowing on a narrow screen has nothing to do with any engine's support; it has to do with a value the two axes take together. Selecting by profile is therefore not enough: it covers every support profile with six runs, but it still cannot see the layout defects, because the support vector never carries the screen class. The third selection, eight runs, adds the missing platform–screen pairs and finds all six of the six defects — with as many runs as two-thirds of the full matrix. ## Threshold Scanning The common rule is written as "cover this much of the users." What this threshold actually buys can be scanned. ```js // threshold.mjs — scanning the covered-share threshold; counts false pass and false fail import { ENVIRONMENT, DEFECT, ROBUST, profile, found, red } from "./matrix.mjs"; const byShare = (threshold) => { const s = []; for (const o of ENVIRONMENT) { if (s.reduce((t, x) => t + x.share, 0) >= threshold) break; s.push(o); } return s; }; const profiles = new Set(ENVIRONMENT.map(profile)).size; console.log(`${"share threshold".padEnd(17)}${"runs".padStart(6)}${"covered".padStart(10)}${"profile".padStart(9)}` + `${"false pass".padStart(14)}${"false fail".padStart(14)}${"cost w=5".padStart(12)}`); for (const e of [80, 90, 95, 99, 100]) { const s = byShare(e); const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST); console.log(`${`${e}%`.padEnd(17)}${String(s.length).padStart(6)}${`${s.reduce((t, o) => t + o.share, 0).toFixed(1)}%`.padStart(10)}` + `${`${new Set(s.map(profile)).size}/${profiles}`.padStart(9)}${String(pass).padStart(14)}${String(fail).padStart(14)}` + `${String(fail + 5 * pass).padStart(12)}`); } // The set selected by the profile-and-layout criterion is placed in the same table. const pair = (o) => `${o.platform}-${o.screen}`; const s = [...new Set(ENVIRONMENT.map(profile))].map((p) => ENVIRONMENT.filter((o) => profile(o) === p)[0]); for (const o of ENVIRONMENT) if (!s.some((x) => pair(x) === pair(o))) s.push(o); const pass = DEFECT.length - found(s, DEFECT), fail = red(s, ROBUST); console.log(`\nprofile-and-layout criterion: ${s.length} runs, false pass ${pass}, false fail ${fail},` + ` cost ${fail + 5 * pass}`); console.log(`runs needed to catch the same cost by share: ${byShare(99).length}`); ``` ``` share threshold runs covered profile false pass false fail cost w=5 80% 3 82.0% 3/6 4 2 22 90% 4 91.0% 4/6 3 3 18 95% 6 96.0% 6/6 2 4 14 99% 9 99.2% 6/6 0 6 6 100% 12 100.0% 6/6 0 8 8 profile-and-layout criterion: 8 runs, false pass 0, false fail 5, cost 5 runs needed to catch the same cost by share: 9 ``` **False pass** is a real defect sitting in an environment left outside the selection; **false fail** is strict verification turning red in an environment that is robust but behaves differently. The two move in opposite directions: as environments are added, missed defects decrease, and runs that turn red for nothing increase. Even the full matrix is not the best option — two of the eight red runs are unavoidable noise anyway. The real finding is in the last two rows. Selecting by share can find every defect only with nine runs; the profile-and-layout criterion gives the same result with eight runs, at a lower cost. **The threshold's source** here is not a measurement but a misconception: covered user share is an intuitive number, but it has nothing to do with how defects are distributed. Defects are distributed **by environment class**, not by user share, and the selection criterion turns out more decisive than the threshold. **Who owns the decision:** every environment left outside the matrix is a known gap. The release decision is tied not to the sentence "all runs are green" but to "this many profiles and this many layout pairs were run"; the classes that were not run go on record. ## The Cost of Testing The run-independent cost lives in three numbers. The first is the run count: eight environments, each demanding its own setup, its own data, and its own time; this number grows multiplicatively as axes are added — adding one value to one axis does not add to the matrix, it multiplies it. The second is maintaining the matrix: the support vectors are written by hand, and the profile count changes once an engine starts supporting a feature. A support table that is not updated either runs two environments it treats as distinguishable for nothing, or never runs one of two environments it fails to treat as distinct. The third is sifting through red runs: five of the eight runs turned red, and two of them were not real defects. This is compatibility testing's permanent burden — not every environment difference is a defect, and what separates a difference from a defect is a decision, not a rule. ## Summary - The compatibility matrix is the product of its axes; twelve environments came down to six distinguishing support profiles over six features, and the matrix shrank by half from a feature standpoint. - Selecting by share covered 91% of users with four runs but missed three of the six defects; all three of the misses arose not from a single axis but from the combination of axes. - Selecting by profile was not enough either, because the support profile does not carry the screen class; the profile-and-layout criterion found all six of the six defects with eight runs — as many as two-thirds of the full matrix. - Threshold scanning moved the two columns in opposite directions: as environments were added, false pass decreased and false fail increased. Even the full matrix was not the lowest-cost option. - Defects are distributed by environment class, not by user share; the selection criterion is more decisive than the covered-share threshold. - The cost is multiplicative: every new axis value grows the matrix by a factor, the support table demands manual upkeep, and a portion of red runs gets sifted through again on every run. ## Next Step This topic's six lessons tested the same system with six separate methods, and all six shared a common assumption: the system's parts are working. Static scanning looked at the code's text, dynamic testing sent requests to the running service, dependency scanning matched installed versions against advisories, penetration testing questioned the combination of rules, the accessibility audit read the rendered tree, compatibility testing ran the same screen in different environments. Both load and attack were pressure coming from **outside**, and in both cases the system itself stayed up. Yet what happens when the database the lending service depends on stops responding, when the queue client loses a message, or when the catalog service returns a half response, was tested in no lesson. The next topic moves the pressure from outside to inside: a fault is deliberately triggered in one of the system's own parts, and how the system meets that fault is measured.