Management of Test Environments

Every row in the previous lesson’s table rested on a silent assumption: a stage’s duration is just the run cost of the tests running there. That amounts to counting the environment as free. But every team besides the fast stage wants a test environment, and an environment cuts cost from two places — queueing, because it exists in limited number, and idle waiting, because it stays reserved continuously.

This lesson’s question is not how an environment is set up; that mechanism was built and counted step by step in the Test Environment Management lesson (four setup steps, two teardown steps). The question here is distribution: how many environments, at which stage, of which tier. The resource distributed is environment-minutes; its two returns are queue time and escaped defects.

Environment Tier and Fidelity

An environment’s distinguishing feature is not its price but its fidelity: which property of production it carries. Four tiers are distinguished. The none tier is not an environment at all, it is the runner itself; only teams that use nothing but their own processes belong here. The ephemeral tier is set up and torn down per run, lives for a single process, and fills with seeded data. The shared tier is a full system standing continuously: multiple services, persistent state, a real restart. The production-like tier adds production-scale data and a real resource limit on top of that.

Fidelity sets a team’s visibility limit. The end-to-end team cannot test card status in a single-process-lifetime environment, because the flow is spread across multiple services. So one more column is added to each row of the defect set: the minimum environment tier required for that class to be visible. The column’s values were read from the source lessons’ measurements — the distribution-dependent report defect is visible with seeded data, so ephemeral; the capacity limit only shows up under a real resource limit, so production-like.

TP11 — fidelity is ordered: none < ephemeral < shared < production-like. A higher tier sees everything a lower tier sees. This is a simplification that reduces fidelity to one dimension.

TP12 — a reserved environment pays cost all day long: a shared environment costs 1440 model minutes a day, a production-like environment four times that. Its cost is measured by its existence, not its use. The ephemeral environment is the opposite: it only pays for the minute it is used, adding a 3-minute setup and teardown per run.

// matrix.mjs -- environment tiers, stage list, and the defect set's tier column
// Stage durations, defect classes and counts read from the previous lesson's inventory.
export const LEVEL = { none: 0, ephemeral: 1, shared: 2, 'production-like': 3 };   // TP11
export const DAILY = { shared: 1440, 'production-like': 5760 };                     // TP12
export const SETUP = 3;   // an ephemeral environment's per-run setup + teardown cost

// [name, frequency, duration, baseline, teams]; baseline is the per-stage feedback time
// from the previous lesson's "three frequencies" row.
export const stages = [
  ['fast', 'change', 8, 8, ['unit', 'contract']],
  ['mid', 'change', 41, 49, ['integration', 'end-to-end']],
  ['nightly', 'nightly', 136, 425, ['perf-run', 'security-auto', 'resilience']],
  ['release', 'release', 836, 2221, ['perf-sample', 'security-manual']],
].map(([name, frequency, duration, baseline, teams]) => ({ name, frequency, duration, baseline, teams }));

// [defectClass, catching team, count in a hundred changes, required minimum environment tier]
export const defects = [
  ['boundary comparison', 'unit', 13, 0],
  ['schema mismatch', 'integration', 8, 1],
  ['card status not updating', 'end-to-end', 7, 2],
  ['status code mapping', 'end-to-end', 6, 1],
  ['breaking change', 'contract', 6, 0],
  ['request shape drift', 'contract', 5, 0],
  ['accessibility criterion', 'security-manual', 5, 2],
  ['field lost from contract', 'contract', 4, 0],
  ['missing warning on rejected request', 'end-to-end', 4, 2],
  ['latency regression', 'perf-run', 4, 3],
  ['data loss in migration', 'integration', 3, 1],
  ['concatenated query', 'security-auto', 3, 0],
  ['semantic mismatch', null, 3, 0],
  ['check-then-act race', 'end-to-end', 2, 2],
  ['distribution-dependent report defect', 'integration', 2, 1],
  ['process-lifetime state', null, 2, 0],
  ['percentile read from few samples', 'perf-sample', 1, 3],
  ['recovery gap', 'resilience', 1, 2],
  ['capacity limit', 'resilience', 1, 3],
].map(([defectClass, team, count, tier]) => ({ defectClass, team, count, tier }));

// An arrangement gives each pool an environment tier, a count, and the stages it serves.
const E = (stages) => ({ tier: 'ephemeral', count: Infinity, stages });
const S = (count, stages) => ({ tier: 'shared', count, stages });
const P = (count, stages) => ({ tier: 'production-like', count, stages });

export const arrangements = [
  { name: 'all ephemeral', pools: [E(['mid', 'nightly', 'release'])] },
  { name: 'ephemeral mid', pools: [E(['mid']), S(1, ['nightly', 'release'])] },
  { name: 'one shared', pools: [S(1, ['mid', 'nightly', 'release'])] },
  { name: 'two shared', pools: [S(2, ['mid', 'nightly', 'release'])] },
  { name: 'four shared', pools: [S(4, ['mid', 'nightly', 'release'])] },
  { name: 'separate heavy environment', pools: [S(2, ['mid']), S(1, ['nightly', 'release'])] },
  { name: 'production-like release', pools: [S(2, ['mid']), S(1, ['nightly']), P(1, ['release'])] },
  { name: 'production-like heavy', pools: [S(2, ['mid']), P(1, ['nightly', 'release'])] },
];

Queue

A reserved environment carries a single run at a time. When two changes arrive at once, one waits, and its wait is added to that change’s feedback time. The queue’s size depends not on average load but on how irregular the arrivals are; that is why arrivals are placed not at even intervals but with a seeded sequence.

TP13 — five changes a day are placed across the workday’s 480 minutes with a seed; environments run without interruption. The previous lesson’s TP9 spread a hundred changes across twenty workdays; the addition here is where those five changes land within the day. The nightly run enters the queue at minute 480 of every day, the release run at the same minute of four release days — the release run competes for the same environment as the nightly run, at the same moment.

// queue.mjs -- a seeded arrival sequence and a k-environment first-come-first-served queue
export const DAYS = 20, WORKDAY = 480, FULL_DAY = 1440, DAILY_CHANGES = 5;

// TP13: linear congruential generator (from an earlier course); seed is visible, sequence is reproducible.
export function makeGenerator(seed) {
  let state = seed;
  return () => { state = (state * 1103515245 + 12345) % 2147483648; return state / 2147483648; };
}

export function arrivals(seed = 20260731) {
  const random = makeGenerator(seed);
  const list = [];
  for (let day = 0; day < DAYS; day += 1) {
    const minutes = [];
    for (let i = 0; i < DAILY_CHANGES; i += 1) minutes.push(Math.floor(random() * WORKDAY));
    for (const m of minutes.sort((a, b) => a - b)) list.push(day * FULL_DAY + m);
  }
  return list;
}

// jobs: { t, duration, stage }. If count is Infinity the environment is always ready.
export function measureQueue(jobs, count) {
  const sorted = [...jobs].sort((a, b) => a.t - b.t);
  const freeAt = count === Infinity ? null : new Array(count).fill(0);
  const wait = new Map();
  let busy = 0;
  for (const job of sorted) {
    let start = job.t;
    if (freeAt !== null) {
      let least = 0;
      for (let i = 1; i < freeAt.length; i += 1) if (freeAt[i] < freeAt[least]) least = i;
      start = Math.max(job.t, freeAt[least]);
      freeAt[least] = start + job.duration;
    }
    const w = wait.get(job.stage) ?? [];
    w.push(start - job.t);
    wait.set(job.stage, w);
    busy += job.duration;
  }
  const summary = new Map();
  for (const [stage, w] of wait) {
    summary.set(stage, { avg: w.reduce((a, x) => a + x, 0) / w.length, worst: Math.max(...w) });
  }
  const occupancy = count === Infinity ? 0 : busy / (count * DAYS * FULL_DAY);
  return { summary, occupancy };
}

Eight Arrangements

The measurement holds the previous lesson’s three-frequency gate arrangement fixed and only changes the environment distribution. If a stage’s environment tier is below the required fidelity, the defect escapes even if a catching team is assigned to it.

// measure.mjs -- eight environment arrangements over the same team list: queue, cost, escaped defects
import { LEVEL, DAILY, SETUP, stages, defects, arrangements } from './matrix.mjs';
import { arrivals, measureQueue, DAYS, WORKDAY, FULL_DAY } from './queue.mjs';

const ARRIVALS = arrivals();
const RELEASE_DAYS = [4, 9, 14, 19];               // TP9: four releases in twenty workdays

const jobList = (stage, extra) => {
  const duration = stage.duration + extra;
  if (stage.frequency === 'change') return ARRIVALS.map((t) => ({ t, duration, stage: stage.name }));
  const days = stage.frequency === 'release' ? RELEASE_DAYS : [...Array(DAYS).keys()];
  return days.map((d) => ({ t: d * FULL_DAY + WORKDAY, duration, stage: stage.name }));
};

export function measure(arrangement) {
  const tierOf = new Map();                          // stage name -> environment tier
  for (const p of arrangement.pools) for (const s of p.stages) tierOf.set(s, p.tier);

  let cost = 0, occupancy = 0;
  const wait = new Map();
  for (const p of arrangement.pools) {
    const extra = p.tier === 'ephemeral' ? SETUP : 0;
    const jobs = p.stages.flatMap((name) => jobList(stages.find((s) => s.name === name), extra));
    const r = measureQueue(jobs, p.count);
    for (const [stage, o] of r.summary) wait.set(stage, o);
    if (p.count === Infinity) cost += jobs.reduce((a, x) => a + x.duration, 0);
    else { cost += p.count * DAILY[p.tier] * DAYS; occupancy = Math.max(occupancy, r.occupancy); }
  }

  let extra = 0, queue = 0;                          // accumulates in stage order
  const feedback = new Map();
  for (const s of stages) {
    if (tierOf.get(s.name) === 'ephemeral') extra += SETUP;
    queue += wait.get(s.name)?.avg ?? 0;
    feedback.set(s.name, s.baseline + extra + queue);
  }

  let missed = 0, weight = 0, total = 0;
  for (const k of defects) {
    const s = stages.find((x) => x.teams.includes(k.team));
    const level = s === undefined ? 0 : LEVEL[tierOf.get(s.name) ?? 'none'];
    if (s === undefined || level < k.tier) { missed += k.count; continue; }
    weight += k.count;
    total += feedback.get(s.name) * k.count;
  }
  return {
    envs: arrangement.pools.reduce((a, p) => a + (p.count === Infinity ? 0 : p.count), 0),
    cost: cost / 100, occupancy, wait,
    queue: wait.get('mid')?.avg ?? 0,
    worst: Math.max(0, ...[...wait.values()].map((o) => o.worst)),
    feedback: total / weight, missed,
  };
}

const s = (x, n) => String(x).padStart(n);
console.log(`${'arrangement'.padEnd(28)}${s('envs', 6)}${s('env-min/chg', 14)}${s('occupancy', 11)}`
  + `${s('mid queue', 13)}${s('worst', 9)}${s('avg feedback', 14)}${s('missed/100', 12)}`);
for (const d of arrangements) {
  const r = measure(d);
  console.log(`${d.name.padEnd(28)}${s(r.envs, 6)}${s(r.cost.toFixed(0), 14)}`
    + `${s(r.envs === 0 ? '-' : `${(r.occupancy * 100).toFixed(0)}%`, 11)}`
    + `${s(r.queue.toFixed(1), 13)}${s(r.worst.toFixed(0), 9)}`
    + `${s(r.feedback.toFixed(1), 14)}${s(r.missed, 12)}`);
}

const oneShared = measure(arrangements.find((d) => d.name === 'one shared'));
console.log(`\nqueue per stage in the one-shared environment (minutes):`);
for (const [stage, o] of oneShared.wait) {
  console.log(`  ${stage.padEnd(9)} average ${s(o.avg.toFixed(1), 6)}, worst ${s(o.worst, 5)}`);
}
const load = 100 * 41 + 20 * 136 + 4 * 836;
console.log(`\narrival sequence seed 20260731; ${ARRIVALS.length} changes, ${DAYS} workdays, `
  + `first arrival minute ${ARRIVALS[0] % FULL_DAY}, workday ${WORKDAY} minutes`);
console.log(`one shared environment's load ${load} minutes, reserved duration ${DAYS * FULL_DAY} minutes`);
const seen = defects.filter((k) => k.team !== null);
const baseline = seen.reduce((a, k) => a + stages.find((x) => x.teams.includes(k.team)).baseline * k.count, 0)
  / seen.reduce((a, k) => a + k.count, 0);
console.log(`average feedback with environment queue counted as zero: ${baseline.toFixed(1)} minutes`);

arrangement                   envs   env-min/chg  occupancy    mid queue    worst  avg feedback  missed/100
all ephemeral                    0           105          -          0.0        0          50.1          30
ephemeral mid                    1           332        21%          0.0      136         262.9          24
one shared                       1           288        35%          6.4      172         228.3          11
two shared                       2           576        18%          0.0       38         212.5          11
four shared                      4          1152         9%          0.0        0         211.6          11
separate heavy environment       3           864        21%          0.0      136         221.4          11
production-like release          4          2016        12%          0.0        0         240.3          10
production-like heavy            3          1728        21%          0.0      136         263.5           5

queue per stage in the one-shared environment (minutes):
  mid       average    6.4, worst    50
  nightly   average   18.6, worst    79
  release   average  145.0, worst   172

arrival sequence seed 20260731; 100 changes, 20 workdays, first arrival minute 74, workday 480 minutes
one shared environment's load 10164 minutes, reserved duration 28800 minutes
average feedback with environment queue counted as zero: 252.6 minutes

Reading the Table

The last line ties back to the previous lesson: with environment queue counted as zero, average feedback is 252.6 minutes — the same number as the previous lesson’s three-frequency row. That number was hiding an environment assumption.

In the one-shared-environment arrangement, occupancy is 35 percent; the environment is idle two-thirds of the time. Yet the queue is not zero, and where it builds up cannot be read from the average: the mid stage waits 6.4 minutes on average, 50 in the worst case; the nightly stage 18.6 and 79; the release stage 145.0 and 172. The queue is largest at the least frequently running stage: the release run competes for the same environment at the same minute as the nightly run and waits behind its 136-minute run. A second environment resolves this directly — the release wait drops from 145 to 7.5, the mid stage’s to zero. In exchange, cost rises from 288 to 576 environment-minutes, and escaped defects do not change — they stay at eleven. The queue is solved with money; fidelity is not.

Four environments take this one step further: the queue zeroes out at every stage, occupancy drops to 9 percent, cost rises to 1152 environment-minutes, and average feedback falls from 212.5 to only 211.6. The third and fourth environments buy 0.9 minutes for 576 environment-minutes; at 18 percent occupancy, replication buys nothing.

The first two rows look the other way. The all-ephemeral arrangement holds no reserved environment at all, is the cheapest row at 105 environment-minutes per change, and raises escaped defects from five to thirty. Even when the ephemeral environment is kept only at the mid stage, escaped defects are twenty-four, because the end-to-end team runs at that stage and cannot see card status, the warning on a rejected request, or the check-then-act race in a single-process-lifetime environment. Its cost is 332 environment-minutes — more expensive than one shared environment, with more than double its escaped defects. An arrangement that loses on both axes at once shows up in the table but not in a narrative.

The average feedback column repeats the previous lesson’s trap: the all-ephemeral arrangement gives the table’s best feedback at 50.1 minutes, because the late-caught classes are no longer caught at all and have dropped out of the average’s denominator.

The last three rows ask where fidelity should go. When the production-like environment is placed at the release gate, escaped defects drop from eleven to only ten; the only class recovered is the percentile read from few samples. When the same environment covers the nightly stage instead, escaped defects drop to five — latency regression and capacity limit are the teams running there — and with three environments instead of four, at 1728 environment-minutes instead of 2016. Fidelity matches the team running at a gate, not how late that gate is; putting the most expensive environment at the last gate invests the money in the wrong place.

Escaped defects do not drop below five. Those five defects belong to two classes that no team in the previous lesson’s inventory sees; even a production-like environment does not catch process-lifetime state, because no test was written to catch it. Environment fidelity widens a test’s visibility limit; it does not write the test itself.

The Return on the Decision

An environment decision’s return shows up in two places. The first is the queue: while a change waits for an environment, its owner moves on to other work, and by the time a red result arrives, the context is gone — this is why queue time is counted in the same column as run time. The second is the environment’s own red: when an environment breaks, every test in that stage drops, and the dropped tests report an environment, not a defect. This result’s owner is not the person who made the change but the team holding the environment; the decision is not to quarantine the test but to bring the environment back. Being unable to tell the two kinds of red apart is the shared environment’s uncounted third cost.

Summary

• An environment splits into four tiers; the tier sets the highest defect class the team running at that stage can see, and it does not write the test itself.
• With environment queue counted as zero, average feedback is 252.6 minutes: the previous lesson’s number rested on a hidden environment assumption.
• One shared environment produced a queue even at 35 percent occupancy; the queue is largest at the least frequently running stage (release 145.0 minutes, mid 6.4 minutes).
• A second environment nearly zeroes the queue, raises cost from 288 to 576 environment-minutes, and does not change escaped defects; a fourth environment buys 0.9 minutes for 576 environment-minutes.
• The ephemeral environment is the cheapest row (105 environment-minutes) and raises escaped defects from five to thirty; the arrangement that keeps the ephemeral environment at the mid stage is both more expensive and misses more defects.
• Moving the production-like environment to the nightly stage instead of the release gate dropped escaped defects from ten to five and lowered cost from 2016 to 1728 environment-minutes.

Next Step

The production-like environment leaves one question open: what goes into that environment for it to resemble production. The data side was measured in the Test Data Management lesson. What was not counted is access credentials, identity, signing keys, and external-service tokens; all of these are secrets, and every secret handed to a test produces two things — a leak surface and a catchable defect class. The next lesson counts this distribution: which secrets enter the environment, how much the exposure surface shrinks when authorization scope is narrowed, and how many steps a time-limited privilege leaves manual.