--- title: 'Infrastructure Testing' source: 'https://academia.sh/en/courses/infrastructure-as-code/infrastructure-testing' course: 'Infrastructure as Code' language: en updated: '2026-09-13T05:08:25+00:00' license: 'CC BY-SA 4.0' objectives: - 'Runs unit, contract, and end-to-end validation over the same defect set and compares their coverage' - "Counts each level's duration, the resources it creates, and its cost per defect" - "Names the three levels' common gap and measures the leaked cost of resources that are not cleaned up" --- # Infrastructure Testing Validating an infrastructure change at three levels: running unit, contract, and end-to-end validation over the same set of fifteen defects, measuring what each level catches, catches alone, its duration, the resources it creates, and its cost in fictional units, naming the common gap all three levels miss, and counting the leftover resources and leaked cost left behind by test resources that are never cleaned up. Policy scanning tested the record's **acceptability** and left behind one rule that stayed undecided in both layers: a field whose value is assigned at apply time. That indecision is not an accident, it is a boundary — the scan reads the record, it does not see the infrastructure running. Does the resource actually come up, does its disk mount, does the message written to the queue reach the data store: none of that is written in the text. **Infrastructure testing** asks these questions and carries the same names as the test levels on the software side — unit, contract, end-to-end — but the alignment is not in the names, it is in **what gets created**: here the top level opens a real resource, runs a real charge, and leaves a trace behind it. This lesson runs the three levels over the same defect set and counts three things for each: the defects it catches, its duration, and its cost. Then two more numbers follow: the common gap all three levels miss together, and the uncleaned resource testing leaves behind. The through-line is the **regional measurement network**, and it is fictional. ## Defect Set and Three Levels **IV13 — the known defect set is fifteen defects, hand-marked, and each defect is tied to a layer**: `definition` (shows up in a single resource's definition), `relation` (shows up only when two resources are compared), `runtime` (shows up only while a resource is running), `change` (shows up only when applied over infrastructure that already exists). **IV14 — unit validation tests a single resource's or module's definition without creating a resource; contract validation runs on the plan and looks only at interface fields.** **IV15 — end-to-end validation creates real resources but only runs the main flow; a defect that stays on a side path is invisible to it.** ```js // levels.mjs — the same defect set is run at three validation levels (model) // [code, what, layer (where it shows up), field, path (main or side, in the end-to-end flow)] const DEFECT = [ ["D01", "machine disk is 10 GB, the collector is writing", "definition", "disk_gb", "main"], ["D02", "queue retention is 0 hours", "definition", "retention", "side"], ["D03", "archive class left as hot", "definition", "class", "side"], ["D04", "collector size is small", "definition", "size", "main"], ["D05", "owner tag not written", "definition", "owner", "side"], ["D06", "machine region does not match the network's region", "relation", "region", "main"], ["D07", "queue and data store are in different regions", "relation", "region", "side"], ["D08", "module output name is not what the consumer expects", "relation", "output", "main"], ["D09", "backup window overlaps the maintenance window", "relation", "window", "side"], ["D10", "security rule cuts off collector traffic", "runtime", "rule", "main"], ["D11", "disk does not mount, no mount point", "runtime", "mount", "main"], ["D12", "identity lacks permission, cannot write to the store", "runtime", "permission", "main"], ["D13", "field change recreates the resource, data is lost", "change", "disk_gb", "main"], ["D14", "in-place update causes an outage", "change", "size", "main"], ["D15", "an out-of-record manual change gets overwritten", "change", "rule", "main"], ].map(([code, what, layer, field, path]) => ({ code, what, layer, field, path })); // IV16: end-to-end validation creates real resources; count, creation time, and hourly rate are fictional const RESOURCE = [["network", 2, 20, 0], ["machine", 3, 45, 4], ["queue", 1, 30, 1], ["data_store", 1, 180, 12], ["object_store", 1, 15, 1]]; const COUNT = RESOURCE.reduce((s, [, n]) => s + n, 0); const CREATE = RESOURCE.reduce((s, [, n, y]) => s + n * y, 0); const HOURLY = RESOURCE.reduce((s, [, n, , u]) => s + n * u, 0); const END_TO_END_DURATION = CREATE + 60 + CREATE / 2; // creation + flow validation + teardown // IV17: contract validation looks only at interface fields const CONTRACT_FIELD = ["region", "output", "disk_gb"]; const LEVEL = [ ["unit", (k) => k.layer === "definition", 12, 0, 0], ["contract", (k) => k.layer === "relation" || (k.layer === "definition" && CONTRACT_FIELD.includes(k.field)), 45, 0, 0], ["end-to-end", (k) => k.layer !== "change" && k.path === "main", END_TO_END_DURATION, COUNT, (HOURLY * END_TO_END_DURATION) / 3600], ]; const caught = LEVEL.map(([, f]) => DEFECT.filter(f).map((k) => k.code)); console.log(`${"level".padEnd(12)}${"caught".padStart(9)}${"catches alone".padStart(16)}` + `${"missed".padStart(11)}${"resources created".padStart(20)}${"duration".padStart(10)}` + `${"cost (fictional units)".padStart(25)}`); LEVEL.forEach(([ad, , dur, resource, cost], j) => { const alone = caught[j].filter((k) => caught.every((t, o) => o === j || !t.includes(k))); console.log(`${ad.padEnd(12)}${`${caught[j].length}/${DEFECT.length}`.padStart(9)}` + `${String(alone.length).padStart(16)}${String(DEFECT.length - caught[j].length).padStart(11)}` + `${String(resource).padStart(20)}${`${dur} s`.padStart(10)}${cost.toFixed(2).padStart(25)}`); }); const all = new Set(caught.flat()); const missed = DEFECT.filter((k) => !all.has(k.code)); console.log(`\nthe three levels together catch ${all.size} of the ${DEFECT.length} defects; ` + `total duration ${LEVEL.reduce((s, [, , t]) => s + t, 0)} s, ` + `cost ${LEVEL.reduce((s, [, , , , u]) => s + u, 0).toFixed(2)} fictional units`); console.log(`\n${missed.length} defects all three levels miss (layer: ${missed[0].layer}):`); for (const k of missed) console.log(` ${k.code} ${k.what}`); console.log(`\ncost per defect at each level: ` + LEVEL.map(([ad, , dur, , cost], j) => `${ad} ${(dur / caught[j].length).toFixed(1)} s/defect` + (cost ? ` and ${(cost / caught[j].length).toFixed(2)} units/defect` : "")).join("; ")); ``` ``` level caught catches alone missed resources created duration cost (fictional units) unit 5/15 3 10 0 12 s 0.00 contract 5/15 2 10 0 45 s 0.00 end-to-end 7/15 3 8 8 660 s 4.77 the three levels together catch 12 of the 15 defects; total duration 717 s, cost 4.77 fictional units 3 defects all three levels miss (layer: change): D13 field change recreates the resource, data is lost D14 in-place update causes an outage D15 an out-of-record manual change gets overwritten cost per defect at each level: unit 2.4 s/defect; contract 9.0 s/defect; end-to-end 94.3 s/defect and 0.68 units/defect ``` ## Who Catches What The three levels together catch twelve of the fifteen defects. Looked at alone, none of them clears half: unit five, contract five, end-to-end seven. The real information is in the "catches alone" column, because that is where the cost of dropping a level from the pipeline is written. If unit validation is removed, three defects go unowned — the queue's zero retention, the archive's wrong class, the unwritten owner tag. All three sit in plain sight in a single resource's definition, and all three are on a side path; end-to-end validation creates the real resource but never travels those paths at all. Contract validation's two defects-of-its-own are there for the same reason: the queue and the data store being in different regions, and the backup window overlapping the maintenance window, only show up when two resources are set side by side, and neither is on the main flow. Contract validation's value is in the **comparison**, not the run; that is why, without creating a resource, it sees what an end-to-end test cannot. Overlap can be counted too. One of the twelve caught defects — the machine's ten-gigabyte disk — trips all three levels at once: it is written in the definition, it is one of the contract fields, and it shows itself while running on the main flow too. All three report the same defect, and the team reads the same thing three times. Three defects trip two levels, the remaining eight are each a single level's work. It is exactly this distribution that decides a pipeline's gate order. End-to-end validation's three defects, by contrast, exist only at the runtime layer: a security rule that cuts traffic, a disk that does not mount, a missing permission. None of these is written wrong in the record — the record is correct, reality differs. This is the clearest form of this course's measurement axis: the gap between text and reality can only be measured once reality is actually run. ## Duration and Cost The last three columns are the price of that same coverage. Unit validation finishes in twelve seconds, creates no resource, and costs nothing. Contract validation takes forty-five seconds — it waits for plan generation — but still creates nothing. End-to-end validation opens eight real resources, takes six hundred sixty seconds, and costs 4.77 fictional units; nine-tenths of its duration is creation and teardown, validation itself is sixty seconds. Looked at per defect, the difference is sharper still: unit 2.4 seconds/defect, contract 9.0 seconds/defect, end-to-end 94.3 seconds and 0.68 units/defect. End-to-end validation catches a defect in forty times the time unit validation takes. That does not mean "end-to-end is bad" — no other level can catch the three defects it catches. What it says is that **each level is cheapest in its own layer**: testing the same defect at two levels means paying for it a second time at the expensive one. The order is also read off this table. Unit validation weeds out five defects in twelve seconds; end-to-end validation, running after it, never looks for those five again. The cost of the reverse order can also be computed: if end-to-end ran first, catching a defect that unit validation would have found in twelve seconds would instead have cost eight real resources created, six hundred sixty seconds waited, and 4.77 units paid. The same defect's price varies by up to forty times depending on which gate catches it. ## The Three Levels' Common Gap The common name for the three missed defects is written in the output: the **change layer**. All three levels validate in a clean environment built from scratch. A real change, though, is not applied to an environment built from scratch — it is applied **on top of infrastructure that already exists** — and that is exactly where these defects are born: a field changing recreates the resource and the data inside it is gone; an in-place update produces an outage; an out-of-record manual change gets overwritten. Testing all three with a test built in an empty environment is impossible, because what needs testing is **history itself**: the data a resource already carries, the traffic it is already taking, the out-of-record change already made to it. What closes this gap is not a test level but the plan read before applying — it is there that which item will be recreated gets written. A manual change the record does not cover shows up in neither the test nor the plan; its only counterpart is measuring drift separately. ## Uncleaned Resources The 4.77-unit cost of end-to-end validation in the table assumes the teardown step succeeds on every run. Deletion is an apply step just like creation, and it can fail too: the pipeline times out, a resource cannot be deleted because it is tied to another resource, the run gets cut off halfway. ```js // leftover.mjs — the leftover resources and leaked cost end-to-end validation leaves behind (model) // IV18: one end-to-end run per day for thirty days; the teardown step fails on one run in five, // and on a failed run each resource stays up with even odds (fictional) const SEED = 20260215; let state = SEED; const random = () => (state = (state * 48271) % 2147483647) / 2147483647; // product stays under 2^53 const RESOURCE = [["network", 2, 0], ["machine", 3, 4], ["queue", 1, 1], ["data_store", 1, 12], ["object_store", 1, 1]]; // kind, count, fictional hourly cost const INSTANCE = RESOURCE.flatMap(([t, n, u]) => Array.from({ length: n }, (_, i) => [`${t}-${i + 1}`, u])); const DAY = 30, FAILURE = 0.2, RUN_COST = 4.77; // 4.77: run cost computed in the previous block const leftover = []; let failed = 0, leaked = 0; for (let g = 1; g <= DAY; g += 1) { if (random() < FAILURE) { failed += 1; for (const [name, u] of INSTANCE) if (random() < 0.5) leftover.push({ name: `${name}#k${g}`, u, day: g }); } leaked += leftover.reduce((s, k) => s + k.u, 0) * 24; // accumulated leftovers run a full day's cost } const intended = DAY * RUN_COST; console.log(`${DAY} days, ${DAY} runs, seed ${SEED}: teardown failed on ${failed} runs, ` + `${leftover.length} leftover resources accumulated`); console.log(`\n${"leftover resource".padEnd(20)}${"day left behind".padStart(17)}` + `${"hourly cost".padStart(14)}${"cost through month end".padStart(25)}`); for (const k of leftover) { console.log(`${k.name.padEnd(20)}${String(k.day).padStart(17)}${String(k.u).padStart(14)}` + `${String(k.u * 24 * (DAY - k.day + 1)).padStart(25)}`); } console.log(`\nintended cost (${DAY} runs x ${RUN_COST}) ${intended.toFixed(2)} units; ` + `leaked cost ${leaked.toFixed(2)} units; ratio ${(leaked / intended).toFixed(1)}x`); const priciest = [...leftover].sort((a, b) => b.u - a.u)[0]; console.log(`hourly cost of leftovers at month end ${leftover.reduce((s, k) => s + k.u, 0)} units; ` + `priciest leftover ${priciest.name} (${priciest.u} units/hour); ` + `leftover resources visible in the record 0`); ``` ``` 30 days, 30 runs, seed 20260215: teardown failed on 4 runs, 11 leftover resources accumulated leftover resource day left behind hourly cost cost through month end network-1#k3 3 0 0 queue-1#k3 3 1 672 data_store-1#k3 3 12 8064 object_store-1#k3 3 1 672 network-2#k12 12 0 0 machine-3#k12 12 4 1824 data_store-1#k12 12 12 5472 machine-1#k24 24 4 672 network-1#k29 29 0 0 queue-1#k29 29 1 48 data_store-1#k29 29 12 576 intended cost (30 runs x 4.77) 143.10 units; leaked cost 18000.00 units; ratio 125.8x hourly cost of leftovers at month end 47 units; priciest leftover data_store-1#k3 (12 units/hour); leftover resources visible in the record 0 ``` Teardown failed on four of the thirty runs, and eleven resources stayed up. The intended cost of the thirty runs was 143 fictional units; the leaked cost was 18,000 units, or **one hundred twenty-six times as much**. What inflates the ratio this far is not the failure count but the duration: a run takes eleven minutes, but a leftover resource sits for days before it is noticed. The third day's data store alone ran up 8,064 units. There is a second thing to read in the table. Seven of the eleven leftovers were left behind on the third and twelfth days, and they produce almost all of the leaked cost: the third day's four leftovers run up 9,408 units, the twelfth day's three run up 7,296, while the twenty-ninth day's three run up only 624. Teardown failure's cost is not constant, it is multiplied by **how long it takes to notice**; the way to cut down leftover resources is therefore not to make teardown one hundred percent reliable, it is to find the leftover early. The last line ties directly back to this course's measurement axis: **the number of leftover resources visible in the record is zero.** The record did not create these resources, testing did; the record's state does not contain them, and no one looking at the record can see them. The leftover testing leaves behind is the quietest form of infrastructure the record does not cover — and the only way to find it is to compare the provider against the record. ## Summary - The fifteen-defect set was run at three validation levels: unit caught 5, contract 5, end-to-end 7; together the three caught 12, and none cleared half on its own. - The counts of defects caught alone are 3, 2, and 3; contract validation catches two relation defects without creating a resource, and end-to-end validation catches three runtime defects no other level can see. - Cost grows with the layer: unit 12 s and 0 resources, contract 45 s and 0 resources, end-to-end 660 s, 8 real resources, and 4.77 fictional units; per defect, 2.4 / 9.0 / 94.3 seconds. - The three levels' common gap is the **change layer**: all three run in a clean environment built from scratch, while a real change is applied on top of infrastructure that already exists (recreation and data loss, in-place update outages, an out-of-record manual change getting overwritten). - Across thirty runs, teardown failed 4 times and 11 resources were left behind; against an intended 143 units, 18,000 units leaked (126 times as much), and the number of these resources visible in the record is zero. ## Next Step This lesson's last two numbers look at the same place: end-to-end validation's run cost and the leaked cost of leftover resources. Both are the **monetary** face of a change, and both were counted only once things had already happened. Yet the same question can be asked before applying: if this change is applied, how much does the monthly bill grow? The next lesson produces an estimate per change from the plan and compares it against what actually happens — which item can be estimated because it has a fixed unit price, which cannot because it depends on usage, how the estimation error is distributed, and how many changes a threshold rule of the form "stop if the estimate exceeds this much" holds up for nothing over a known set of changes.