--- title: 'Format and Validation Checks' source: 'https://academia.sh/en/courses/infrastructure-as-code/format-and-validation-checks' course: 'Infrastructure as Code' language: en updated: '2026-09-13T05:08:25+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes three static checks over a configuration record and runs them against a known defect set' - 'Measures what each check catches, catches alone, and misses across the known defect set' - 'Counts the run cost of static checking and names the defect class it cannot see' --- # Format and Validation Checks Testing the infrastructure record before it is applied: actually running format, syntax, and type/reference checks over configuration definition files, measuring what each check catches, catches alone, and misses across sixteen known defects, counting run cost in files read and comparisons, and naming the defect class static checking cannot see. The record is written: the infrastructure is defined in a configuration language, split across files, and the repeated parts are turned into modules. What has been measured so far is the record's **structure**; what has never been measured is its **correctness**. When these files are applied, does the desired infrastructure come out, or does the apply stop halfway? If the question is not asked before applying, the apply itself answers it, and the cost of that answer is a resource that has already been created. **Static checking** reads the record before it is applied: it touches no resource, connects to no provider, and reads only the text on disk. This lesson writes three static checks and runs them over fictional definition files — format, syntax, and type-and-reference consistency. For each check, this lesson counts what it **catches and catches alone** in the known defect set, what it **misses**, and its **run cost**; at the end, the defect class static checking cannot see is measured against the apply. The through-line is the **regional measurement network**, and it is fictional. ## Record and Known Defect Set **IV1 — the production environment of the measurement network is held in four definition files across eleven resources: three networks, five machines, one queue, one data store, one object store (fictional).** **IV2 — the configuration language's format is our own small format**: a block header at column 0, below it two-space-indented fields, `=` a plain value, `->` a reference. **IV3 — the known defect set is sixteen defects, hand-marked**; each defect is injected into a specific line of the clean record, and which defect a finding corresponds to is read from the line number. ```js // setup.mjs — writes the measurement network's record, injects known defects into a copy (fictional) import { mkdirSync, writeFileSync } from "node:fs"; // format: "resource /:" at column 0, below it two-space-indented field lines; // "=" a plain value, "->" a reference to another resource, "#" a comment, one blank line between blocks const block = (kind, name, fields) => `resource ${kind}/${name}:\n` + fields.map(([a, y, d]) => ` ${a.padEnd(10)}${y} ${d}`).join("\n") + "\n"; const MACHINE = [["west-1", "west", "medium", 40], ["west-2", "west", "medium", 40], ["west-3", "west", "large", 80], ["east-1", "east", "medium", 40], ["east-2", "east", "large", 80]]; const FILE = { "backbone.def": ["west", "east", "north"].map((b, i) => block("network", `backbone-${b}`, [["region", "=", b], ["block", "=", `10.2${i}.0.0/16`], ["private", "=", "yes"]])).join("\n"), "machine.def": MACHINE.map(([n, b, sz, d]) => block("machine", `collector-${n}`, [["region", "=", b], ["size", "=", sz], ["disk_gb", "=", d], ["network", "->", `network/backbone-${b}`]])).join("\n"), "flow.def": block("queue", "measurement-queue", [["region", "=", "west"], ["retention", "=", 72], ["network", "->", "network/backbone-west"]]) + "\n" + block("data_store", "measurement-store", [["region", "=", "west"], ["size", "=", "large"], ["disk_gb", "=", 500], ["backup", "=", 14], ["network", "->", "network/backbone-west"]]), "hold.def": block("object_store", "measurement-archive", [["region", "=", "west"], ["versioned", "=", "yes"], ["class", "=", "cold"]]), }; // known defect set: [code, file, line in the clean record, replacement, defect's location] const DEFECT = [ ["D01", "backbone.def", 2, " region = west "], ["D02", "backbone.def", 9, " private = open"], ["D03", "machine.def", 3, " size = medium"], ["D04", "machine.def", 5, " network -> object_store/measurement-archive"], ["D05", "machine.def", 10, " disk_gb=40"], ["D06", "machine.def", 16, " disk_gb = eighty"], ["D07", "machine.def", 18, "@delete", 19], ["D08", "machine.def", 22, " disk_gb = 40\n disk_gb = 60"], ["D09", "machine.def", 23, " network -> network/backbone-south"], ["D10", "machine.def", 25, "resource machine/collector-east-1:"], ["D11", "backbone.def", 11, "resource network/backbone-north"], ["D12", "flow.def", 2, " region = west\n regions = west,east"], ["D13", "flow.def", 3, "\tretention = 72"], ["D14", "flow.def", 8, " size ="], ["D15", "flow.def", 11, " # network connection added later", 6], ["D16", "hold.def", 4, " class = cold\n label measurement archive"], ]; mkdirSync("record", { recursive: true }); mkdirSync("draft", { recursive: true }); const location = []; for (const [file, content] of Object.entries(FILE)) { writeFileSync(`record/${file}`, content); const patch = new Map(DEFECT.filter((k) => k[1] === file).map((k) => [k[2], k[3]])); const newLines = [], lineMap = {}; content.split("\n").forEach((s, i) => { // a defect can delete a line or add lines const d = patch.get(i + 1); if (d === "@delete") return; newLines.push(...(d ?? s).split("\n")); lineMap[i + 1] = newLines.length; // clean-record line -> its final counterpart in the draft }); writeFileSync(`draft/${file}`, newLines.join("\n")); for (const [code, , line, , k] of DEFECT.filter((x) => x[1] === file)) location.push({ code, file, line: lineMap[k ?? line] }); } writeFileSync("defects.json", JSON.stringify(location)); const text = Object.values(FILE).join(""); console.log(`record: ${Object.keys(FILE).length} files, ${text.match(/^resource /gm).length} resources, ` + `${text.split("\n").length} lines; draft: same record + ${DEFECT.length} known defects\n`); console.log(FILE["flow.def"].split("\n").slice(0, 4).join("\n")); ``` ``` record: 4 files, 11 resources, 59 lines; draft: same record + 16 known defects resource queue/measurement-queue: region = west retention = 72 network -> network/backbone-west ``` ## Three Static Checks **IV4 — the format rules are six in number and team-accepted.** **IV5 — every resource kind's fields, types, and requirements live in a schema kept alongside the record.** All three produce findings in the same shape — file, line, rule — but their contexts differ: format looks at a single line, syntax looks at the line and its block, type/reference wants the whole record, because a machine's `network` field points to a resource in another file. ```js // check.mjs — same record, three static checks: format, syntax, type/reference import { readFileSync, readdirSync } from "node:fs"; const SCHEMA = { // kind -> "field:type" list; "!" required, "@t" a reference to a resource of kind t network: "region:text! block:text! private:boolean!", machine: "region:text! size:text! disk_gb:number! network:@network!", queue: "region:text! retention:number! network:@network!", data_store: "region:text! size:text! disk_gb:number! backup:number! network:@network!", object_store: "region:text! versioned:boolean! class:text!", }; const schemaFields = (kind) => Object.fromEntries(SCHEMA[kind].split(" ").map((x) => x.split(":"))); const HEADER = /^resource ([a-z_]+)\/([a-z0-9-]+):$/, FIELD = /^\s+([a-z_]+)\s*(=|->)\s*(.*)$/; const read = (dir) => readdirSync(dir).filter((d) => d.endsWith(".def")) .map((d) => [d, readFileSync(`${dir}/${d}`, "utf8").split("\n")]); const counts = { format: 0, syntax: 0, type: 0 }; // 1) format: six line-local rules; reads nothing but the single line const R = [ ["R1", (s) => s.length && s !== s.trimEnd()], // trailing whitespace ["R2", (s) => /^ *\t/.test(s)], // tab in the indent ["R3", (s) => /^ +\S/.test(s) && !/^ {2}\S/.test(s)], // indent is not two spaces ["R4", (s) => /=|->/.test(s) && !/^\s*#/.test(s) && !/ (=|->)( |$)/.test(s)], // no space around separator ["R5a", (s) => /^[^\s#]/.test(s) && !s.endsWith(":")], // header does not end with a colon ["R5b", (s, i, L) => /^[^\s#]/.test(s) && i > 0 && L[i - 1] !== ""], // no blank line before block ]; function format(files) { const b = []; for (const [d, L] of files) L.forEach((s, i) => R.forEach(([k, f]) => { counts.format += 1; if (f(s, i, L)) b.push({ file: d, line: i + 1, rule: k }); })); return b; } // 2) syntax: line- and block-local; never looks outside its own file function syntax(files) { const b = [], seen = new Map(); for (const [d, L] of files) { let keys = null; L.forEach((s, i) => { const E = (k) => b.push({ file: d, line: i + 1, rule: k }); counts.syntax += 3; if (s === "" || /^\s*#/.test(s)) return; if (/^\S/.test(s)) { const m = HEADER.exec(s); keys = null; if (!m) return E("P1 header pattern does not match"); keys = new Set(); const path = `${m[1]}/${m[2]}`; if (seen.has(path)) E(`P4 duplicate resource name (${seen.get(path)})`); else seen.set(path, `${d}:${i + 1}`); return; } const m = FIELD.exec(s); if (!m) return E("P2 unrecognized line"); if (m[3].trim() === "") return E("P2 no value"); if (keys?.has(m[1])) E(`P3 duplicate key (${m[1]})`); keys?.add(m[1]); }); } return b; } // 3) type/reference: wants the whole record, because references cross file boundaries function typeRef(files) { const resources = [], b = []; for (const [d, L] of files) { let current = null; L.forEach((s, i) => { const m = /^\S/.test(s) ? HEADER.exec(s) : FIELD.exec(s); if (/^\S/.test(s)) current = m ? { kind: m[1], name: m[2], file: d, line: i + 1, field: {} } : null; else if (current && m) current.field[m[1]] = { value: m[3].trim(), line: i + 1 }; else return; if (/^\S/.test(s) && current) resources.push(current); }); } const literalType = (v) => (["yes", "no"].includes(v) ? "boolean" : /^-?\d+$/.test(v) ? "number" : "text"); for (const k of resources) { const schema = schemaFields(k.kind); for (const [a, v] of Object.entries(k.field)) { const E = (rule) => b.push({ file: k.file, line: v.line, rule }); counts.type += 2; if (!schema[a]) { E(`C1 not in schema (${a})`); continue; } const t = schema[a].replace("!", ""); if (!t.startsWith("@")) { if (v.value === "" || literalType(v.value) !== t) E(`C2 not ${t} ("${v.value}")`); continue; } let h = null; for (const x of resources) { counts.type += 1; if (`${x.kind}/${x.name}` === v.value) { h = x; break; } } if (!h) E(`C4 target does not exist (${v.value})`); else if (h.kind !== t.slice(1)) E(`C5 wrong kind (${h.kind})`); } for (const [a, t] of Object.entries(schema)) { counts.type += 1; if (t.endsWith("!") && !k.field[a]) b.push({ file: k.file, line: k.line, rule: `C3 required field missing (${a})` }); } } return b; } const draft = read("draft"), clean = read("record"); const LEVEL = [["format", format, 1], ["syntax", syntax, 1], ["type/reference", typeRef, 4]]; const finding = LEVEL.map(([ad, f]) => [ad, f(draft)]); const totals = [counts.format, counts.syntax, counts.type]; // counters are read only from the draft run const DEFECT = JSON.parse(readFileSync("defects.json", "utf8")); const code = (f) => DEFECT.find((x) => x.file === f.file && x.line === f.line)?.code ?? "??"; console.log(`${"check".padEnd(16)}${"code".padEnd(6)}${"location".padEnd(20)}rule`); for (const [ad, l] of finding) for (const f of l) console.log(`${ad.padEnd(16)}${code(f).padEnd(6)}${`${f.file}:${f.line}`.padEnd(20)}${f.rule}`); const caught = finding.map(([, l]) => new Set(l.map(code))); console.log(`\n${"check".padEnd(16)}${"findings".padStart(10)}${"caught".padStart(9)}` + `${"catches alone".padStart(16)}${"missed".padStart(11)}${"comparisons".padStart(15)}` + `${"files read".padStart(16)}`); finding.forEach(([ad, l], j) => { const y = [...caught[j]].filter((k) => caught.every((s, o) => o === j || !s.has(k))).length; console.log(`${ad.padEnd(16)}${String(l.length).padStart(10)}${String(caught[j].size).padStart(9)}` + `${String(y).padStart(16)}${String(DEFECT.length - caught[j].size).padStart(11)}` + `${String(totals[j]).padStart(15)}${String(LEVEL[j][2]).padStart(16)}`); }); console.log(`\nthe three checks together catch ${DEFECT.length} of the known defects, ` + `${new Set(caught.flatMap((s) => [...s])).size}; findings matching no defect ` + `${finding.flatMap(([, l]) => l).filter((f) => code(f) === "??").length}; ` + `total findings on the clean record ${LEVEL.reduce((s, [, f]) => s + f(clean).length, 0)}`); ``` ``` check code location rule format D01 backbone.def:2 R1 format D11 backbone.def:11 R5a format D13 flow.def:4 R2 format D03 machine.def:3 R3 format D05 machine.def:10 R4 format D07 machine.def:18 R5b syntax D11 backbone.def:11 P1 header pattern does not match syntax D14 flow.def:9 P2 no value syntax D16 hold.def:5 P2 unrecognized line syntax D08 machine.def:22 P3 duplicate key (disk_gb) syntax D10 machine.def:25 P4 duplicate resource name (machine.def:18) type/reference D02 backbone.def:9 C2 not boolean ("open") type/reference D12 flow.def:3 C1 not in schema (regions) type/reference D14 flow.def:9 C2 not text ("") type/reference D15 flow.def:7 C3 required field missing (network) type/reference D04 machine.def:5 C5 wrong kind (object_store) type/reference D06 machine.def:16 C2 not number ("eighty") type/reference D09 machine.def:23 C4 target does not exist (network/backbone-south) check findings caught catches alone missed comparisons files read format 6 6 5 10 384 1 syntax 5 5 3 11 192 1 type/reference 7 7 6 9 131 4 the three checks together catch 16 of the known defects, 16; findings matching no defect 0; total findings on the clean record 0 ``` ## Coverage, Overlap, and Cost The three checks together catch all sixteen of the sixteen defects, but none is sufficient alone: even type/reference, the broadest in coverage, misses nine defects. Overlap happens in two places — the missing colon in a header (D11) trips both format and syntax, and a value left empty (D14) trips both syntax and type. The remaining fourteen defects are each one check's own catch; the "catches alone" column reads 5, 3, and 6. The comparison counts say the opposite of what intuition suggests: **the check that makes the most comparisons wants the least context.** Format makes 384 comparisons (64 lines times six rules), type/reference only 131. The cost is not in the comparisons but in the **scope**: when a single file changes, the first two checks can read that file and decide; type/reference has to read all four. What is more, because reference resolution scans the resource list from the start, the type check's comparisons grow quadratically with the number of resources. Two quiet numbers sit in the table as well: findings matching no defect, and total findings on the clean record, are both zero. ## The Class Static Checking Cannot See The common name for the defects none of the three checks can see is **defects that depend on provider truth**: the state that decides is not in the record — the region's quota, the machine sizes available in the region, a name held in the name pool, permission to create that resource kind. None of it is written in the text. **IV6 — the provider's quota, region-size catalog, name pool, permissions, and resource-creation durations are fictional.** The block below applies the clean record — the one static checking scores zero findings on — to this model; items are processed in the order the record is read, and a rejected item is skipped. ```js // apply.mjs — the clean record is applied to a provider model; the provider and durations are fictional import { readFileSync, readdirSync } from "node:fs"; const PROVIDER = { quota: { west: 2, east: 4 }, // machine ceiling per region catalog: { west: ["medium", "large"], east: ["medium"] }, // sizes available in the region reservedName: ["measurement-archive"], // name is globally unique permission: { network: 1, machine: 1, queue: 1, data_store: 0, object_store: 1 }, }; const DURATION = { network: 20, machine: 45, queue: 30, data_store: 180, object_store: 15, reject: 5 }; const resource = []; for (const d of readdirSync("record")) for (const s of readFileSync(`record/${d}`, "utf8").split("\n")) { const h = /^resource ([a-z_]+)\/([a-z0-9-]+):$/.exec(s), a = /^\s+([a-z_]+)\s*=\s*(.+)$/.exec(s); if (h) resource.push({ kind: h[1], name: h[2], field: {} }); else if (a) resource.at(-1).field[a[1]] = a[2].trim(); } let seconds = 0, created = 0; const rejected = [], counted = {}; for (const k of resource) { let error = null; if (!PROVIDER.permission[k.kind]) error = "permission denied"; else if (PROVIDER.reservedName.includes(k.name)) error = "name reserved"; else if (k.kind === "machine") { counted[k.field.region] = (counted[k.field.region] ?? 0) + 1; if (counted[k.field.region] > PROVIDER.quota[k.field.region]) error = "quota exceeded"; else if (!PROVIDER.catalog[k.field.region].includes(k.field.size)) error = "not in catalog"; } if (error) rejected.push({ name: `${k.kind}/${k.name}`, error, seconds, created }); else created += 1; seconds += error ? DURATION.reject : DURATION[k.kind]; } console.log(`${resource.length} items applied: ${created} created, ${rejected.length} rejected, ` + `${seconds} seconds total\n`); console.log(`${"rejected item".padEnd(35)}${"reason".padEnd(19)}${"at second".padStart(17)}` + `${"created so far".padStart(21)}`); for (const r of rejected) console.log(`${r.name.padEnd(35)}${r.error.padEnd(19)}` + `${String(r.seconds).padStart(17)}${String(r.created).padStart(21)}`); ``` ``` 11 items applied: 7 created, 4 rejected, 245 seconds total rejected item reason at second created so far data_store/measurement-store permission denied 90 4 object_store/measurement-archive name reserved 95 4 machine/collector-west-3 quota exceeded 190 6 machine/collector-east-2 not in catalog 240 7 ``` All four defects are in the record that static checking scored as clean; the difference is in time and cost. The three checks scan the 64 lines with 707 comparisons and deliver their result without creating a single resource; the first provider-dependent defect shows up at second 90, the last at second 240, and by those moments four and seven resources have already been created. By the time the run ends, the infrastructure is half-built: the queue exists, the data store that feeds it does not. The only way to move this class into static checking is to bring provider state into the record — a new record, a new source of drift. ## Summary - The fictional measurement network's record holds eleven resources across four files; sixteen known defects were injected into a copy, and three static checks were run over the real files. - Format produced 6 findings (5 catches-alone), syntax 5 (3), type/reference 7 (6); together the three caught 16 of 16 defects, and the broadest check alone still missed 9. - Overlap sits in two defects: a missing colon trips both format and syntax, an empty value trips both syntax and type. - The cost is in scope, not comparisons: format makes 384 comparisons, syntax 192, type/reference 131, but when a single file changes the first two read 1 file and type/reference reads 4. - The class it cannot see is **defects that depend on provider truth** (quota, region catalog, a reserved name, permission); on a record that passed with zero findings, four of them showed up during apply (the first at second 90, the last at second 240) and left behind a half-built infrastructure with 7 created resources. ## Next Step This lesson's checks tested the record's **own rules**. None of them tested a rule like "this resource must not face the open network"; rules like that come from outside and concern not the record's correctness but its **acceptability**. The next lesson runs a policy set over the same record: should a policy run on the **definition**, or on the **plan**? The two scans are compared on the same set of violations — which one sees computed values, and how many violations an exemption opens up.