Format and Validation Checks

The record is written: the infrastructure is defined in a configuration language, split across files, and the repeated parts are turned into modules. What has been measured so far is the record’s structure; what has never been measured is its correctness. When these files are applied, does the desired infrastructure come out, or does the apply stop halfway? If the question is not asked before applying, the apply itself answers it, and the cost of that answer is a resource that has already been created.

Static checking reads the record before it is applied: it touches no resource, connects to no provider, and reads only the text on disk. This lesson writes three static checks and runs them over fictional definition files — format, syntax, and type-and-reference consistency. For each check, this lesson counts what it catches and catches alone in the known defect set, what it misses, and its run cost; at the end, the defect class static checking cannot see is measured against the apply. The through-line is the regional measurement network, and it is fictional.

Record and Known Defect Set

IV1 — the production environment of the measurement network is held in four definition files across eleven resources: three networks, five machines, one queue, one data store, one object store (fictional). IV2 — the configuration language’s format is our own small format: a block header at column 0, below it two-space-indented fields, = a plain value, -> a reference. IV3 — the known defect set is sixteen defects, hand-marked; each defect is injected into a specific line of the clean record, and which defect a finding corresponds to is read from the line number.

// setup.mjs — writes the measurement network's record, injects known defects into a copy (fictional)
import { mkdirSync, writeFileSync } from "node:fs";

// format: "resource <kind>/<name>:" at column 0, below it two-space-indented field lines;
// "=" a plain value, "->" a reference to another resource, "#" a comment, one blank line between blocks
const block = (kind, name, fields) => `resource ${kind}/${name}:\n` +
  fields.map(([a, y, d]) => `  ${a.padEnd(10)}${y} ${d}`).join("\n") + "\n";
const MACHINE = [["west-1", "west", "medium", 40], ["west-2", "west", "medium", 40],
  ["west-3", "west", "large", 80], ["east-1", "east", "medium", 40], ["east-2", "east", "large", 80]];
const FILE = {
  "backbone.def": ["west", "east", "north"].map((b, i) => block("network", `backbone-${b}`,
    [["region", "=", b], ["block", "=", `10.2${i}.0.0/16`], ["private", "=", "yes"]])).join("\n"),
  "machine.def": MACHINE.map(([n, b, sz, d]) => block("machine", `collector-${n}`,
    [["region", "=", b], ["size", "=", sz], ["disk_gb", "=", d],
      ["network", "->", `network/backbone-${b}`]])).join("\n"),
  "flow.def": block("queue", "measurement-queue", [["region", "=", "west"], ["retention", "=", 72],
    ["network", "->", "network/backbone-west"]]) + "\n" + block("data_store", "measurement-store",
    [["region", "=", "west"], ["size", "=", "large"], ["disk_gb", "=", 500], ["backup", "=", 14],
      ["network", "->", "network/backbone-west"]]),
  "hold.def": block("object_store", "measurement-archive", [["region", "=", "west"],
    ["versioned", "=", "yes"], ["class", "=", "cold"]]),
};

// known defect set: [code, file, line in the clean record, replacement, defect's location]
const DEFECT = [
  ["D01", "backbone.def", 2, "  region    = west  "],
  ["D02", "backbone.def", 9, "  private   = open"],
  ["D03", "machine.def", 3, "   size      = medium"],
  ["D04", "machine.def", 5, "  network   -> object_store/measurement-archive"],
  ["D05", "machine.def", 10, "  disk_gb=40"],
  ["D06", "machine.def", 16, "  disk_gb   = eighty"],
  ["D07", "machine.def", 18, "@delete", 19],
  ["D08", "machine.def", 22, "  disk_gb   = 40\n  disk_gb   = 60"],
  ["D09", "machine.def", 23, "  network   -> network/backbone-south"],
  ["D10", "machine.def", 25, "resource machine/collector-east-1:"],
  ["D11", "backbone.def", 11, "resource network/backbone-north"],
  ["D12", "flow.def", 2, "  region    = west\n  regions   = west,east"],
  ["D13", "flow.def", 3, "\tretention = 72"],
  ["D14", "flow.def", 8, "  size      ="],
  ["D15", "flow.def", 11, "  # network connection added later", 6],
  ["D16", "hold.def", 4, "  class     = cold\n  label measurement archive"],
];

mkdirSync("record", { recursive: true });
mkdirSync("draft", { recursive: true });
const location = [];
for (const [file, content] of Object.entries(FILE)) {
  writeFileSync(`record/${file}`, content);
  const patch = new Map(DEFECT.filter((k) => k[1] === file).map((k) => [k[2], k[3]]));
  const newLines = [], lineMap = {};
  content.split("\n").forEach((s, i) => { // a defect can delete a line or add lines
    const d = patch.get(i + 1);
    if (d === "@delete") return;
    newLines.push(...(d ?? s).split("\n"));
    lineMap[i + 1] = newLines.length; // clean-record line -> its final counterpart in the draft
  });
  writeFileSync(`draft/${file}`, newLines.join("\n"));
  for (const [code, , line, , k] of DEFECT.filter((x) => x[1] === file))
    location.push({ code, file, line: lineMap[k ?? line] });
}
writeFileSync("defects.json", JSON.stringify(location));

const text = Object.values(FILE).join("");
console.log(`record: ${Object.keys(FILE).length} files, ${text.match(/^resource /gm).length} resources, ` +
  `${text.split("\n").length} lines; draft: same record + ${DEFECT.length} known defects\n`);
console.log(FILE["flow.def"].split("\n").slice(0, 4).join("\n"));

record: 4 files, 11 resources, 59 lines; draft: same record + 16 known defects

resource queue/measurement-queue:
  region    = west
  retention = 72
  network   -> network/backbone-west

Three Static Checks

IV4 — the format rules are six in number and team-accepted. IV5 — every resource kind’s fields, types, and requirements live in a schema kept alongside the record. All three produce findings in the same shape — file, line, rule — but their contexts differ: format looks at a single line, syntax looks at the line and its block, type/reference wants the whole record, because a machine’s network field points to a resource in another file.

// check.mjs — same record, three static checks: format, syntax, type/reference
import { readFileSync, readdirSync } from "node:fs";

const SCHEMA = { // kind -> "field:type" list; "!" required, "@t" a reference to a resource of kind t
  network: "region:text! block:text! private:boolean!",
  machine: "region:text! size:text! disk_gb:number! network:@network!",
  queue: "region:text! retention:number! network:@network!",
  data_store: "region:text! size:text! disk_gb:number! backup:number! network:@network!",
  object_store: "region:text! versioned:boolean! class:text!",
};
const schemaFields = (kind) => Object.fromEntries(SCHEMA[kind].split(" ").map((x) => x.split(":")));
const HEADER = /^resource ([a-z_]+)\/([a-z0-9-]+):$/, FIELD = /^\s+([a-z_]+)\s*(=|->)\s*(.*)$/;
const read = (dir) => readdirSync(dir).filter((d) => d.endsWith(".def"))
  .map((d) => [d, readFileSync(`${dir}/${d}`, "utf8").split("\n")]);
const counts = { format: 0, syntax: 0, type: 0 };

// 1) format: six line-local rules; reads nothing but the single line
const R = [
  ["R1", (s) => s.length && s !== s.trimEnd()],                          // trailing whitespace
  ["R2", (s) => /^ *\t/.test(s)],                                        // tab in the indent
  ["R3", (s) => /^ +\S/.test(s) && !/^ {2}\S/.test(s)],                  // indent is not two spaces
  ["R4", (s) => /=|->/.test(s) && !/^\s*#/.test(s) && !/ (=|->)( |$)/.test(s)],  // no space around separator
  ["R5a", (s) => /^[^\s#]/.test(s) && !s.endsWith(":")],                 // header does not end with a colon
  ["R5b", (s, i, L) => /^[^\s#]/.test(s) && i > 0 && L[i - 1] !== ""],   // no blank line before block
];
function format(files) {
  const b = [];
  for (const [d, L] of files) L.forEach((s, i) => R.forEach(([k, f]) => {
    counts.format += 1;
    if (f(s, i, L)) b.push({ file: d, line: i + 1, rule: k });
  }));
  return b;
}

// 2) syntax: line- and block-local; never looks outside its own file
function syntax(files) {
  const b = [], seen = new Map();
  for (const [d, L] of files) {
    let keys = null;
    L.forEach((s, i) => {
      const E = (k) => b.push({ file: d, line: i + 1, rule: k });
      counts.syntax += 3;
      if (s === "" || /^\s*#/.test(s)) return;
      if (/^\S/.test(s)) {
        const m = HEADER.exec(s);
        keys = null;
        if (!m) return E("P1 header pattern does not match");
        keys = new Set();
        const path = `${m[1]}/${m[2]}`;
        if (seen.has(path)) E(`P4 duplicate resource name (${seen.get(path)})`);
        else seen.set(path, `${d}:${i + 1}`);
        return;
      }
      const m = FIELD.exec(s);
      if (!m) return E("P2 unrecognized line");
      if (m[3].trim() === "") return E("P2 no value");
      if (keys?.has(m[1])) E(`P3 duplicate key (${m[1]})`);
      keys?.add(m[1]);
    });
  }
  return b;
}

// 3) type/reference: wants the whole record, because references cross file boundaries
function typeRef(files) {
  const resources = [], b = [];
  for (const [d, L] of files) {
    let current = null;
    L.forEach((s, i) => {
      const m = /^\S/.test(s) ? HEADER.exec(s) : FIELD.exec(s);
      if (/^\S/.test(s)) current = m ? { kind: m[1], name: m[2], file: d, line: i + 1, field: {} } : null;
      else if (current && m) current.field[m[1]] = { value: m[3].trim(), line: i + 1 };
      else return;
      if (/^\S/.test(s) && current) resources.push(current);
    });
  }
  const literalType = (v) => (["yes", "no"].includes(v) ? "boolean" : /^-?\d+$/.test(v) ? "number" : "text");
  for (const k of resources) {
    const schema = schemaFields(k.kind);
    for (const [a, v] of Object.entries(k.field)) {
      const E = (rule) => b.push({ file: k.file, line: v.line, rule });
      counts.type += 2;
      if (!schema[a]) { E(`C1 not in schema (${a})`); continue; }
      const t = schema[a].replace("!", "");
      if (!t.startsWith("@")) {
        if (v.value === "" || literalType(v.value) !== t) E(`C2 not ${t} ("${v.value}")`);
        continue;
      }
      let h = null;
      for (const x of resources) { counts.type += 1; if (`${x.kind}/${x.name}` === v.value) { h = x; break; } }
      if (!h) E(`C4 target does not exist (${v.value})`);
      else if (h.kind !== t.slice(1)) E(`C5 wrong kind (${h.kind})`);
    }
    for (const [a, t] of Object.entries(schema)) {
      counts.type += 1;
      if (t.endsWith("!") && !k.field[a])
        b.push({ file: k.file, line: k.line, rule: `C3 required field missing (${a})` });
    }
  }
  return b;
}

const draft = read("draft"), clean = read("record");
const LEVEL = [["format", format, 1], ["syntax", syntax, 1], ["type/reference", typeRef, 4]];
const finding = LEVEL.map(([ad, f]) => [ad, f(draft)]);
const totals = [counts.format, counts.syntax, counts.type]; // counters are read only from the draft run
const DEFECT = JSON.parse(readFileSync("defects.json", "utf8"));
const code = (f) => DEFECT.find((x) => x.file === f.file && x.line === f.line)?.code ?? "??";

console.log(`${"check".padEnd(16)}${"code".padEnd(6)}${"location".padEnd(20)}rule`);
for (const [ad, l] of finding) for (const f of l)
  console.log(`${ad.padEnd(16)}${code(f).padEnd(6)}${`${f.file}:${f.line}`.padEnd(20)}${f.rule}`);

const caught = finding.map(([, l]) => new Set(l.map(code)));
console.log(`\n${"check".padEnd(16)}${"findings".padStart(10)}${"caught".padStart(9)}` +
  `${"catches alone".padStart(16)}${"missed".padStart(11)}${"comparisons".padStart(15)}` +
  `${"files read".padStart(16)}`);
finding.forEach(([ad, l], j) => {
  const y = [...caught[j]].filter((k) => caught.every((s, o) => o === j || !s.has(k))).length;
  console.log(`${ad.padEnd(16)}${String(l.length).padStart(10)}${String(caught[j].size).padStart(9)}` +
    `${String(y).padStart(16)}${String(DEFECT.length - caught[j].size).padStart(11)}` +
    `${String(totals[j]).padStart(15)}${String(LEVEL[j][2]).padStart(16)}`);
});
console.log(`\nthe three checks together catch ${DEFECT.length} of the known defects, ` +
  `${new Set(caught.flatMap((s) => [...s])).size}; findings matching no defect ` +
  `${finding.flatMap(([, l]) => l).filter((f) => code(f) === "??").length}; ` +
  `total findings on the clean record ${LEVEL.reduce((s, [, f]) => s + f(clean).length, 0)}`);

check           code  location            rule
format          D01   backbone.def:2      R1
format          D11   backbone.def:11     R5a
format          D13   flow.def:4          R2
format          D03   machine.def:3       R3
format          D05   machine.def:10      R4
format          D07   machine.def:18      R5b
syntax          D11   backbone.def:11     P1 header pattern does not match
syntax          D14   flow.def:9          P2 no value
syntax          D16   hold.def:5          P2 unrecognized line
syntax          D08   machine.def:22      P3 duplicate key (disk_gb)
syntax          D10   machine.def:25      P4 duplicate resource name (machine.def:18)
type/reference  D02   backbone.def:9      C2 not boolean ("open")
type/reference  D12   flow.def:3          C1 not in schema (regions)
type/reference  D14   flow.def:9          C2 not text ("")
type/reference  D15   flow.def:7          C3 required field missing (network)
type/reference  D04   machine.def:5       C5 wrong kind (object_store)
type/reference  D06   machine.def:16      C2 not number ("eighty")
type/reference  D09   machine.def:23      C4 target does not exist (network/backbone-south)

check             findings   caught   catches alone     missed    comparisons      files read
format                   6        6               5         10            384               1
syntax                   5        5               3         11            192               1
type/reference           7        7               6          9            131               4

the three checks together catch 16 of the known defects, 16; findings matching no defect 0; total findings on the clean record 0

Coverage, Overlap, and Cost

The three checks together catch all sixteen of the sixteen defects, but none is sufficient alone: even type/reference, the broadest in coverage, misses nine defects. Overlap happens in two places — the missing colon in a header (D11) trips both format and syntax, and a value left empty (D14) trips both syntax and type. The remaining fourteen defects are each one check’s own catch; the “catches alone” column reads 5, 3, and 6.

The comparison counts say the opposite of what intuition suggests: the check that makes the most comparisons wants the least context. Format makes 384 comparisons (64 lines times six rules), type/reference only 131. The cost is not in the comparisons but in the scope: when a single file changes, the first two checks can read that file and decide; type/reference has to read all four. What is more, because reference resolution scans the resource list from the start, the type check’s comparisons grow quadratically with the number of resources. Two quiet numbers sit in the table as well: findings matching no defect, and total findings on the clean record, are both zero.

The Class Static Checking Cannot See

The common name for the defects none of the three checks can see is defects that depend on provider truth: the state that decides is not in the record — the region’s quota, the machine sizes available in the region, a name held in the name pool, permission to create that resource kind. None of it is written in the text.

IV6 — the provider’s quota, region-size catalog, name pool, permissions, and resource-creation durations are fictional. The block below applies the clean record — the one static checking scores zero findings on — to this model; items are processed in the order the record is read, and a rejected item is skipped.

// apply.mjs — the clean record is applied to a provider model; the provider and durations are fictional
import { readFileSync, readdirSync } from "node:fs";

const PROVIDER = {
  quota: { west: 2, east: 4 },                             // machine ceiling per region
  catalog: { west: ["medium", "large"], east: ["medium"] }, // sizes available in the region
  reservedName: ["measurement-archive"],                    // name is globally unique
  permission: { network: 1, machine: 1, queue: 1, data_store: 0, object_store: 1 },
};
const DURATION = { network: 20, machine: 45, queue: 30, data_store: 180, object_store: 15, reject: 5 };

const resource = [];
for (const d of readdirSync("record")) for (const s of readFileSync(`record/${d}`, "utf8").split("\n")) {
  const h = /^resource ([a-z_]+)\/([a-z0-9-]+):$/.exec(s), a = /^\s+([a-z_]+)\s*=\s*(.+)$/.exec(s);
  if (h) resource.push({ kind: h[1], name: h[2], field: {} });
  else if (a) resource.at(-1).field[a[1]] = a[2].trim();
}

let seconds = 0, created = 0;
const rejected = [], counted = {};
for (const k of resource) {
  let error = null;
  if (!PROVIDER.permission[k.kind]) error = "permission denied";
  else if (PROVIDER.reservedName.includes(k.name)) error = "name reserved";
  else if (k.kind === "machine") {
    counted[k.field.region] = (counted[k.field.region] ?? 0) + 1;
    if (counted[k.field.region] > PROVIDER.quota[k.field.region]) error = "quota exceeded";
    else if (!PROVIDER.catalog[k.field.region].includes(k.field.size)) error = "not in catalog";
  }
  if (error) rejected.push({ name: `${k.kind}/${k.name}`, error, seconds, created });
  else created += 1;
  seconds += error ? DURATION.reject : DURATION[k.kind];
}

console.log(`${resource.length} items applied: ${created} created, ${rejected.length} rejected, ` +
  `${seconds} seconds total\n`);
console.log(`${"rejected item".padEnd(35)}${"reason".padEnd(19)}${"at second".padStart(17)}` +
  `${"created so far".padStart(21)}`);
for (const r of rejected) console.log(`${r.name.padEnd(35)}${r.error.padEnd(19)}` +
  `${String(r.seconds).padStart(17)}${String(r.created).padStart(21)}`);

11 items applied: 7 created, 4 rejected, 245 seconds total

rejected item                      reason                     at second       created so far
data_store/measurement-store       permission denied                 90                    4
object_store/measurement-archive   name reserved                     95                    4
machine/collector-west-3           quota exceeded                   190                    6
machine/collector-east-2           not in catalog                   240                    7

All four defects are in the record that static checking scored as clean; the difference is in time and cost. The three checks scan the 64 lines with 707 comparisons and deliver their result without creating a single resource; the first provider-dependent defect shows up at second 90, the last at second 240, and by those moments four and seven resources have already been created. By the time the run ends, the infrastructure is half-built: the queue exists, the data store that feeds it does not. The only way to move this class into static checking is to bring provider state into the record — a new record, a new source of drift.

Summary

• The fictional measurement network’s record holds eleven resources across four files; sixteen known defects were injected into a copy, and three static checks were run over the real files.
• Format produced 6 findings (5 catches-alone), syntax 5 (3), type/reference 7 (6); together the three caught 16 of 16 defects, and the broadest check alone still missed 9.
• Overlap sits in two defects: a missing colon trips both format and syntax, an empty value trips both syntax and type.
• The cost is in scope, not comparisons: format makes 384 comparisons, syntax 192, type/reference 131, but when a single file changes the first two read 1 file and type/reference reads 4.
• The class it cannot see is defects that depend on provider truth (quota, region catalog, a reserved name, permission); on a record that passed with zero findings, four of them showed up during apply (the first at second 90, the last at second 240) and left behind a half-built infrastructure with 7 created resources.

Next Step

This lesson’s checks tested the record’s own rules. None of them tested a rule like “this resource must not face the open network”; rules like that come from outside and concern not the record’s correctness but its acceptability. The next lesson runs a policy set over the same record: should a policy run on the definition, or on the plan? The two scans are compared on the same set of violations — which one sees computed values, and how many violations an exemption opens up.