Policy and Compliance Scanning

The previous lesson tested the record’s correctness: does the format hold, is the syntax valid, do references resolve. All of that was the record’s own rules. But a flawless record can still be unacceptable — an archive with encryption turned off, a machine facing the open network, or a data store with a three-day backup are all flawless as far as syntax is concerned.

Policy scanning runs externally sourced rules over the record. The caught/missed/false-alarm triple was already established when the fitness function was measured over architectural rules; since its mechanics were measured there, they are not repeated here — what changes is the object being measured. This lesson’s question is: should a policy run on the definition, or on the plan? The definition is early and never connects to a provider, but it sees only what is written; the plan is late and needs a provider call, but it sees the resolved value. The two scans run the same policy set over the same record, and four things are counted: caught, missed, false alarms, and the opening an exemption creates. The through-line is the regional measurement network, and it is fictional.

Written Value vs. Resolved Value

Not every value written in a record reads the way it is written. Text and reality split apart in four places: a variable (the value is defined on another line), a provider default (the field is left out, and the provider supplies the value), set-based iteration (a single block expands into several instances), and a condition (the block is never created in a given environment). The question of which layer a policy should run on exists precisely because of these four.

IV7 — the measurement network’s production environment is held in a single record file and contains all four (fictional). IV8 — the provider defaults are fictional: if a machine’s public-network field is left out, yes comes back; if an object store’s encryption field is left out, no comes back. IV9 — values assigned at apply time stand in the record as @pending and are not resolved in the plan either.

// record.mjs — writes the measurement network's production-environment record; contains variables,
// provider defaults, count-based iteration, and conditions (fictional)
import { writeFileSync } from "node:fs";

const RECORD = `variable environment = production
variable owner = measurement-team
variable backup = 3

resource network/backbone-west:
  region = west
  block  = 10.20.0.0/16
  owner = $owner

resource network/backbone-external:
  region = west
  block  = 198.51.100.0/24
  owner = $owner

resource machine/collector:
  iterate      = west,east,north
  region       = $iterate
  public_ip    = yes
  public_address = @pending
  owner        = $owner

resource machine/gateway:
  region       = west
  public_ip    = yes
  public_address = @pending
  owner        = $owner
  exemption    = P2:2026-03-31

resource machine/archiver:
  region       = west
  public_ip    = @default
  public_address = @pending
  owner        = $owner

resource machine/trial:
  condition    = environment=staging
  region       = west
  public_ip    = yes
  public_address = @pending
  owner        = $owner

resource data_store/measurement-store:
  region    = west
  backup    = $backup
  owner     = $owner
  exemption = *:2026-06-30

resource queue/measurement-queue:
  region   = west
  retention = 720

resource queue/trial-queue:
  condition = environment=staging
  region   = west
  retention = 720
  owner    = $owner

resource object_store/measurement-archive:
  region    = west
  encrypted = @default
  owner     = $owner
  exemption = P1:2025-11-30
`;

writeFileSync("record.def", RECORD);
const count = (k) => (RECORD.match(k) ?? []).length;
console.log(`record.def: ${RECORD.split("\n").length - 1} lines, ${count(/^variable /gm)} variables, ` +
  `${count(/^resource /gm)} blocks\n`);
console.log(`values to resolve: ${count(/= \$/gm)} variable references, ` +
  `${count(/@default/g)} provider defaults, ${count(/@pending/g)} assigned at apply; ` +
  `${count(/^  iterate /gm)} block using iteration, ${count(/^  condition /gm)} conditional blocks, ` +
  `${count(/^  exemption /gm)} exemption records`);

record.def: 62 lines, 3 variables, 10 blocks

values to resolve: 11 variable references, 2 provider defaults, 4 assigned at apply; 1 block using iteration, 2 conditional blocks, 3 exemption records

Ten blocks are written but ten blocks do not enter the plan: one expands into three instances through iteration, two never enter because their condition does not hold. At the definition layer, seventeen values still sit unresolved.

Two Scans

IV10 — the policy set is seven rules and is team-accepted (fictional). P1: an object store must be encrypted, P2: a machine must not face the open network, P3: a data store’s backup must be at least seven days, P4: a queue’s retention must be at most 168 hours, P5: every resource’s owner must be written, P6: a network’s block must be in a private range, P7: a machine’s public address must be in a registered range. IV11 — an exemption record is tied to a policy code (or the * wildcard) and an expiry date; today is taken to be 2026-02-15 (fictional). IV12 — plan generation makes one provider call per kind; a call is four seconds, plan computation is one second per instance (fictional).

Every policy declares which field it reads. If a scan cannot resolve that field’s value — the value starts with $ or @ — the rule stays undecided and counts as neither a violation nor compliant. The plan scan sees reality; the definition scan is compared against it.

// scan.mjs — the same policy set runs first on the definition, then on the plan (model)
import { readFileSync } from "node:fs";

const TODAY = "2026-02-15";
const DEFAULT = { "machine.public_ip": "yes", "object_store.encrypted": "no" }; // fictional
// [code, rule, field the rule reads (null: does not look at a value)]
const POLICY = [
  ["P1", (t, a) => t !== "object_store" || a.encrypted === "yes", "encrypted"],
  ["P2", (t, a) => t !== "machine" || a.public_ip === "no", "public_ip"],
  ["P3", (t, a) => t !== "data_store" || Number(a.backup) >= 7, "backup"],
  ["P4", (t, a) => t !== "queue" || Number(a.retention) <= 168, "retention"],
  ["P5", (t, a) => a.owner !== undefined, null],
  ["P6", (t, a) => t !== "network" || a.block.startsWith("10."), "block"],
  ["P7", (t, a) => t !== "machine" || a.public_address.startsWith("10."), "public_address"],
];

const VARIABLE = {}, BLOCK = [];
let current = null;
for (const s of readFileSync("record.def", "utf8").split("\n")) {
  const d = /^variable (\w+) = (.+)$/.exec(s), h = /^resource ([a-z_]+)\/([a-z0-9-]+):$/.exec(s);
  const f = /^\s+(\w+)\s*=\s*(.+)$/.exec(s);
  if (d) VARIABLE[d[1]] = d[2];
  else if (h) BLOCK.push((current = { kind: h[1], name: h[2], field: {} }));
  else if (f && current) current.field[f[1]] = f[2].trim();
}

// plan: the condition is evaluated, iteration is expanded, variables and provider defaults are
// resolved; "@pending" is not resolved, because its value is assigned at apply time
const PLAN = [];
for (const b of BLOCK) {
  const [k, v] = (b.field.condition ?? "=").split("=");
  if (b.field.condition && VARIABLE[k] !== v) continue;
  for (const item of b.field.iterate ? b.field.iterate.split(",") : [null]) {
    const field = {};
    for (const [fieldName, fieldValue] of Object.entries(b.field)) {
      if (["iterate", "condition", "exemption"].includes(fieldName)) continue;
      field[fieldName] = fieldValue === "@default" ? DEFAULT[`${b.kind}.${fieldName}`]
        : fieldValue === "$iterate" ? item
          : fieldValue.startsWith("$") ? VARIABLE[fieldValue.slice(1)] : fieldValue;
    }
    PLAN.push({ kind: b.kind, block: b.name, name: item ? `${b.name}-${item}` : b.name, field,
      exemption: b.field.exemption });
  }
}

const unresolved = (v) => v !== undefined && /^[$@]/.test(v);
const isExempt = (n, code) => {
  const [p, date] = (n.exemption ?? ":").split(":");
  return (p === "*" || p === code) && date >= TODAY;
};
function scan(items) {
  const finding = [];
  let undecided = 0;
  for (const n of items) for (const [code, rule, field] of POLICY) {
    if (field && unresolved(n.field[field])) { undecided += 1; continue; }
    if (!rule(n.kind, n.field)) finding.push({ name: n.name, block: n.block, code, exempt: isExempt(n, code) });
  }
  return { finding, undecided };
}

const definitionObjects = BLOCK.map((b) => ({ kind: b.kind, name: b.name, block: b.name, field: b.field,
  exemption: b.field.exemption }));
const defScan = scan(definitionObjects), planScan = scan(PLAN);
const sameFinding = (x, y) => x.block === y.block && x.code === y.code;
const caught = planScan.finding.filter((p) => defScan.finding.some((t) => sameFinding(t, p)));
const falseAlarm = defScan.finding.filter((t) => !planScan.finding.some((p) => sameFinding(t, p)));
const DURATION = new Set(BLOCK.map((b) => b.kind)).size * 4 + PLAN.length; // IV12: fictional seconds

console.log(`${"scan".padEnd(12)}${"value seen".padEnd(15)}${"findings".padStart(10)}` +
  `${"caught".padStart(11)}${"missed".padStart(8)}${"false alarms".padStart(14)}` +
  `${"exempt".padStart(8)}${"reported".padStart(12)}${"undecided".padStart(11)}${"ready".padStart(8)}`);
for (const [name, value, S, yak, yan, dur] of [
  ["definition", "written", defScan, caught.length, falseAlarm.length, 0],
  ["plan", "resolved", planScan, planScan.finding.length, 0, DURATION]]) {
  const exempt = S.finding.filter((b) => b.exempt).length;
  console.log(`${name.padEnd(12)}${value.padEnd(15)}${String(S.finding.length).padStart(10)}` +
    `${`${yak}/${planScan.finding.length}`.padStart(11)}${String(planScan.finding.length - yak).padStart(8)}` +
    `${String(yan).padStart(14)}${String(exempt).padStart(8)}` +
    `${String(S.finding.length - exempt).padStart(12)}${String(S.undecided).padStart(11)}` +
    `${`${dur} s`.padStart(8)}`);
}

console.log(`\nviolations only the plan sees:`);
for (const p of planScan.finding.filter((p) => !defScan.finding.some((t) => sameFinding(t, p)))) {
  const field = POLICY.find(([k]) => k === p.code)[2];
  console.log(`  ${p.code}  ${p.name.padEnd(21)}${field.padEnd(12)}definition has ` +
    `"${BLOCK.find((b) => b.name === p.block).field[field]}", plan has "${PLAN.find((n) => n.name === p.name).field[field]}"`);
}

const EXEMPTION = BLOCK.filter((b) => b.field.exemption).map((b) => ({ name: `${b.kind}/${b.name}`,
  record: b.field.exemption, inEffect: b.field.exemption.split(":")[1] >= TODAY }));
console.log(`\n${"exemption".padEnd(35)}${"record".padEnd(16)}${"status".padEnd(14)}` +
  `${"policies closed".padStart(19)}${"violations dropped".padStart(21)}`);
for (const e of EXEMPTION) {
  const dropped = planScan.finding.filter((b) => b.exempt && `${PLAN.find((n) => n.name === b.name).kind}/${b.block}` === e.name);
  const coverage = e.inEffect ? (e.record.startsWith("*") ? POLICY.length : 1) : 0;
  console.log(`${e.name.padEnd(35)}${e.record.padEnd(16)}` +
    `${(e.inEffect ? "in effect" : "expired").padEnd(14)}` +
    `${String(coverage).padStart(19)}${String(dropped.length).padStart(21)}`);
}
console.log(`\ndefinition scan's false alarms: ` +
  falseAlarm.map((y) => `${y.code} ${y.name}`).join(", ") + ` (condition is false, never enters the plan); ` +
  `the policy neither scan can decide is P7 — public_address is assigned at apply time`);

scan        value seen       findings     caught  missed  false alarms  exempt    reported  undecided   ready
definition  written                 7       7/10       3             2       1           6          7     0 s
plan        resolved               10      10/10       0             0       2           8          5    30 s

violations only the plan sees:
  P2  archiver             public_ip   definition has "@default", plan has "yes"
  P3  measurement-store    backup      definition has "$backup", plan has "3"
  P1  measurement-archive  encrypted   definition has "@default", plan has "no"

exemption                          record          status            policies closed   violations dropped
machine/gateway                    P2:2026-03-31   in effect                       1                    1
data_store/measurement-store       *:2026-06-30    in effect                       7                    1
object_store/measurement-archive   P1:2025-11-30   expired                         0                    0

definition scan's false alarms: P2 trial, P4 trial-queue (condition is false, never enters the plan); the policy neither scan can decide is P7 — public_address is assigned at apply time

Who Sees What

The plan scan finds ten violations; the definition scan catches seven of them, misses three, and adds two false alarms on top. What the missed three have in common is spelled out one by one in the second part of the output: all three are computed values. The archiver machine’s public-network field reads @default in the definition and becomes yes in the plan; the data store’s backup reads $backup in the definition and becomes 3 in the plan; the archive’s encryption field reads @default in the definition and becomes no in the plan. In all three, the record text does not hide the violation — it only fails to show it; the value lives elsewhere in the record, or at the provider.

The reason for the two false alarms is the other end of the same asymmetry: two blocks whose condition does not hold never enter the plan at all, but they still sit in the definition text with their violating values. The definition scan reports them, and two of the six reported findings — one in three — belong to resources that will never be created.

Running the two scans together is not free either. Four of the definition report’s six lines and six of the plan report’s eight lines describe the same four defects; the team sees the same violation in two separate reports and has to keep a third record just to track which one has been closed. For a pipeline that has to put only one of them at the gate, the question is not “which one is more correct” but which failure mode is more expensive: a false alarm burns human time and ends with the rule being waved through, while a missed violation ships to production.

A third difference is hidden in the numbers. The definition scan produces seven findings, but only five of them are real violations; those five findings cover the plan’s seven violations, because the one block using iteration expands into three instances. The same defect is reported once in the definition and three times in the plan. The fix is a single line either way; what differs is the report’s length and how the count should be read.

Earliness vs. Accuracy

The table’s last two columns are this lesson’s trade-off. The definition scan is ready in zero seconds: it reads a single file, connects to no provider, and can run every time the record changes. The plan scan waits thirty seconds — one provider call per resource kind, five of them, then computation for ten plan instances — and only runs in an environment that can produce a plan; it cannot run at all on a contribution pipeline with no such permission.

In practice, the two also have different places to live. The definition scan can run where the record is written, before the contribution is even merged; a false alarm is cheapest there, because the person reading the report is the same person who knows the block is conditional. The plan scan, by contrast, sits at the gate where the plan is generated, and the decision it makes there is binding.

What it gets in exchange is accuracy: zero missed, zero false alarms. What follows from this is not “which one is better” but that the two are different failure modes. The definition scan produces false alarms and misses computed values; the plan scan arrives late and depends on a provider connection. The undecided column shows the shared boundary of both: the definition cannot decide on seven rules, the plan on five, and the reason neither can decide on P7 is the same — the public address is assigned at apply time and is not written in either layer.

Exemption’s Opening

There are three exemption records, and the three show three separate behaviors. The gateway’s exemption is tied to a single policy and is in effect: it drops one violation from the report, and the gateway’s other six policies stay open. The data store’s exemption is a wildcard: today it still drops only one violation, but the number of policies it closes is seven — every violation later added to that resource passes silently too, until the exemption expires. The archive’s exemption has expired and drops zero violations; encryption being turned off has come back into the report.

The exemption itself is a record too, and no one owns its upkeep. One of the three records sits expired; no one has deleted it. An expired exemption is visible — it brings the violation back into the report and announces itself. The dangerous end is the other one: an exemption that has not expired and has been forgotten is silent, shortens the report, and leaves the violation in place. The only thing that audits an exemption record is the date; no rule audits a wildcard record’s scope.

As a number: the plan scan found ten violations, two were dropped by exemption, eight were reported. The opening an exemption creates is two violations wide, and one of those is the opening the wildcard record creates. An exemption’s cost is measured not by the number of violations it drops but by the number of policies it closes: an exemption tied to one policy opens one gate, a wildcard exemption opens seven at once.

Summary

• The same seven-rule policy set was run over two layers of the fictional measurement network’s record: the written definition text and the resolved plan.
• The plan scan found 10 violations; the definition scan caught 7, missed 3, and produced 2 false alarms — 2 of the 6 findings it reported belonged to resources that would never be created.
• All three missed violations are computed values: two are provider defaults (@default → yes and no), one is a variable ($backup → 3); the definition text does not show them.
• Earliness is traded for accuracy: the definition scan is ready in 0 seconds without connecting to a provider, the plan scan takes 30 seconds and needs five provider calls; the value of P7, which neither can decide, is assigned at apply time (7 rules undecided in the definition, 5 in the plan).
• Two of the three exemption records are in effect and drop 2 of the 10 violations from the report; the wildcard exemption alone closes 7 policies at once while the policy-tied exemption closes 1, and the expired exemption brings its violation back into the report.

Next Step

Both scans read the record — one as it is written, the other as it is resolved. What neither can see is the resource’s running state: P7 stayed undecided because the public address is assigned at apply time, and no rule in the policy set asks “does this machine actually come up, does the message written to the queue reach the data store.” The next lesson asks that question at three levels: unit validation that looks at the infrastructure definition, contract validation that tests the agreement between two resources, and end-to-end validation that creates real resources. Over the same defect set, what each of the three catches, its duration, and its cost are all counted — and the common gap all three miss together is named.