Twelve-Factor App

The twelve factors name the contract between whoever writes an application and whoever runs it. What the factors say was built in the Server-Side Fundamentals course and is not repeated here. This lesson’s question is different: can a factor be turned into a runnable check that looks at files on disk and says yes or no? If it can, what does the check catch, what does it miss, what does it raise a false alarm over? If it cannot, what replaces it?

A factor carrying a decision does not depend on its sentence sounding true — it depends on someone seeing it when it is violated. A factor that cannot be turned into a check is only visible when someone thinks to look, and the look that has to be thought of is an unmeasured one. The measurement is done again on the regional measurement network: fictional software that collects water meter readings, verifies them, converts them into invoices, and opens field work orders. The file tree below is genuinely written to disk, the check genuinely runs; the files and the dependency names are all fiction.

DC23. The checker sees only the file tree on disk; it does not see repository history, operating steps, or team habits. DC24. A factor counts as translatable if it can produce a binary decision on a single tree in a single run; a factor that needs a second environment, a second process, or human judgment for its decision counts as untranslatable. DC25. The defects are deliberately planted before the scan, and their count is known; counting what escapes depends on that. DC26. All the checks operate at the level of text and files; there is no syntax-tree parsing. The class of defect that escapes follows directly from this choice.

The Eight Principles Turned Into Checks

The table below does not write what the principle says — it writes what the principle looks at on disk.

Principle	What the check looks at	What decides
Dependencies	is there a lock file next to the declaration file	file existence
Configuration	embedded default and embedded server address in the source	the ?? pattern, address pattern
Port binding	does the source open its own socket	the .listen( call
Logs	is the log written to a file instead of standard output	a file-write call
Disposability	does every declared entry point handle the shutdown signal	the SIGTERM string
Processes	is persistent state kept in process memory	a top-level mutable container
Build, release, run	is an environment value embedded in the build artifact	address pattern
Admin processes	does the one-off job use the application’s code	import path

What the eight share is that the decision can be read from a single tree. The remaining four principles do not appear in this table; the reasons are counted after the measurement.

The third column carries the table’s real information. A principle is a sentence of intent; a check is a text pattern. The translation between them is a narrowing on every line: the sentence “logs are an event stream” is reduced to the rule “no file-write call should appear in the source directory”; the sentence “processes are stateless” to the rule “no top-level mutable container should be declared.” The narrowing produces loss in both directions. Where the rule stays narrower than the principle, a defect escapes; where it reaches into a file the principle does not cover, a false alarm is born. The numbers below measure both losses separately, because a single “pass / fail” total does not say which direction the loss went in.

The Fictional Application’s Files

#!/usr/bin/env bash
# Writes the fictional measurement network application's file tree to disk. Dependency names are fictional too.
set -e
mkdir -p source test script build

cat > package.json <<'SON'
{ "name": "measurement-network", "type": "module",
  "entrypoints": ["source/server.mjs", "script/nightly-fix.mjs"],
  "dependency": { "measurement-resolver": "2.4.0", "invoice-formatter": "1.1.3" } }
SON

cat > source/config.mjs <<'SON'
export const config = {
  env: process.env.ENVIRONMENT_LABEL,
  dataAddress: process.env.MEASUREMENT_DATA_ADDRESS,
  port: Number(process.env.MEASUREMENT_PORT),
  rereadDays: Number(process.env.REREAD_DAYS ?? 30),
};
SON

cat > source/server.mjs <<'SON'
import { createServer } from "node:http";
import { config } from "./config.mjs";
export const server = createServer((request, response) => response.end(config.env ?? "none"));
server.listen(config.port);
SON

cat > source/log.mjs <<'SON'
import { appendFileSync } from "node:fs";
export const log = (k) => appendFileSync("/var/log/measurement.log", JSON.stringify(k) + "\n");
SON

cat > source/report.mjs <<'SON'
import { writeFileSync } from "node:fs";
export const writeInvoice = (f) => writeFileSync(`output/invoice-${f.no}.json`, JSON.stringify(f));
SON

cat > source/batch-job.mjs <<'SON'
const WORK_ORDER_ADDRESS = "work-order-production.internal:8081";
export const queue = (() => {
  const pending = new Map();
  return { add: (id, v) => pending.set(id, v), size: () => pending.size, WORK_ORDER_ADDRESS };
})();
SON

cat > test/fake-measurement.mjs <<'SON'
// In-process fake measurement store; the address does not go to a real destination.
export const fakeAddress = "127.0.0.1:5432";
SON

cat > script/nightly-fix.mjs <<'SON'
const ADDRESS = process.env.MEASUREMENT_DATA_ADDRESS;
console.log(`fix ran -> ${ADDRESS}`);
SON

cat > build/package.mjs <<'SON'
// Build artifact file (fictional): the production address was embedded at compile time.
export const TARGET = "measure-prod.internal:" + 5432;
SON

Eight defects are planted in this tree. Because their count is written inside the checker, what escapes can be counted too; none of the checks reads that list.

The Run

// audit.mjs — runs the translatable principles against the fictional application's real files.
import { readFileSync, existsSync } from "node:fs";

const FILES = ["package.json", "source/config.mjs", "source/server.mjs", "source/log.mjs",
  "source/report.mjs", "source/batch-job.mjs", "test/fake-measurement.mjs",
  "script/nightly-fix.mjs", "build/package.mjs"];
const text = Object.fromEntries(FILES.map((y) => [y, readFileSync(y, "utf8")]));
const under = (prefix) => FILES.filter((y) => y.startsWith(prefix));
const search = (scope, pattern) => scope
  .flatMap((y) => text[y].split("\n").map((s, i) => ({ path: y, lineNo: i + 1, line: s })))
  .filter((b) => pattern.test(b.line));
const note = (path, line) => ({ path, lineNo: 0, line });
const ADDRESS = /["'][\w.-]+:\d{4,5}["']/;

const CHECKS = [
  ["dependencies", () => existsSync("package.lock") ? [] : [note("package.json", "no package.lock")]],
  ["configuration", () => [...search(under("source/"), /process\.env\.\w+\s*\?\?/),
    ...search([...under("source/"), ...under("test/"), ...under("script/")], ADDRESS)]],
  ["port-binding", () => search(under("source/"), /\.listen\(/).length
    ? [] : [note("source/", "no listen call")]],
  ["logs", () => search(under("source/"), /(appendFileSync|writeFileSync|createWriteStream)\(/)],
  ["disposability", () => JSON.parse(text["package.json"]).entrypoints
    .filter((y) => !text[y].includes("SIGTERM")).map((y) => note(y, "no SIGTERM handler"))],
  ["processes", () => search(under("source/"), /^(const|let|var)\s+\w+\s*=\s*new (Map|Set)\(/)],
  ["build-release-run", () => search(under("build/"), ADDRESS)],
  ["admin-processes", () => under("script/").filter((y) => !text[y].includes("source/"))
    .map((y) => note(y, "no import from source/"))],
];

const PLANTED = [                                     // defects planted before the scan
  ["package.json", "package.lock"], ["source/config.mjs", "?? 30"],
  ["source/batch-job.mjs", "work-order-production.internal"], ["source/log.mjs", "appendFileSync"],
  ["source/server.mjs", "SIGTERM"], ["source/batch-job.mjs", "new Map("],
  ["build/package.mjs", "measure-prod.internal:"], ["script/nightly-fix.mjs", "source/"],
];

const findings = CHECKS.flatMap(([name, f]) => f().map((b) => ({ ...b, check: name })));
const real = (b) => PLANTED.some(([y, trace]) => y === b.path && b.line.includes(trace));
const escaped = PLANTED.filter(([y, trace]) => !findings.some((b) => b.path === y && b.line.includes(trace)));
const falsePositives = findings.filter((b) => !real(b));

const row = (a, b, c, d) => a.padEnd(24) + b.padStart(10) + c.padStart(8) + d.padStart(8);
console.log(row("check", "findings", "real", "false"));
for (const [name, f] of CHECKS) {
  const b = f();
  console.log(row(name, `${b.length}`, `${b.filter(real).length}`,
    `${b.filter((x) => !real(x)).length}`));
}
console.log(`translatable principles: ${CHECKS.length}   untranslatable: ${12 - CHECKS.length}`);
console.log(`planted defects: ${PLANTED.length}   reported findings: ${findings.length}`);
console.log(`caught: ${PLANTED.length - escaped.length}   escaped: ${escaped.length}   ` +
  `false positives: ${falsePositives.length}`);
for (const [y, trace] of escaped) console.log(`  [ESCAPED] ${y}  trace=${trace}`);
for (const b of falsePositives) console.log(`  [FALSE] ${b.path}:${b.lineNo}  check=${b.check}`);

check                     findings    real   false
dependencies                     1       1       0
configuration                    3       2       1
port-binding                     0       0       0
logs                             2       1       1
disposability                    2       1       1
processes                        0       0       0
build-release-run                0       0       0
admin-processes                  1       1       0
translatable principles: 8   untranslatable: 4
planted defects: 8   reported findings: 9
caught: 6   escaped: 2   false positives: 3
  [ESCAPED] source/batch-job.mjs  trace=new Map(
  [ESCAPED] build/package.mjs  trace=measure-prod.internal:
  [FALSE] test/fake-measurement.mjs:2  check=configuration
  [FALSE] source/report.mjs:2  check=logs
  [FALSE] script/nightly-fix.mjs:0  check=disposability

Caught, Escaped, False Positive

Six of eight defects were caught, two escaped, and the checks raised three false positives. The rows that actually need reading in the table are the ones reporting zero. port-binding reported zero findings and was right: the source opens its own socket. processes and build-release-run also reported zero, and both were wrong — the defect was sitting right there in both. Zero findings measures the rule’s coverage, not the absence of defects, and the table writes both situations the same way.

The two escaped defects share a class. batch-job.mjs keeps a map in process memory, but the map sits inside a closure: the rule looks for a top-level declaration, and the container is two levels deep. The production address in the build artifact, meanwhile, was concatenated at compile time ("measure-prod.internal:" + 5432), so it does not sit in the form the address pattern searches for at all. Both are examples of a rule operating at the text level failing to see a fact that sits at the structural level; the secret scanner in the previous lesson missed exactly this same class. This is not carelessness in the checks — it is the direct consequence of DC26: the rule looks at text, the defect sits in structure.

All three false positives flagged correct behavior. The fixed address in the test fixture points to a fake store that stays in-process; reading from configuration there would be meaningless. The call that writes the invoice file is not a log — it is the product itself. The nightly fix script is a one-off job and has no need to handle a shutdown signal. All three arise from the gap between the rule’s scope and the principle’s subject: the rule looks at the file, while the principle asks what that file is for.

The cost of a false positive runs in the same direction as the cost of an escape. If three of a nine-line report are empty, whoever reads the report a second time stops opening lines one by one; the chance of finding the two escaped defects drops right at that point. There is no single number that evaluates a set of checks — unless caught, escaped, and false positive are written separately, which one is rising stays invisible.

What the Escapes Look Like at Runtime

In the check table, the two escaped defects were each just a gap. To see what that gap corresponds to, both are run in the same process.

// leaked.mjs — shows the two defects the audit could not see, in a run.
import { queue } from "./source/batch-job.mjs";
import { TARGET } from "./build/package.mjs";

queue.add("meter-1041", 3120);                      // first reading packet
queue.add("meter-2277", 8840);                      // second packet, same process
console.log(`ENVIRONMENT_LABEL     : ${process.env.ENVIRONMENT_LABEL}`);
console.log(`records left in queue : ${queue.size()}`);
console.log(`build artifact target : ${TARGET}`);

#!/usr/bin/env bash
# Runs the same build artifact under two environment labels.
for e in test production; do ENVIRONMENT_LABEL="$e" node leaked.mjs; done

ENVIRONMENT_LABEL     : test
records left in queue : 2
build artifact target : measure-prod.internal:5432
ENVIRONMENT_LABEL     : production
records left in queue : 2
build artifact target : measure-prod.internal:5432

Two records were left in the queue. The second reading packet found what the first left in process memory. When the process restarts, these records vanish; once scale changes, a second process’s queue starts empty, and two processes run the same nightly job with two different states. The defect the check could not see shows up in the run as a number.

The second line is sharper. Whether the environment label is test or production, the build artifact’s target stays the same: the production data store’s address. The build artifact carries the environment; the environment does not carry the build artifact. A nightly batch job running in the test environment looks at production data, and no configuration file declares this, because the difference is not in configuration — it is hidden in a decision made at compile time. The previous lesson’s environment-difference count could never have seen this value; what it counted was configuration files, and this address appears in none of them.

This is the course’s rule in its narrowest form. The same software did not behave differently across two environments — it behaved the same, and that is exactly the problem. The environment label changed, the behavior did not; the difference is embedded somewhere the environment variable cannot reach, so the environment lost its authority to change it. The zero on the build-release-run row in the check table was what covered up this difference.

The way to catch both defects is known: the rule looks not at text but at a parsed syntax tree; only at that level can it see a declaration inside a closure or a value concatenated at compile time. Its cost is removing DC26 — the check no longer just reads the file, it has to parse the language too. The eight checks’ cheapness came from that choice; once the cheapness is lost, whether the check can still run frequently is up for debate too.

What Replaces the Four Untranslatable Principles

Principle	Why it cannot be measured on a single tree	What replaces it
Codebase	how many deployments come out of a single codebase is a repository-level fact; the tree shows a single deployment	matching the source identity the build artifact carries against deployment records
Backing services	that the address comes from configuration is checked, but that the source is swappable can only be seen by actually swapping it	swapping the address and repeating the run
Concurrency	that it scales through the process model cannot be read from a single-process tree	running under load with two processes and output that does not change with the process count
Dev/prod parity	requires comparing two environments; the tree is a single environment	the previous lesson’s environment-difference count

What all four share is that their decision needs a second observation: a second deployment, a second address, a second process, a second environment. The boundary runs exactly here, and its name is decidability from a single observation. The eight checks decided on a single tree; the four needed a comparison.

This distinction determines cost too. The eight checks run in seconds on every change; the remaining four need a separate environment, a separate run, or a load-generating harness. What is cheap runs often, what is expensive runs rarely — and a defect a rarely-run measurement catches has usually already reached production by the time it is caught. The previous lesson’s environment-difference count was what replaced one of the four expensive principles, and none of this lesson’s checks stands in for it.

The cost difference brings an ownership difference too. The eight cheap checks are in the writing team’s hands; they run the instant the code changes, and whoever changed it sees the result. The four expensive measurements fall into the domain of the operating team, which holds the second environment, the second process, or the load harness. Half of the same set of principles is measured by one team, half by the other; which team a violation becomes visible to depends on the kind of violation. The signal gap between the measurement network’s two teams comes from exactly this split.

Summary

• Eight of twelve principles were turned into checks that decide on a single file tree, four could not be; the criterion that separates them is decidability from a single observation.
• Six of eight planted defects were caught, two escaped; the checks reported nine findings and three were false positives.
• One of the three checks reporting zero findings was right, two were wrong. Zero findings measures the rule’s coverage, not the absence of defects.
• Both escaped defects belong to the same class: the rule looks at text, the defect sits in structure — a state inside a closure and an address concatenated at compile time. Both showed up at runtime: two records were left in process memory, and the build artifact’s target came out as the production address under both environment labels.
• All three false positives flagged correct behavior; they arose from the gap between the rule’s scope and the principle’s subject.
• What replaces the four untranslatable principles all needs a second observation; that is why they run rarely, and the defects they catch are caught late.

Next Step

All four untranslatable principles needed a second observation, and two of them named that observation directly: a second environment. Up to this point, environments were assumed to exist — development, test, and production existed, the difference between them was counted, checks were run against the same tree. How the environment itself comes to exist was never asked. Is it set up by hand, does a script set it up, or is it a declaration stating the desired end state? The next lesson compares these three forms and asks the same question: how many differences remain between two environments set up twice from the same declaration, and where is that difference hidden?