Build Artifacts and Immutability

The previous lesson measured the four environments as links in the chain leading to production and counted how each link could only see the defects on dimensions where it matched production. That measurement assumed one thing: the object that travels between links is always the same object. If what is verified in test is not what runs in production, the assurance testing provides shrinks by exactly that much.

This lesson names that object. A build artifact is the single object produced from source and deployed: an archive, a package, one executable file. The questions are measurable: is output produced twice from the same source actually the same, and if not, where does the difference come from, and how many separate objects does recompiling for each environment produce?

DC7. The build step below is a model: three source files are read and merged into a single file. It is not what a real compiler does, but it carries the same fragilities with respect to reproducibility. DC8. Two machines are modeled with a manifest — A is the writing team’s machine, B is the runner that executes the build — and the time is written in the manifest too, so the measurement can be repeated. A1 and A2 are two separate runs of the same machine. DC9. The digest is the first sixteen hexadecimal digits of the file’s sha256 value; the truncation is only for readability.

Build Once, Deploy Everywhere

The immutability principle is one sentence: a build artifact is produced once, is never changed after it is produced, and the same one is deployed to every environment. Patching a file in place in production turns the object verified in test into an unknown object; if a fix is needed, a new artifact is produced and a new digest is born. Rollback works by the same rule: the previous artifact is redeployed, the current artifact is not edited backward.

The violation is concrete too. Hand-fixing a single file in production creates a new object with no record anywhere in that environment: one object circulates through the chain while a second one sits in production, that second object’s digest is written in no deployment record, and the environment manifest does not show it. The next deployment silently erases the fix; the reason the behavior disappeared is not found in any of the places anyone would look. This is the cost of patching in place: the difference is hidden somewhere entirely outside the measurement’s reach.

The principle working depends on exactly one condition: the artifact must have an identity. Identity is the content digest. Whether two artifacts are the same can only be said by comparing their digests; a file name, a production date, or a version tag cannot say it. That is why the measurement starts from the digest.

// measurement-network/build.mjs — writes the source to disk, produces a single-file build artifact (model)
import { mkdirSync, writeFileSync, readFileSync, rmSync } from "node:fs";
import { createHash } from "node:crypto";

const SOURCE = "source";

export const writeSource = () => {
  rmSync(SOURCE, { recursive: true, force: true });
  mkdirSync(SOURCE, { recursive: true });
  writeFileSync(`${SOURCE}/reading-resolver.js`, "export const resolve = (s) => Number(s.slice(4));\n");
  writeFileSync(`${SOURCE}/billing-rules.js`, "export const amount = (m3) => m3 * 7.4;\n");
  writeFileSync(`${SOURCE}/work-order.js`, "export const openOrder = (no) => ({ no, type: 'onsite' });\n");
};

// Machine manifest: the time is written in the manifest too, so the measurement can be repeated.
export const A1 = { name: "A1", time: "21:04:07", dir: "/home/writer/measurement-network", builtBy: "writer-team",
  readOrder: ["reading-resolver.js", "billing-rules.js", "work-order.js"] };
export const A2 = { ...A1, name: "A2", time: "21:19:52" };
export const B = { name: "B", time: "22:41:30", dir: "/opt/runner/job-1873", builtBy: "pipeline-runner",
  readOrder: ["work-order.js", "billing-rules.js", "reading-resolver.js"] };

export const FOUR_SOURCES = ["timestamp", "directoryOrder", "environmentVariable", "absolutePath"];
export const only = (name) => Object.fromEntries(FOUR_SOURCES.map((k) => [k, k === name]));

export const build = (manifest, options, target, embedded = null) => {
  const files = options.directoryOrder ? manifest.readOrder : [...manifest.readOrder].sort();
  const header = [];
  if (options.timestamp) header.push(`// build time: ${manifest.time}`);
  if (options.environmentVariable) header.push(`// built by: ${manifest.builtBy}`);
  if (options.absolutePath) header.push(`// source root: ${manifest.dir}/${SOURCE}`);
  if (embedded) header.push(`export const CONFIG = ${JSON.stringify(embedded)};`);
  const body = files.map((f) => readFileSync(`${SOURCE}/${f}`, "utf8")).join("");
  mkdirSync("dist", { recursive: true });
  writeFileSync(`dist/${target}`, header.join("\n") + (header.length ? "\n" : "") + body);
  return createHash("sha256").update(readFileSync(`dist/${target}`)).digest("hex").slice(0, 16);
};

The Four Sources That Break Reproducibility

DC10. Four separate breaking sources are planted in the build step, and each can be closed independently: writing the build moment into the output (timestamp), merging source files in directory read order (directoryOrder), embedding the name of whoever ran the build (environmentVariable), and writing the source root’s absolute path (absolutePath). All four are sources encountered in real build steps.

The scan makes three runs — two moments on the same machine (A1, A2) and a second machine (B) — and counts how many distinct digests each option set produces.

// measurement-network/scan.mjs — scans the sources that break reproducibility, one at a time
import { writeSource, build, only, FOUR_SOURCES, A1, A2, B } from "./build.mjs";

writeSource();
const all = Object.fromEntries(FOUR_SOURCES.map((k) => [k, true]));
const none = Object.fromEntries(FOUR_SOURCES.map((k) => [k, false]));

const digests = (options) =>
  [A1, A2, B].map((k) => build(k, options, `package-${k.name}.js`));

const row = (label, options) => {
  const [a1, a2, b] = digests(options);
  const distinct = new Set([a1, a2, b]).size;
  console.log(label.padEnd(26) + (a1 === a2 ? "SAME   " : "DIFF   ").padEnd(12) +
    (a1 === b ? "SAME   " : "DIFF   ").padEnd(11) + distinct);
};

console.log("active source".padEnd(26) + "A1-A2".padEnd(12) + "A1-B".padEnd(11) + "distinct digests");
row("all four", all);
for (const k of FOUR_SOURCES) row("only " + k, only(k));
row("none", none);

const [a1, a2, b] = digests(none);
console.log(`\nwith all four sources closed, the digest of all three runs: ${a1}, ${a2}, ${b}`);
console.log("same input, same output: " + (a1 === a2 && a2 === b));

active source             A1-A2       A1-B       distinct digests
all four                  DIFF        DIFF       3
only timestamp            DIFF        DIFF       3
only directoryOrder       SAME        DIFF       2
only environmentVariable  SAME        DIFF       2
only absolutePath         SAME        DIFF       2
none                      SAME        SAME       1

with all four sources closed, the digest of all three runs: 0a690fcb84c2b166, 0a690fcb84c2b166, 0a690fcb84c2b166
same input, same output: true

The table separates two things. First: each of the four sources alone is enough to break reproducibility. Even with only one active, the digest of the output produced on the second machine differs; two objects produced from the same source cannot be said to be the same.

Second, and more important: three of the four sources are invisible across two runs on the same machine. Directory read order, the environment variable, and the absolute path all stay fixed on a single machine, so they produce no difference between A1 and A2. The writing team runs the build twice on its own machine, sees the same digest, and concludes the build is reproducible. The difference only surfaces the moment the output is produced on a second machine. The previous lesson’s distinction holds here too: this is not a declared difference — it is a difference visible only at runtime, and only on a run on a different machine at that — and the defect class it conceals is named unverified binary: whether the object that passed testing is the object headed to production has not been measured.

With all four closed, the three runs produce a single digest. This means the digest now names not the run but the input: same digest, same source. The way to close each source is known too, and all four live inside the build step — not writing the timestamp into the output, sorting files by name instead of directory read order, not embedding the identity of whoever ran the build, and keeping paths relative to the source root.

What the digest says and what it does not say must also be kept separate. The digest states byte identity: if two outputs have the same digest, the two objects are identical, and one can be written in place of the other in a deployment record. There are two things it does not say. First, it does not say which source the output was produced from; the digest is computed from the output itself, not from the source’s identity. This gap can close once the build step is reproducible: someone else running the build a second time from the same source gets the same digest, and the claim becomes testable — this is reproducibility’s value beyond mere tidiness. Second, a digest difference does not say the behavior difference is meaningful; a single comment line added to the output changes the digest end to end. A different digest is not proof — it is a signal that needs investigating.

Per-Environment Compilation Versus a Single Output

Compiling separately for each link in the chain has a concrete counterpart: values that change by environment get written into the output at build time. DC11. In the fictional network, these values are four: the environment name, whether the billing gateway is real or fake, the log level, and the nightly batch job’s threshold. DC12. In the single-output layout, the same four values are read from the environment at startup; how the read is done and which value counts as a secret is the next lesson’s subject.

// measurement-network/per-environment.mjs — compares per-environment recompilation against a single output
import { writeSource, build, FOUR_SOURCES, A1, A2, B } from "./build.mjs";

const CONFIG = {
  development: { envName: "development", billingGateway: "fake", log: "verbose", threshold: 500 },
  test: { envName: "test", billingGateway: "fake", log: "info", threshold: 20000 },
  production: { envName: "production", billingGateway: "real", log: "warn", threshold: 250000 },
};
const ENV = Object.keys(CONFIG);
const none = Object.fromEntries(FOUR_SOURCES.map((k) => [k, false]));
const all = Object.fromEntries(FOUR_SOURCES.map((k) => [k, true]));
writeSource();

console.log("embedded value keys: " + Object.keys(CONFIG.production).join(", "));
const separate = ENV.map((o) => [o, build(A1, none, `package-${o}.js`, CONFIG[o])]);
for (const [o, digest] of separate) {
  const diff = Object.keys(CONFIG[o]).filter((k) => CONFIG[o][k] !== CONFIG.production[k]);
  console.log(`${o.padEnd(11)} digest ${digest}  values differing from the production pair ${diff.length}`);
}
console.log(`per-environment build: ${new Set(separate.map(([, h]) => h)).size} distinct binaries, ` +
  `embedded values across binaries ${ENV.length * Object.keys(CONFIG.production).length}`);

const single = ENV.map((o) => build(A1, none, `package-single-${o}.js`));
console.log(`single output layout : ${new Set(single).size} distinct binary, digest ${single[0]}, embedded values 0`);

console.log("\nif the per-environment build runs on different machines and at different times:");
const noise = [[A1, "test"], [B, "production"], [A2, "production"]]
  .map(([k, o]) => [`${o} @ ${k.name}`, build(k, all, `package-noise-${o}-${k.name}.js`, CONFIG[o])]);
for (const [label, digest] of noise) console.log("  " + label.padEnd(17) + "digest " + digest);
console.log("  same environment, two runs gave the same digest: " + (noise[1][1] === noise[2][1]));

embedded value keys: envName, billingGateway, log, threshold
development digest 9f8d94167519742a  values differing from the production pair 4
test        digest 41579ab330e96a22  values differing from the production pair 4
production  digest 75cc8f6293d5a489  values differing from the production pair 0
per-environment build: 3 distinct binaries, embedded values across binaries 12
single output layout : 1 distinct binary, digest 0a690fcb84c2b166, embedded values 0

if the per-environment build runs on different machines and at different times:
  test @ A1        digest 78ffd2e872d44825
  production @ B   digest ca355c34a8a96b0c
  production @ A2  digest 68bb8cdccf606938
  same environment, two runs gave the same digest: false

Per-environment compilation produces three separate binaries, and a total of twelve embedded values are carried inside those three binaries. The object test approves diverges from the object headed to production on four values: environment name, billing gateway, log level, and batch threshold. These four sit inside the output; someone looking from outside sees that all three files carry the same name, and only the digest tells the difference. The billing gateway being fake in the test binary and real in the production binary makes the previous lesson’s contract-drift class invisible in every run of testing.

In the single-output layout, the three environments share a single binary: the digest is the same across all three, and the embedded-value count is zero. The twelve values do not vanish — they change location, moving from inside the output to the environment manifest. In the previous lesson’s terms: twelve undeclared, runtime-only differences turn into twelve declared differences. The amount of difference stays the same; where it is hidden changes — and that is what becomes measurable.

The last three lines show the second cost of per-environment compilation. When builds run on different machines and at different moments, even two outputs produced for the same environment carry different digests. The digest no longer names the environment; it names the run. When two digests differ, whether the cause is an environment difference or build noise cannot be told apart, and the answer to “is what is running in production what test verified” becomes unmeasurable.

Whether the output really sits on disk and whether the digest is that file’s digest can also be verified directly.

# verify.sh — is the output really on disk, does its digest match what node printed
ls dist/package-single-*.js
shasum -a 256 dist/package-single-production.js | cut -c1-16
cat dist/package-single-production.js

dist/package-single-development.js
dist/package-single-production.js
dist/package-single-test.js
0a690fcb84c2b166
export const amount = (m3) => m3 * 7.4;
export const resolve = (s) => Number(s.slice(4));
export const openOrder = (no) => ({ no, type: 'onsite' });

The content of the files written for the three environments is identical, and the digest taken from the shell matches the 0a690fcb84c2b166 value node printed. Even though the file names differ, the object is one; this is the immutability principle in its measured form. It is also visible that the files are merged in name order — because directory read order is closed.

How Many Objects Circulate Through the Chain

The previous lesson’s chain was a single object advancing through four stops. The measurement also counts what the two layouts give that chain. In per-environment compilation, three separate objects circulate through the chain and three separate build runs happen; each run is exposed all over again to the four breaking sources counted in the scan. In the single-output layout, the object is one, the run is one, and moving between links reduces to a copy operation.

The difference shows up in how many questions can be asked. In the single-output layout, the question “is the object test verified the object headed to production” is answered by a single digest comparison: yes if the two values are equal, no otherwise. In per-environment compilation, the same question cannot even be asked, because the digests of two environments already differ by definition, and how much of the difference comes from environment settings versus build noise cannot be separated out. A measurable assurance turns into an unmeasurable habit.

The distinction is also written into the delivery metrics. Getting a fix to production requires three build runs under per-environment compilation; all three have to succeed and reach the right environment, which adds three builds and three deployment preparations to lead time. On the change-failure-rate side, there is the unverified-binary class: because the object that passed testing does not go to production, some of the defect classes testing caught can be reborn in production. The single-output layout moves both metrics in the same direction — but its cost is that it forces the twelve environment-varying values to be kept somewhere outside the output.

Summary

• A build artifact is the single object produced from source and deployed; the immutability principle says this object is produced once, is never changed in place, and the same one goes to every environment.
• Whether two artifacts are the same can only be said by their content digest; a file name, a date, and a version tag cannot say it.
• Four sources that break reproducibility were measured — timestamp, directory read order, the environment variable, absolute path — and each alone was enough to produce a different digest on a second machine. With all four closed, three runs gave a single digest.
• Three of these four sources are invisible across two runs on the same machine; the difference only surfaces once the build runs on a different machine. The defect class it conceals is the unverified binary.
• Per-environment recompilation produced three separate binaries and twelve embedded values; in the single-output layout, the binary count dropped to one and the embedded-value count to zero. The values did not vanish — they moved from inside the output to the environment manifest.

Next Step

The single-output layout left one question unanswered: where does this one produced object sit, and how is it named? A digest is an identity, but it is not a name; hand-writing sixteen hexadecimal digits into deployment records is not a sustainable arrangement. And if the artifact is produced once and deployed everywhere, it has to sit somewhere between where it is produced and where it is deployed, and the previous version has to still be findable when a rollback is needed. The next lesson measures that storage place: how is a version tagged, in how many deployments does the difference between an immutable version and a movable tag show itself, and where does a retention rule close the rollback window?