--- title: 'Configuration and Secret Separation' source: 'https://academia.sh/en/courses/devops-fundamentals/configuration-and-secret-separation' course: 'DevOps Culture and Fundamentals' language: en updated: '2026-08-23T16:55:09+00:00' license: 'CC BY-SA 4.0' --- # Configuration and Secret Separation The three environments' configuration files are written to disk, and a running checker counts how many keys actually change, how many sit in configuration without changing, and how many are undefined in an environment; the four surfaces a secret can leak through are scanned with a scanner, and caught, escaped, and false positives are written separately. What makes the same software the same software across three environments is not the code being identical — it is where the values the code reads sit. An application reading its configuration from the environment, keeping secret values separate from source, was built in the **Server-Side Fundamentals** course. This lesson does not repeat that setup. The unit here is not a single application — it is the **difference between environments**: how many values actually change from environment to environment, how many sit in configuration without changing, how many are undefined in some environment at all, and through how many separate paths do values counted as secrets leak out. The measurement is done on the **regional measurement network**: **fictional** software that collects readings from water meters, verifies the readings, converts them into invoices, and opens work orders for field crews. It has three environments and a nightly batch job. The files below are genuinely written to disk, the checkers genuinely run; every value they contain is fiction, and none of them is a real secret. **DC19.** The only distinction between environments is the configuration files; code, the build artifact, and operating steps are the same across the three environments. **DC20.** Configuration sits in plain files in `KEY=value` form; keys counted as secrets are identified by a name pattern, there is no separate secret store. **DC21.** There are four surfaces a secret can leak through, and all four are represented by a file on disk: build artifact, log, error message, and the file listing committed to the version repository. **DC22.** The leaks are deliberately planted before the scan, and their count is known; counting what escapes depends on that. ## The Three Environments' Files ```bash #!/usr/bin/env bash # Writes the fictional regional measurement network's three-environment configuration to disk. set -e mkdir -p config source cat > config/development.env <<'SON' ENVIRONMENT_LABEL=development MEASUREMENT_DATA_ADDRESS=127.0.0.1:5432 MEASUREMENT_DATA_POOL=4 MEASUREMENT_DATA_PASSWORD=fake-dev-p4t9 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=2 FIELD_WORK_ORDER_ADDRESS=127.0.0.1:8081 INVOICE_CURRENCY=TRY LOG_LEVEL=detail MAX_RETRIES=3 SON cat > config/test.env <<'SON' ENVIRONMENT_LABEL=test MEASUREMENT_DATA_ADDRESS=measure-test.internal:5432 MEASUREMENT_DATA_POOL=8 MEASUREMENT_DATA_PASSWORD=fake-test-r7w2 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=4 FIELD_WORK_ORDER_ADDRESS=work-order-test.internal:8081 INVOICE_CURRENCY=TRY LOG_LEVEL=detail MAX_RETRIES=3 TEST_SEED=4471 SON cat > config/production.env <<'SON' ENVIRONMENT_LABEL=production MEASUREMENT_DATA_ADDRESS=measure-prod.internal:5432 MEASUREMENT_DATA_POOL=32 MEASUREMENT_DATA_PASSWORD=fake-prod-h3m8 METER_READ_TIMEOUT_MS=2000 VERIFICATION_THRESHOLD_PERCENT=12 BATCH_JOB_START=02:30 BATCH_JOB_CONCURRENCY=16 INVOICE_CURRENCY=TRY LOG_LEVEL=summary MAX_RETRIES=3 INVOICE_SIGNING_KEY=fake-sig-9c1e SON ``` The application's read point is singular and contains no branching that looks at the environment's name. ```js // source/config.mjs — the fictional application's single settings point: every value is read from the environment. export const config = { env: process.env.ENVIRONMENT_LABEL, dataAddress: process.env.MEASUREMENT_DATA_ADDRESS, password: process.env.MEASUREMENT_DATA_PASSWORD, workOrderAddress: process.env.FIELD_WORK_ORDER_ADDRESS, signingKey: process.env.INVOICE_SIGNING_KEY, pool: Number(process.env.MEASUREMENT_DATA_POOL), timeoutMs: Number(process.env.METER_READ_TIMEOUT_MS ?? 2000), thresholdPercent: Number(process.env.VERIFICATION_THRESHOLD_PERCENT ?? 12), batchStart: process.env.BATCH_JOB_START ?? "02:30", concurrency: Number(process.env.BATCH_JOB_CONCURRENCY ?? 2), currency: process.env.INVOICE_CURRENCY ?? "TRY", logLevel: process.env.LOG_LEVEL ?? "summary", maxRetries: Number(process.env.MAX_RETRIES ?? 3), rereadDays: Number(process.env.REREAD_DAYS ?? 30), }; ``` ## Counting the Difference The checker reads the three files and the source's read points separately, then compares the two. ```js // check.mjs — compares the three environment files against the source's read points. import { readFileSync } from "node:fs"; const ENVIRONMENTS = ["development", "test", "production"]; const readEnvFile = (a) => Object.fromEntries(readFileSync(`config/${a}.env`, "utf8") .split("\n").filter(Boolean).map((s) => [s.slice(0, s.indexOf("=")), s.slice(s.indexOf("=") + 1)])); const files = Object.fromEntries(ENVIRONMENTS.map((a) => [a, readEnvFile(a)])); const keys = [...new Set(ENVIRONMENTS.flatMap((a) => Object.keys(files[a])))].sort(); const sourceText = readFileSync("source/config.mjs", "utf8"); // which keys are read, const sourceReads = new Map(); // how many have an embedded default for (const [, name, opt] of sourceText.matchAll(/process\.env\.([A-Z_]+)(\s*\?\?)?/g)) sourceReads.set(name, Boolean(opt)); const counts = { changing: [], fixed: [], missing: [] }; for (const k of keys) { const v = ENVIRONMENTS.map((a) => files[a][k]); if (v.some((x) => x === undefined)) counts.missing.push(k); else if (new Set(v).size === 1) counts.fixed.push(k); else counts.changing.push(k); } const hiddenDefault = [...sourceReads].filter(([k]) => !keys.includes(k)).map(([k]) => k); const required = [...sourceReads].filter(([, v]) => !v).map(([k]) => k); const print = (label, list) => { // print the count, wrap names three per line console.log(`${label.padEnd(37)}: ${list.length}`); for (let i = 0; i < list.length; i += 3) console.log(" " + list.slice(i, i + 3).join(" ")); }; console.log("total keys (across three files)".padEnd(37) + `: ${keys.length}`); print(" present in all, value differs", counts.changing); print(" present in all, value the same", counts.fixed); print(" missing in at least one env", counts.missing); console.log("keys the source reads".padEnd(37) + `: ${sourceReads.size}, without a default: ${required.length}`); print(" in no environment file", hiddenDefault); print(" in a file, source does not read it", keys.filter((k) => !sourceReads.has(k))); for (const a of ENVIRONMENTS) print(`fails at runtime / ${a}`, required.filter((k) => files[a][k] === undefined)); ``` ``` total keys (across three files) : 14 present in all, value differs : 6 BATCH_JOB_CONCURRENCY ENVIRONMENT_LABEL LOG_LEVEL MEASUREMENT_DATA_ADDRESS MEASUREMENT_DATA_PASSWORD MEASUREMENT_DATA_POOL present in all, value the same : 5 BATCH_JOB_START INVOICE_CURRENCY MAX_RETRIES METER_READ_TIMEOUT_MS VERIFICATION_THRESHOLD_PERCENT missing in at least one env : 3 FIELD_WORK_ORDER_ADDRESS INVOICE_SIGNING_KEY TEST_SEED keys the source reads : 14, without a default: 6 in no environment file : 1 REREAD_DAYS in a file, source does not read it : 1 TEST_SEED fails at runtime / development : 1 INVOICE_SIGNING_KEY fails at runtime / test : 1 INVOICE_SIGNING_KEY fails at runtime / production : 1 FIELD_WORK_ORDER_ADDRESS ``` Of fourteen keys, only six are defined across all three environments and actually differ in value. **This is where the difference is hidden — these six lines.** The remaining eight keys do not carry the difference; they carry its appearance. Five keys carry the same value in all three environments. These are not configuration — they are constants; sitting in the configuration file has a cost: they can be hand-changed in a single environment. When `VERIFICATION_THRESHOLD_PERCENT` is pulled from twelve to fifteen in production, verification behavior silently diverges, and this change falls under no code review, because the code did not change. This is where the declared difference and the difference visible only at runtime split apart. The declared kind is the six lines in the files: written, readable, comparable. The runtime-only kind splits in two. `REREAD_DAYS` is in no environment file; its default in the source is thirty. So **a fourth environment definition sits inside the source**, and all three environments use it. Changing its value takes changing code, not configuration; the principle that configuration lives in the environment breaks exactly here, and where it breaks is read from no file. `TEST_SEED` is this pattern's mirror image: it is written in the test file, but the source never reads it. A declared difference cannot be assumed to be effective; someone who thinks test runs with a fixed seed is looking at a test that runs seedless. The defect class the difference conceals shows up in how the missing keys are distributed. `FIELD_WORK_ORDER_ADDRESS` is missing only in production: the work-order-opening path runs fine in development and test, and in production it goes to an undefined address on its first work order. This defect class's name is **the path visible only in production**, and test cannot see it, because nothing is missing in test. `INVOICE_SIGNING_KEY` is the mirror case: because it is defined only in production, the signing path has never run outside production at all. Both gaps carry the same count — one per environment — but one blows up in production, and the other is tried there for the first time. ## When the Difference Shows Up The count reported one missing required key in each environment; it did not report **when** the gap would show up. The same number corresponds to two separate defect classes, and what determines the difference between them is whether the application reads that key at startup or on first use. ```js // start.mjs — validates the required keys at startup; the signing path runs at job time. import { config } from "./source/config.mjs"; const STARTUP_REQUIRED = ["env", "dataAddress", "password", "workOrderAddress"]; const missing = STARTUP_REQUIRED.filter((k) => config[k] === undefined); if (missing.length) { console.log(` startup rejected -> missing: ${missing.join(", ")}`); process.exit(1); } console.log(` started -> pool=${config.pool} concurrency=${config.concurrency} log=${config.logLevel}`); try { // nightly batch job's signing step if (!config.signingKey) throw new Error("signing key undefined"); console.log(` 03:00 batch job -> signed`); } catch (e) { console.log(` 03:00 batch job -> failed: ${e.message}`); } ``` ```bash #!/usr/bin/env bash # Loads each environment's file and starts the application separately. for o in development test production; do echo "$o:" ( set -a; . "config/$o.env"; set +a; node start.mjs ) || true done ``` ``` development: started -> pool=4 concurrency=2 log=detail 03:00 batch job -> failed: signing key undefined test: started -> pool=8 concurrency=4 log=detail 03:00 batch job -> failed: signing key undefined production: startup rejected -> missing: workOrderAddress ``` All three environments were missing a required key, and the count was one in all three. The run produced three separate outcomes. Production was rejected at startup: this is the cheapest form of the gap, because the failure is visible while the eyes of whoever is deploying are on the screen. Development and test started, and the failure came out at three in the morning, in the middle of the nightly batch job. The gap is the same; its cost is not — **the only thing that produces the difference is the length of the startup-validation list.** Because `signingKey` is not on that list, it fails late in all three environments. The output also shows two differences that are not the measurement's subject. The log level is detail in two environments and summary in production. The record of a path visible only in production is kept in a log that is throttled in that same environment: the difference throttles, in the same environment, the very signal that would let someone see the defect it produces. The nightly batch job's concurrency is two, four, and sixteen. A race condition that passes with four workers in test shows up on the first night in production with sixteen. This is a declared difference — it is written in the file — but being declared does not mean it is been tested. Who changes these three values is a separate question too. Pool size, concurrency, and log level are in the operating team's hands and usually change after an incident; the verification threshold and the payment form are the writing team's decision. Because both sit in the same file, the changes look identical, and which difference came from which team can only be told apart by looking at the file's history. ## The Secret's Four Ways Out A secret is also a value that changes from environment to environment, and it is included in the count up to this point. The reason it is counted separately is a single asymmetry: a wrong port gets fixed, a leaked secret cannot be fixed — it can only be rotated, and it stays valid until it is. The four surfaces it can leak through are written to disk below with fictional content. ```bash #!/usr/bin/env bash # Four surfaces: build artifact, log, error message, file listing committed to the repository. set -e mkdir -p build log error cat > build/package.mjs <<'SON' // Build artifact file (fictional): source bundled into a single file together with production values. const config = { address: "measure-prod.internal:5432", MEASUREMENT_DATA_PASSWORD: "fake-prod-h3m8", signature: "fake-sig-" + "9c1e", }; export const connect = () => config; SON cat > log/nightly.jsonl <<'SON' {"time":"03:12:04","event":"batch_job_started","env":"production","meterCount":48210} {"time":"03:12:04","event":"data_connection","connection":"measure:fake-prod-h3m8@measure-prod.internal:5432"} {"time":"03:12:05","event":"config_summary","MEASUREMENT_DATA_PASSWORD":"****","LOG_LEVEL":"summary"} {"time":"04:41:19","event":"invoice_signed","signatureB64":"ZmFrZS1zaWctOWMxZQ==","count":48210} SON cat > error/last-error.txt <<'SON' Uncaught error: invoice signature could not be verified (nightly batch job, production) request: POST /invoice/sign?signature=fake-sig-9c1e config : { env: 'production', MEASUREMENT_DATA_PASSWORD: 'fake-prod-h3m8', pool: 32 } stack : batch-job.mjs:118 -> sign.mjs:42 SON cat > repository-listing.txt <<'SON' source/config.mjs config/example.env config/production.env package.json SON ``` The scanner has two rules: a literal match on the secret's value, and the secret key's name pattern. ```js // scanner.mjs — searches four surfaces for secrets; rule=value is a literal match, rule=name is the name pattern. import { readFileSync } from "node:fs"; const SURFACES = { build: "build/package.mjs", log: "log/nightly.jsonl", error: "error/last-error.txt", repo: "repository-listing.txt" }; const NAME_PATTERN = /PASSWORD|KEY|\.env$/; const values = ["development", "test", "production"] // secret values across the three files .flatMap((a) => readFileSync(`config/${a}.env`, "utf8").split("\n").filter(Boolean)) .filter((s) => NAME_PATTERN.test(s.slice(0, s.indexOf("=")))) .map((s) => s.slice(s.indexOf("=") + 1)); const findings = []; for (const [name, path] of Object.entries(SURFACES)) readFileSync(path, "utf8").split("\n").filter(Boolean).forEach((line, i) => { const rule = [values.some((v) => line.includes(v)) && "value", NAME_PATTERN.test(line.trim()) && "name"].filter(Boolean); if (rule.length) findings.push({ name, lineNo: i + 1, line, rule: rule.join("+") }); }); // Planted leaks: surface name + that line's distinguishing trace. const PLANTED = [["build", "MEASUREMENT_DATA_PASSWORD:"], ["build", 'signature: "fake-sig-'], ["log", '"connection"'], ["log", '"signatureB64"'], ["error", "MEASUREMENT_DATA_PASSWORD:"], ["error", "signature=fake"], ["repo", "production.env"]]; const matched = (f) => PLANTED.some(([s, trace]) => s === f.name && f.line.includes(trace)); const escaped = PLANTED.filter(([s, trace]) => !findings.some((f) => f.name === s && f.line.includes(trace))); console.log(`secret values: ${values.length} surfaces: ${Object.keys(SURFACES).length} ` + `planted leaks: ${PLANTED.length} reported lines: ${findings.length}`); console.log(`caught: ${PLANTED.length - escaped.length} escaped: ${escaped.length} ` + `false positives: ${findings.filter((f) => !matched(f)).length}`); for (const f of findings) console.log(` [${matched(f) ? "real" : "FALSE"}] ${f.name}:${f.lineNo} rule=${f.rule}`); for (const [s, trace] of escaped) console.log(` [ESCAPED] ${s} trace=${trace}`); ``` ``` secret values: 4 surfaces: 4 planted leaks: 7 reported lines: 7 caught: 5 escaped: 2 false positives: 2 [real] build:4 rule=value+name [real] log:2 rule=value [FALSE] log:3 rule=name [real] error:2 rule=value [real] error:3 rule=value+name [FALSE] repo:2 rule=name [real] repo:3 rule=name [ESCAPED] build trace=signature: "fake-sig- [ESCAPED] log trace="signatureB64" ``` ## Caught, Escaped, False Positive The scanner reported seven lines, and seven leaks were planted. **The counts are equal — the sets are not.** This equality is a warning for every reading that judges coverage by looking at a scan report's total. What the two escaped leaks share is that the secret's value does not sit in the form being searched for. One is split in two by concatenation (`"fake-sig-" + "9c1e"`), the other is turned into base 64 (`ZmFrZS1zaWctOWMxZQ==`). Both carry the same secret, and both fall outside the value-matching rule. The general shape of this is: **a literal value search falls behind every step that transforms the value** — a build tool's line concatenation, a log writer's encoding, a message formatter's truncation. All five caught leaks are lines where the value sits unchanged. The way to raise the catch rate is known: transformed forms of the searched value are also generated — its encoded form, its truncated form, each of its parts — and all of them are searched separately. The cost of this is written directly to false positives: the shorter the value, the greater the chance of a random match, and a four-character fragment like `9c1e` turns up in ordinary text too. This is the relationship between the two rules; every widening that reduces escapes increases false positives, and unless the two counts are written separately, which one is rising stays invisible. Both false positives come from the name rule, and both **are correct behavior itself**: the masked log line was written to hide the secret, and the example configuration file is exactly what belongs in the version repository. Its cost is not numeric — it is behavioral. Once two false positives are on the list, whoever reads that list stops checking lines one by one on the next scan; the chance of finding the two escaped leaks drops right along with the false-positive count. The four surfaces also have separate owners: the build artifact belongs to the compile step, the log to operations, the error message to application code, the repository listing to version control. When a leak is closed on one surface, the others stay open, because whoever closes it only sees their own surface. The decision the number delivers here is this: secret separation is not a decision made in one place — it is a boundary maintained separately across four distinct surfaces. ## Summary - Six of the fourteen keys in the three environment files actually change; this is where the difference is hidden. Five keys carry the same value in all three environments, meaning they are constants, not configuration. - Three keys are missing in at least one environment; one of the six required keys stays undefined in every environment. The work order address missing in production blows up only in production, while the signing key defined only in production means the signing path has never run outside it. - In the run, the same count produced three separate outcomes: production was rejected at startup, the other two environments started and failed at three in the morning in the middle of the batch job. What produces the difference is the length of the startup-validation list. - The declared difference is the six lines in the files; the difference visible only at runtime is the fourth environment defined by a default embedded in the source (`REREAD_DAYS`) and the dead key no code reads (`TEST_SEED`). - Five of the seven leaks planted across four surfaces were caught, two escaped; both escapees were transformed forms of the value (concatenation and base 64). - The scanner reported seven lines, and there were seven leaks; two were false positives, and both flagged correct behavior. Equal totals are not proof of coverage. ## Next Step Configuration living in the environment is not a rule that stands alone — it is a member of an interdependent set; this lesson measured one member of that set and, while measuring, saw the rule silently break through a default embedded in the source. The full set was built with its names in the **Server-Side Fundamentals** course. Rather than repeat those names, the next lesson asks a single question: can each principle be turned into a check that runs against the files on disk? The ones that can are run against the fictional application's real files, with caught and escaped counted separately; for the ones that cannot, what replaces them is written down too.