Secret Management

The previous two lessons measured the desired state staying in the repository and being carried from there to the cluster. Both silently left one item out: the secrets the services need. The credentials the collector carries to connect to the measurement network’s endpoints, billing’s database password, the work-order service’s archive key — these are part of the desired state, but they cannot be put into the repository as plain text. If they are not put into the repository, the desired state stays incomplete and the reconciler agent cannot apply it.

The shape of the solution is set: the secret sits sealed in the repository, and decrypt authority is managed on a separate axis. What this lesson measures is that axis — where the key sits, how many subjects can decrypt how many records, what happens when the key is rotated, and what the history of a sealed record staying in the repository means. Which paths a secret leak can take through was scanned earlier — log lines, environment variables, build-output layers — and that scan is not repeated here.

The measurement is over the regional measurement network and is fictional. Encryption runs for real with node:crypto; all secret values are fabricated and written with a fixture- prefix.

RD13. The inventory is 33 secret records: four services, three environments, two to three secret types per service. RD14. Encryption is AES-256-GCM, the key is derived with scrypt. The initialization vector is derived from the record ID so the output is reproducible; in a real deployment it is generated randomly. RD15. Three key layouts are compared: single key, per-environment key, per-env-service key. RD16. There are 29 subjects that can read the repository; 14 of them carry no key at all. Each subject’s needed (environment, service) pairs are fixed in writing. RD17. Over a 180-day history, each record changed one to four times; each change sits in a separate commit. RD18. Production on-call needs two services’ secrets; there are 24 night events over 180 days, and every event left without a key adds 35 minutes of escalation.

Where the Key Sits, Who Can Decrypt

// secret.mjs — sealed secret record in the repository: inventory, key layout, decrypt
// authority. SECRET VALUES ARE FICTIONAL, no real secret is written. Encryption runs
// for real with node:crypto. The IV is derived from the record ID and key version so the output is reproducible.
import { scryptSync, createCipheriv, createDecipheriv, createHash } from "node:crypto";

export const ENV = ["development", "staging", "production"];
export const SERVICE = { collector: ["database", "measurement-network-token", "cache"],
  verifier: ["database", "signing"], billing: ["database", "payment-token", "signing"],
  workorder: ["database", "notification-token", "archive"] };

export const RECORD = ENV.flatMap((e) => Object.entries(SERVICE).flatMap(([s, tl]) =>
  tl.map((t) => ({ id: `${e}/${s}/${t}`, env: e, service: s,
    value: `fixture-${t}-${s}-${e}-0001` }))));               // fictional value, not a real secret

export const counter = { derive: 0, encrypt: 0, decrypt: 0 };
const keyStore = new Map();
export const key = (name, version = 1) => {                // password is fictional too
  const k = `${name}#${version}`;
  if (!keyStore.has(k)) { counter.derive++;
    keyStore.set(k, scryptSync(`fixture-password-${k}`, `salt-${name}`, 32, { N: 16384 })); }
  return keyStore.get(k);
};

export const LAYOUT = { single: () => "all", env: (k) => k.env,
  "env-service": (k) => `${k.env}/${k.service}` };

export const encrypt = (record, name, version, value = record.value) => {
  const iv = createHash("sha256").update(`${record.id}#${name}#${version}#${value}`).digest().subarray(0, 12);
  const c = createCipheriv("aes-256-gcm", key(name, version), iv);
  const g = Buffer.concat([c.update(value, "utf8"), c.final()]);
  counter.encrypt++;
  return { iv, body: g, tag: c.getAuthTag(), name, version };
};

export const decrypt = (k) => {
  const d = createDecipheriv("aes-256-gcm", key(k.name, k.version), k.iv);
  d.setAuthTag(k.tag); counter.decrypt++;
  return Buffer.concat([d.update(k.body), d.final()]).toString("utf8");
};

// subjects: each needs specific (env, service) pairs. the repo reader has no keys.
export const SUBJECT = [
  { name: "agent/development", n: 1, need: [["development", "*"]] },
  { name: "agent/staging", n: 1, need: [["staging", "*"]] },
  { name: "agent/production", n: 1, need: [["production", "*"]] },
  { name: "measurement team", n: 6, need: [["development", "collector"], ["development", "verifier"],
    ["staging", "collector"]] },
  { name: "billing team", n: 4, need: [["development", "billing"], ["staging", "billing"]] },
  { name: "production on-call", n: 2, need: [["production", "billing"], ["production", "collector"]] },
  { name: "repo reader", n: 14, need: [] },
];
const matches = (k, [e, s]) => k.env === e && (s === "*" || k.service === s);
export const needed = (z) => RECORD.filter((k) => z.need.some((c) => matches(k, c)));

export function authorize(layout) {                        // records decryptable under a layout
  const f = LAYOUT[layout];
  return SUBJECT.map((z) => {
    const carries = new Set(needed(z).map(f));              // key names granted to the subject
    return { z, decryptable: RECORD.filter((k) => carries.has(f(k))), need: needed(z).length };
  });
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const subjectCount = SUBJECT.reduce((a, z) => a + z.n, 0);
  console.log(`inventory: ${RECORD.length} secret records (fictional values), ${subjectCount} repo-reader subjects`);
  const sample = encrypt(RECORD[0], "development", 1);
  console.log(`sample record ${RECORD[0].id} -> ${sample.body.toString("base64")} ` +
    `| decrypted: ${decrypt(sample)}`);
  const B = [13, 6, 15, 12, 9, 9, 16];
  const yz = (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join("");
  console.log("\n" + yz(["layout", "keys", "decrypt pairs", "need pairs", "surplus",
    "blocked", "widest subject"]));
  for (const layout of Object.keys(LAYOUT)) {
    const y = authorize(layout);
    const keys = new Set(RECORD.map(LAYOUT[layout])).size;
    const pair = y.reduce((a, x) => a + x.z.n * x.decryptable.length, 0);
    const need = y.reduce((a, x) => a + x.z.n * x.need, 0);
    for (const x of y) if (x.decryptable.length) decrypt(encrypt(x.decryptable[0], LAYOUT[layout](x.decryptable[0]), 1));
    console.log(yz([layout, keys, pair, need, pair - need,
      subjectCount * RECORD.length - pair, Math.max(...y.map((x) => x.decryptable.length))]));
  }
  console.log(`\noperation counter: ${counter.derive} key derivations, ` +
    `${counter.encrypt} encryptions, ${counter.decrypt} decryptions`);
}

inventory: 33 secret records (fictional values), 29 repo-reader subjects
sample record development/collector/database -> 2w41e6WkaaQMnq/d3iUgX3C7n5OAdHW0TUpYFVHQUBKjJ0HWGhgaAridVQ== | decrypted: fixture-database-collector-development-0001

layout         keys  decrypt pairs  need pairs  surplus  blocked  widest subject
single            1            495         117      378      462              33
env               3            275         117      158      682              22
env-service      12            117         117        0      840              11

operation counter: 8 key derivations, 19 encryptions, 19 decryptions

Encryption genuinely ran: the sample record was encrypted, decrypted back, and the fixture value returned unchanged. The real table is below, and it answers a single question three times — how many subjects can decrypt how many records. The unit of measure is the (subject, record) pair; with 29 subjects and 33 records, the largest possible value is 957.

The key does not sit inside the repository — if it did, encryption would lose its meaning. The key sits in the subjects’ hands, and the table counts that distribution’s result. In the single-key layout, the 15 subjects carrying the key can decrypt all 33 records: 495 pairs. The number of pairs these subjects actually need is 117, and this number is the same across all three layouts, because need is independent of the key layout. The 378 pairs in between are pure surplus — authority no one asked for, granted only because the key is coarse.

When switching to a per-environment key, surplus falls from 378 to 158; in the per-env-service key, it is zeroed out, and a subject that can decrypt, decrypts exactly what it needs. The widest subject’s authority also drops from 33 to 11: the number of records a single person’s key opens shrinks threefold. The price of this is the key count — from 1 to 12 — and that number is the subject of the next section.

The Cost of Key Rotation and the Record Left in the Past

// rotate.mjs — the cost of key rotation and how a sealed record stays in the past.
// Historical encrypted values are genuinely decrypted; the values are fictional.
import { createHash } from "node:crypto";
import { RECORD, LAYOUT, SUBJECT, authorize, encrypt, decrypt, counter } from "./secret.mjs";

const FILE = (k) => `${k.env}/${k.service}/secrets`;      // records are spread across files like this
const times = (k) => 1 + (createHash("sha256").update(k.id).digest()[0] % 4);
const value = (k, i) => `fixture-${k.id.split("/")[2]}-${k.service}-${k.env}-` +
  String(i).padStart(4, "0");

// 180-day history: each record changed times(k) times; each change sits in a separate commit.
const history = RECORD.flatMap((k) => Array.from({ length: times(k) }, (_, i) =>
  ({ record: k, plain: value(k, i + 1) })));
const current = new Map(RECORD.map((k) => [k.id, value(k, times(k))]));

const W = [13, 14, 17, 15, 15];
const row = (h) => h.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0])).join("");
console.log(`history: ${history.length} encrypted record versions / ${RECORD.length} records, ` +
  `${new Set(RECORD.map(FILE)).size} files`);
console.log("\n" + row(["layout", "keys", "single rotation", "files", "rotate all"]));
for (const [layout, f] of Object.entries(LAYOUT)) {
  const target = f(RECORD.find((k) => k.env === "production" && k.service === "billing"));
  const affected = RECORD.filter((k) => f(k) === target);
  console.log(row([layout, new Set(RECORD.map(f)).size, affected.length,
    new Set(affected.map(FILE)).size, `${RECORD.length} / ${new Set(RECORD.map(FILE)).size} files`]));
}

// in the env layout, the production key is rotated 1 -> 2. Current records are re-encrypted.
const f = LAYOUT.env;
const production = RECORD.filter((k) => k.env === "production");
const historyEncrypted = history.map((g) => ({ ...g, box: encrypt(g.record, f(g.record), 1, g.plain) }));
const rotated = production.map((k) => encrypt(k, f(k), 2, current.get(k.id)));
console.log(`\nproduction key 1 -> 2: ${rotated.length} records re-encrypted, ` +
  `${new Set(production.map(FILE)).size} files touched, ${new Set(production.map(FILE)).size} new commits`);

// what the history yields if the old key leaks
const old = historyEncrypted.filter((g) => g.box.name === "production" && g.box.version === 1);
const opened = old.map((g) => decrypt(g.box));
const correct = opened.every((a, i) => a === old[i].plain);
const stillCurrent = old.filter((g, i) => opened[i] === current.get(g.record.id)).length;
const idx = old.findIndex((g, i) => opened[i] === current.get(g.record.id));
console.log(`if the old key (version 1) leaks: ${old.length} versions in history decrypt ` +
  `(all correct: ${correct}); ${stillCurrent} of these are the TODAY'S value of ${production.length} records`);
console.log(`example: ${old[idx].record.id} from an old commit -> ${opened[idx]}`);

// what the decrypt-authority gate wrongly holds: events left unkeyed during the night shift
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = generator(90210), SRV = ["collector", "verifier", "billing", "workorder"];
const event = Array.from({ length: 24 }, () => SRV[Math.floor(r() * 4)]);
const ONCALL = ["billing", "collector"];                   // what production on-call needs
console.log("\n" + row(["layout", "night events", "unkeyed", "extra minutes", "blocked pairs"]));
for (const layout of Object.keys(LAYOUT)) {
  const carries = (s) => layout === "env-service" ? ONCALL.includes(s) : true;
  const missing = event.filter((s) => !carries(s)).length;
  const pair = authorize(layout).reduce((a, x) => a + x.z.n * x.decryptable.length, 0);
  console.log(row([layout, event.length, missing, missing * 35,
    SUBJECT.reduce((a, z) => a + z.n, 0) * RECORD.length - pair]));
}
console.log(`\noperation counter: ${counter.derive} key derivations, ${counter.encrypt} encryptions, ` +
  `${counter.decrypt} decryptions`);

history: 76 encrypted record versions / 33 records, 12 files

layout                 keys  single rotation          files     rotate all
single                    1               33             12  33 / 12 files
env                       3               11              4  33 / 12 files
env-service              12                3              1  33 / 12 files

production key 1 -> 2: 11 records re-encrypted, 4 files touched, 4 new commits
if the old key (version 1) leaks: 24 versions in history decrypt (all correct: true); 11 of these are the TODAY'S value of 11 records
example: production/collector/database from an old commit -> fixture-database-collector-production-0002

layout         night events          unkeyed  extra minutes  blocked pairs
single                   24                0              0            462
env                      24                0              0            682
env-service              24               11            385            840

operation counter: 4 key derivations, 87 encryptions, 24 decryptions

The rotation table gives the second face of key count. In the single-key layout, rotating one key re-encrypts all 33 records and touches 12 files; in the env-service layout, the same operation is 3 records and 1 file. But the last column honestly shows the trade-off: rotating all keys is 33 records and 12 files in all three layouts. Fine-grained key partitioning does not reduce total work, it narrows the echo of a single leak.

When the production key was rotated from 1 to 2, 11 records were re-encrypted, 4 files were touched, and 4 new commits were produced. Everything up to here is expected. This lesson’s core is in the next line, and it is not a guess, it is a decrypt that was actually run: if the old key leaks, all of the 24 encrypted versions in history decrypt — the all correct: true line shows this was tested — and 11 of these 24 versions are the today’s value of the 11 production records.

The result is: rotating the key does not close off the past. The new commit holds the new encrypted value, but the old commit still holds the old encrypted value as it was, and when that value is opened with the old key, it yields a secret that is still valid today. The example line makes this concrete: the value decrypted from the collector’s production database record’s old commit is the same value the service running today uses. The only operation that actually closes off the past is changing the secret’s value at its source — rotating the password in the database, revoking the token on the measurement network’s side. Key rotation touches 4 files; value rotation touches every consumer, and the 4 files on the repository side are the cheapest part of that work.

The Decrypt-Authority Gate’s Ledger

The gate in this topic is not a review or a reconciliation round, it is a decrypt authority. Its ledger reads in three items.

What it holds, and holds only by itself. In the per-environment key layout, the blocked (subject, record) pairs number 682. The entirety of this number is marginal, because all 29 subjects can already read the repository — the repository-access gate stops none of these pairs. If encryption were removed, 682 reads would become free; even under a coarse layout, the gate does work entirely on its own. In the env-service layout, the same number rises to 840.

What it misses. There are two classes. The first is the past: 24 encrypted versions can be read through a path the gate never enters at all, because there is no authority check in place when the old key leaks. The second is surplus: in the env layout, 158 pairs open more than the legitimately key-carrying subjects need. The gate does not hold these, it lets them through openly — the measurement team’s 6 members can also decrypt the development environment’s billing secrets.

What it wrongly holds, and the time it adds. The last table closes the trade-off. The env-service layout zeroes out the surplus, but in 11 of the 24 night events the production on-call did not carry the key it needed, and each event added a 35-minute escalation: 385 minutes. In the env layout this number is zero, in exchange for 158 open surplus pairs. The decision is made between these two numbers, and the sentence “a finer-grained key is more secure” does not carry the decision by itself. The gate’s operating load is also counted: the 180-day history of 33 records costs 87 encryption operations, its verification costs 24 decrypt operations; every secret change asks for one encryption, every reconciliation round asks the agent for one decryption.

Summary

• Of the 957 possible (subject, record) pairs from 29 subjects and 33 records, the single-key layout opens 495, the per-environment key 275, the per-env-service key 117; need is 117 in all three layouts.
• Surplus authority falls from 378 to 158, then to 0; the records a compromised widest subject can decrypt fall from 33 to 11.
• Rotating one key is 33 records and 12 files in the single-key layout, 3 records and 1 file in the env-service layout; but rotating all keys is 33 records and 12 files in all three layouts.
• After the production key is rotated, if the old key leaks, 24 encrypted versions in history decrypt, and 11 of them are the today’s value of 11 production records — key rotation does not close off the past, value rotation does.
• The decrypt-authority gate’s marginal value is 682 blocked pairs in the env layout, and the entirety of it is marginal, because all 29 subjects can already read the repository.
• The gate’s price was measured: in the env-service layout, 11 of 24 night events were left without a key and added 385 minutes of escalation; in the env layout, delay is 0, open surplus is 158 pairs.

Next Step

Secrets can now stay sealed in the repository, and who can decrypt what is countable. This was a question about the content of the repository’s desired state. The next lesson turns to another face of content: what components a release actually consists of. An application is not made up of its own code alone — direct and transitive dependencies, runtime libraries, base-layer content. What gets measured is this inventory’s coverage: how many dependencies show up, how many are transitive, how long producing the inventory takes, and how many components go entirely uncounted in a pipeline that does not produce an inventory.