Secrets Management

The previous lesson hardened the transport: an incoming request now enters through an encrypted channel, and the headers telling the browser what it may do are set server-side. With both ends closed, what remains is what the channel carries. The loan system’s seven services prove who they are to each other with a signing key; the catalog ID, the payment key, and the notification token belong to the same class. These are secrets, and the transport layer protects them only on the network — not where they sit.

Distributing these values to the test environment was measured in the Secrets and Data Management lesson: which value enters, how much scope it takes, and how many points it leaks from in a run’s artifacts. Production asks a different question. There the value is already in, its scope already wide, and no one takes it back after a run. What is measured here is three quantities: how many places the value stands in, how many requests break while rotating, and, once it leaks, how long it keeps working.

AS10 — seven services share a single signing-key ring. Both the issuer and the verifier of a ticket read from the same ring; the ring is versioned, and the store keeps the signing and verifying version sets apart. AS11 — a service ticket is valid for 120 seconds, and a service does not refresh it until it expires. AS12 — the new value reaches services one at a time, 45 seconds apart; the last of the seven reads it 270 seconds after the rotation moment.

Where the Value Stands

The store gives verification the versions to accept as a set bound to time: only the old version before the rotation moment, both through the window, only the new one after it closes.

// vault.mjs — the secret store: versioned key ring, the signing and verification set
import { createHmac, timingSafeEqual } from 'node:crypto';

export const TICKET_LIFETIME = 120;   // AS11: a service ticket is valid for 120 seconds
export const ROTATION_MOMENT = 300;   // the moment the new version is produced
export const ROLLOUT_INTERVAL = 45;   // AS12: services read the new value 45 seconds apart
export const HORIZON = 900;
export const SERVICES = ['catalog', 'loan', 'membership', 'notification', 'penalty', 'search', 'reporting'];

// The store keeps two sets apart: the signer has one version, the verifier one or two.
export function openVault(window) {
  const key = { 1: 'v1-ring-4c9a1d', 2: 'v2-ring-77e5b0' };
  const closing = window === Infinity ? Infinity : ROTATION_MOMENT + window;
  return {
    verifying: (t) => (t < ROTATION_MOMENT ? [1] : t < closing ? [1, 2] : [2]),
    sign: (version, body) => createHmac('sha256', key[version]).update(body).digest('base64url'),
    verify(t, body, signature) {
      for (const version of this.verifying(t)) {
        const expected = Buffer.from(this.sign(version, body));
        const incoming = Buffer.from(signature);
        if (expected.length === incoming.length && timingSafeEqual(expected, incoming)) return version;
      }
      return 0;
    },
  };
}

// How many places the value stands: every service reads it in both its process and its scheduled job.
export const PLACEMENT = {
  distributed: { locations: SERVICES.length * 2, reads: 0 },
  centralized: { locations: 1, reads: SERVICES.length },
};

The last lines meet this course’s third question: how many places the setting repeats in. Standing separately in every service’s process and scheduled job puts it in fourteen places; read from a single store at run time, it is in one place with seven read points. The difference shows up in the cost of rotation.

The Rotation Window

The run below stands up a real verification service and walks a 900-second timeline in five-second steps. Time is virtual: every request reports the time it carries in a header, so counts do not depend on the wall clock. Window width is swept, and each width runs twice — once with services refreshing tickets in the ordinary order, once with the worst alignment, where each service refreshes its last old ticket one second before reading the new value.

// rotation.mjs — overlap-window scan: which key verified, how many requests were rejected
import { createServer } from 'node:http';
import { openVault, TICKET_LIFETIME, ROTATION_MOMENT, ROLLOUT_INTERVAL, HORIZON, SERVICES, PLACEMENT } from './vault.mjs';

let vault = openVault(0);
const server = createServer((request, response) => {
  const t = Number(request.headers['x-time']);
  const body = request.headers['x-ticket'];
  const issued = Number(body.split('.')[1]);
  const version = vault.verify(t, body, request.headers['x-signature'] ?? '');
  const [status, result] = version === 0 ? [401, 'sig'] : t >= issued + TICKET_LIFETIME ? [401, 'expired'] : [200, version];
  response.writeHead(status, { 'content-type': 'application/json' });
  response.end(JSON.stringify({ result }));
});
server.listen(8951, '127.0.0.1');
const BASE = 'http://127.0.0.1:8951/loan';

// Service i reads the new value at ROTATION_MOMENT + i*ROLLOUT_INTERVAL; it holds its ticket until it expires.
const readMoment = (i) => ROTATION_MOMENT + i * ROLLOUT_INTERVAL;
const LAST_READ = readMoment(SERVICES.length - 1);

// align: each service's last v1 ticket is refreshed one second before it reads the new value (worst case).
async function run(window, align = false) {
  vault = openVault(window);
  const ticket = SERVICES.map(() => null);
  const aligned = SERVICES.map(() => false);
  const counter = { 1: 0, 2: 0, sig: 0, expired: 0 };
  let lastReject = null;
  for (let t = 0; t <= HORIZON; t += 5) {
    const i = (t / 5) % SERVICES.length;
    const worst = align && !aligned[i] && ticket[i] !== null && t >= readMoment(i);
    if (ticket[i] === null || t >= ticket[i].issued + TICKET_LIFETIME || worst) {
      const issued = worst ? readMoment(i) - 1 : t;
      const version = issued < readMoment(i) ? 1 : 2;
      const body = `${SERVICES[i]}.${issued}`;
      ticket[i] = { version, body, signature: vault.sign(version, body), issued };
      aligned[i] ||= worst;
    }
    const y = await fetch(BASE, { headers: { 'x-time': String(t), 'x-ticket': ticket[i].body, 'x-signature': ticket[i].signature } });
    const { result } = await y.json();
    counter[result] += 1;
    if (y.status === 401) lastReject = t;
  }
  // A leaked old version tried at the horizon: does the value still open the door.
  const body = `search.${HORIZON}`;
  const leaked = await fetch(BASE, { headers: { 'x-time': String(HORIZON), 'x-ticket': body, 'x-signature': vault.sign(1, body) } });
  return { ...counter, lastReject, leaked: leaked.status };
}

const s = (x, n) => String(x).padStart(n);
console.log(`${'window'.padEnd(12)}${s('v1', 6)}${s('v2', 6)}${s('reject', 9)}${s('last reject', 12)}`
  + `${s('worst reject', 13)}${s('leaked v1', 10)}`);
for (const window of [0, 100, 200, 300, 400, Infinity]) {
  const r = await run(window);
  const k = await run(window, true);
  const label = window === Infinity ? 'no cleanup' : `${window} s`;
  console.log(`${label.padEnd(12)}${s(r[1], 6)}${s(r[2], 6)}${s(r.sig + r.expired, 9)}`
    + `${s(r.lastReject ?? '-', 12)}${s(k.sig + k.expired, 13)}${s(r.leaked, 10)}`);
}
server.close();

const required = LAST_READ - ROTATION_MOMENT + TICKET_LIFETIME;
console.log(`\nlast read ${LAST_READ} s, ticket lifetime ${TICKET_LIFETIME} s -> arithmetic lower bound ${required} s`);
for (const [label, y] of Object.entries(PLACEMENT)) {
  console.log(`${label}: the value stands in ${y.locations} places, ${y.reads} reads at run time, `
    + `4 rotations a year make ${y.locations * 4} manual steps`);
}
const day = 24 * 60;
for (const period of [90, 30, 7]) {
  console.log(`rotation period ${period} days -> average validity of a leaked key ${period / 2} days, `
    + `worst case ${period} days; a ticket leaked at the same moment ${TICKET_LIFETIME / 60} minutes (${(period * day / (TICKET_LIFETIME / 60)).toFixed(0)}x shorter)`);
}

window          v1    v2   reject last reject worst reject leaked v1
0 s             60    85       36         555           51       401
100 s           80    85       16         555           31       401
200 s           90    85        6         555           15       401
300 s           96    85        0           -            3       401
400 s           96    85        0           -            0       401
no cleanup      96    85        0           -            0       200

last read 570 s, ticket lifetime 120 s -> arithmetic lower bound 390 s
distributed: the value stands in 14 places, 0 reads at run time, 4 rotations a year make 56 manual steps
centralized: the value stands in 1 places, 7 reads at run time, 4 rotations a year make 4 manual steps
rotation period 90 days -> average validity of a leaked key 45 days, worst case 90 days; a ticket leaked at the same moment 2 minutes (64800x shorter)
rotation period 30 days -> average validity of a leaked key 15 days, worst case 30 days; a ticket leaked at the same moment 2 minutes (21600x shorter)
rotation period 7 days -> average validity of a leaked key 3.5 days, worst case 7 days; a ticket leaked at the same moment 2 minutes (5040x shorter)

Reading the Window

The first row is rotation without a window, and it is this course’s clearest example of the rule. The moment the new version is produced, verification stops accepting the old one; there is no configuration error anywhere — store, signature, services all fine. Still, 36 of 181 requests are rejected, and not at the rotation moment: they run all the way to 255 seconds after it, last reject at the 555th second. The cause is a service that does not drop its held ticket before it expires; the last to read still holds one signed with the old version until the 570th second.

As the window widens, rejects drop from 36 to 16, to 6, to zero — but note which width reaches zero: 300 seconds sufficed here, yet the same width rejects three requests under the worst alignment. The run-independent quantity sits below the table: last-read moment minus rotation moment, plus ticket lifetime — 270 + 120 = 390 seconds. The measured value (300) gives only that run’s alignment, not the correct setting; a team picking the window from a measurement will drop requests again the moment the schedule slips, and those drops, arriving four minutes after the rotation, will not trace back to the cause.

The last row lies in the opposite direction. In the no-cleanup setup, verification accepts both versions forever: drops zero, signature errors zero, dashboard clean, rotation looking “successful.” An attempt at the 900th second with the leaked old value still gets 200 — the other five rows get 401. Producing the new version does not invalidate the old one; the cleanup step does, and skipping it worsens no measure at all. That is the whole point of rotation, and it fails silently at exactly that step.

The Lifetime of a Leaked Value

When a leaked value is noticed, the question is not “how long did it work” but “how long will it keep working,” and the answer depends on the rotation period. Since the leak can fall anywhere within the period, expected exposure is half the period and the worst case is the whole period: forty-five days in a ninety-day period, fifteen in a thirty-day one, three and a half in a seven-day one. This shows what shortening the period buys — not closing the door, but how long it stays open.

The second half of the same lines shows the real lever. If what leaks is a ticket produced with the key rather than the key itself, its lifetime is 120 seconds — sixty-four thousand times shorter than a ninety-day period. Holding the long-lived value in few places and deriving a short-lived ticket from it shortens the exposure window by four orders of magnitude, independent of rotation frequency. The placement rows write out the price: fourteen locations means fifty-six manual steps a year for four rotations, any of which can be missed; one location means four steps, with seven services reading it at run time.

The Value Left in the Code

All the arithmetic above assumes the value stands in configuration. A value sitting in the code does not go invalid on rotation — the old value keeps signing until the code ships, so the last-read moment shifts back by the release cycle’s length, and the required window grows with it. This is why scanning for a hardcoded value is part of the rotation mechanism.

// scan.mjs — scanning for a value hardcoded in the source: false accept and false reject
// [file, line, is it really a secret]
const source = [
  ['loan/ticket.mjs', "const SIGNING_KEY = 'v2-ring-77e5b0';", true],
  ['report/export.mjs', "const STORE_PASSWORD = 'k3M-pW9-zQ4-vB7-nR2';", true],
  ['notification/send.mjs', "const config = { upstream: 'T-4f8a1c9d2b6e0357a9c4f1b8' };", true],
  ['search/index.mjs', "const refresh = 'Zq8Rv3Nw6Lp1Ke5Ty9Xu2Bd7Hm4';", true],
  ['membership/test.mjs', "const TEST_TOKEN = 'sb-9d31-77af-2c60';", true],
  ['catalog/record.mjs', "const CATALOG_ID = 'ac-catalog-public-record';", false],
  ['loan/config.mjs', "const KEY_NAME = 'signing-key-v2';", false],
  ['ui/icon.mjs', "const ICON = 'iVBORw0KGgoAAAANSUhEUgAAAAgAAAAI';", false],
  ['penalty/config.mjs', "const secretDir = './data/tmp/penalty';", false],
  ['notification/sample.mjs', "const TOKEN_PLACEHOLDER = 'CHANGE-ME-DURING-SETUP';", false],
  ['membership/identity.mjs', "const memberId = '9f2c1a4e-77b0-4d31-8c6a-2e5b90f14d73';", false],
];

const value = (line) => (line.match(/'([^']+)'/) ?? [, ''])[1];
const name = (line) => (line.match(/(?:const|let)\s+(\w+)|(\w+)\s*:/) ?? [, ''])[1] ?? '';

function entropy(d) {
  const count = {};
  for (const c of d) count[c] = (count[c] ?? 0) + 1;
  return -Object.values(count).reduce((a, n) => a + (n / d.length) * Math.log2(n / d.length), 0);
}

const NAME_PATTERN = /key|secret|passphrase|token|signature|password/i;
const ALLOWED = [/^ac-/, /^CHANGE-ME/];

const rules = {
  'name pattern': (s) => NAME_PATTERN.test(name(s)) && value(s).length >= 8,
  'entropy': (s) => value(s).length >= 20 && entropy(value(s)) >= 3.6,
  'union': (s) => rules['name pattern'](s) || rules.entropy(s),
  'union + allowlist': (s) => rules.union(s)
    && !ALLOWED.some((r) => r.test(value(s))) && !value(s).includes('/'),
};

const s = (x, n) => String(x).padStart(n);
console.log(`${'rule'.padEnd(24)}${s('hits', 7)}${s('caught', 11)}${s('false accept', 14)}${s('false reject', 14)}`);
for (const [rname, rule] of Object.entries(rules)) {
  const hits = source.filter(([, line]) => rule(line));
  const correct = hits.filter(([, , secret]) => secret).length;
  console.log(`${rname.padEnd(24)}${s(hits.length, 7)}${s(`${correct}/5`, 11)}`
    + `${s(5 - correct, 14)}${s(hits.length - correct, 14)}`);
}

console.log('\nreal secrets the name pattern missed:');
for (const [file, line, secret] of source) {
  if (secret && !rules['name pattern'](line)) {
    console.log(`  ${file.padEnd(24)} name=${name(line).padEnd(10)} `
      + `length=${s(value(line).length, 2)} entropy=${entropy(value(line)).toFixed(2)}`);
  }
}
console.log('harmless strings the entropy rule falsely rejected:');
for (const [file, line, secret] of source) {
  if (!secret && rules.entropy(line)) {
    console.log(`  ${file.padEnd(24)} length=${s(value(line).length, 2)} `
      + `entropy=${entropy(value(line)).toFixed(2)} allowed=${ALLOWED.some((r) => r.test(value(line)))}`);
  }
}

rule                       hits     caught  false accept  false reject
name pattern                  6        3/5             2             3
entropy                       5        2/5             3             3
union                        10        5/5             0             5
union + allowlist             7        5/5             0             2

real secrets the name pattern missed:
  notification/send.mjs    name=config     length=26 entropy=4.09
  search/index.mjs         name=refresh    length=27 entropy=4.75
harmless strings the entropy rule falsely rejected:
  catalog/record.mjs       length=24 entropy=3.61 allowed=true
  notification/sample.mjs  length=22 entropy=3.75 allowed=true
  membership/identity.mjs  length=36 entropy=4.00 allowed=false

The name pattern alone finds three of the five real values. The two it misses sit at the end of the list by name: the variable is named config and refresh. A scan relying on a naming convention cannot see a line that does not follow it, and whoever hardcoded a secret has already not followed it. Entropy catches these two but is worse alone: two of five. Short, regular-shaped values — a nineteen-character password, a hyphen-split token — stay under the threshold.

The union of the two finds all five, at a cost of five false rejects. Three are eliminable by rule: the open catalog ID’s prefix, the setup placeholder, the directory path carrying a slash. The remaining two cannot be eliminated — one is a constant holding the key’s name, the other a member ID, and both look exactly like a secret. The trade-off here runs through the whole course: a false accept costs a value stuck in the code, unrotatable; a false reject costs a two-line manual review. Tuning the threshold to the second cost means accepting the first.

Summary

• The three measures of a secret in production are how many places it stands in, how many requests it breaks while rotating, and how long it works once it leaks; the leak surface measured in the test environment is already at its widest here.
• Rotation without a window rejected 36 of 181 requests, running until 255 seconds after the rotation; no component raised an error.
• The required window is read from arithmetic, not measurement: last-read moment minus rotation moment plus ticket lifetime, 390 seconds here. This run’s 300 seconds also gave zero rejects; the worst alignment gave three.
• Skipping the cleanup step keeps dropped requests at zero, the dashboard clean, and a leaked old key still gets 200 at the 900th second; only the cleanup step completes a rotation.
• Holding the key in few places and deriving a 120-second ticket from it shortens a leaked value’s lifetime sixty-four thousand times against a ninety-day period; fourteen locations cost fifty-six manual steps a year, one location costs four.
• In the hardcoded-value scan, the name pattern caught three of five, entropy two of five; their union caught five of five for five false rejects — three eliminable by rule, two not.

Next Step

Every measurement here stayed inside code we wrote ourselves: we produced the key, chose the ticket lifetime, set the scan’s threshold. But most of the code serving a request is not ours. When the loan service’s process comes up, hundreds of files load, and few are lines the team wrote; each is pinned to a version number that changes one day. The next lesson treats this dependency as a configuration decision — counting the trade-off between upgrade frequency and the accumulating risk versus the versions that break, by running three policies over one dependency tree.