Secret Storage Services

When encryption at rest and in transit were set up, both rested on the same thing: a key or credential sits somewhere, and services read it. That somewhere is a secret storage service, and like every rented layer it hands off a decision. The decision this lesson measures collapses into a single number: the rotation window. When a secret rotates, the old value does not become invalid instantly; for a span, the old and new values are both accepted, and clients pick up the new value during that span. Once the window closes, the old value is rejected, and every client that has not read the new value by then breaks.

The window is measured in two directions at once, and the two directions point opposite ways. A short window produces missed clients; a long window extends the time the old secret stays valid — a leaked value keeps working for that whole span. Which paths secrets leak through — log lines, environment variables, build-output layers — was scanned earlier and is not repeated here; this lesson’s subject is rotation itself.

The measurement runs on the regional measurement network, is fictional, and is a model written with node. No real secret value exists in the model: only secret names, client counts, and times are carried.

CS13. The secret storage service holds 41 secrets: three environments, five services, five secret types, and three organization-wide secrets. CS14. There are 83 client instances reading secrets. Read mode is three-valued: reads every request, reads cached (cache lifetime 240 minutes), reads only at startup (rollout interval 4,320 minutes). CS15. The rotation window is the span where the old and new secret are valid together; a client that has not picked up the new value before the window ends counts as missed and breaks. CS16. Every client instance makes 12 requests a minute (fictional unit). CS17. The rotation period is 90 days (fictional policy). Some secrets are rotated automatically, some manually, and some never. CS18. Randomness comes from a self-written linear generator; the seed is 20260415 and appears in the output. Human reads in the last 90 days are counted in the model.

Scanning the Window

// secret.mjs — secret storage service, client inventory, and rotation-window scan. MODEL.
// The regional measurement network is FICTIONAL. NO REAL SECRET EXISTS: only secret NAMES and counts are modeled.
// Randomness comes from a self-written linear generator; the seed is visible below.
export const SEED = 20260415;
export const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };

const ENV = ["development", "staging", "production"];
const CATALOG = [
  ["collector", ["database-password", "intake-key"]],
  ["validator", ["database-password", "signing-key"]],
  ["billing", ["database-password", "external-endpoint-token", "signing-key"]],
  ["work-order", ["database-password", "external-endpoint-token"]],
  ["archivist", ["database-password", "maintenance-identity"]],
];
export const SECRET = [
  ...ENV.flatMap((e) => CATALOG.flatMap(([s, tl]) => tl.map((t) =>
    ({ name: `${e}/${s}/${t}`, env: e, service: s, type: t })))),
  ...CATALOG.map(([s]) => ({ name: `production/${s}/maintenance-identity`, env: "production", service: s,
    type: "maintenance-identity" })),
  ...["measurement-network-maintenance", "secondary-region-copy", "partner-feed"].map((a) =>
    ({ name: `org/${a}`, env: "org", service: "shared", type: "external-endpoint-token" })),
];

// Client instances: the processes reading each secret. Read mode determines pickup time.
const r = generator(SEED);
export const CLIENT = [];
for (const s of SECRET) {
  const n = 1 + Math.floor(r() * ({ production: 6, org: 3, staging: 2, development: 2 })[s.env]);
  for (let i = 0; i < n; i++) {
    const u = r();
    CLIENT.push({ secret: s.name, env: s.env, phase: r(),
      mode: u < 0.45 ? "cache" : u < 0.75 ? "every-request" : "startup" });
  }
}
for (const s of SECRET) {                                 // rotation history and human reads
  s.auto = r() < 0.55;
  s.age = 40 + Math.floor(r() * 860);                     // days since the secret was created
  s.last = s.auto ? Math.floor(r() * 90) : (r() < 0.45 ? Math.floor(r() * 420) : null);
  s.human = Math.floor(r() * 4);                           // human reads in the last 90 days
}

export const REQ_PER_MIN = 12;                             // requests/min per client instance (fictional)
export const PROFILE = {                                   // the client side's pickup behavior
  "current state": { cache: 240, rollout: 4320, changed: [] },
  "cache 60 min": { cache: 60, rollout: 4320, changed: ["cache"] },
  "startup readers": { cache: 240, rollout: 60, changed: ["startup"] },
  "everyone re-reads": { cache: 60, rollout: 60, changed: ["cache", "startup"] },
};
export const pickup = (c, p) => c.mode === "every-request" ? 1
  : Math.ceil(c.phase * (c.mode === "cache" ? p.cache : p.rollout));
export const missed = (p, W) => CLIENT.filter((c) => pickup(c, p) > W);
export const PERIOD = 90, YEAR = 365;                      // fictional rotation period, days

const B = [13, 20, 18, 18, 24];
export const row = (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))
  .join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  const modeCount = (m) => CLIENT.filter((c) => c.mode === m).length;
  console.log(`model (seed ${SEED}): ${SECRET.length} secrets, ${CLIENT.length} client instances — ` +
    `${modeCount("every-request")} read every request, ${modeCount("cache")} cached, ` +
    `${modeCount("startup")} read only at startup`);
  const p = PROFILE["current state"];
  console.log(`\ncurrent state: cache lifetime ${p.cache} min, rollout interval ${p.rollout} min; ` +
    `latest pickup ${Math.max(...CLIENT.map((c) => pickup(c, p)))} min`);
  console.log("\n" + row(["window(min)", "missed clients", "secrets affected", "broken req/min",
    "yearly exposure (min)"]));
  const yearly = Math.round((YEAR / PERIOD) * SECRET.length);   // total rotations per year
  for (const W of [0, 1, 30, 60, 120, 240, 720, 1440, 2880, 3624]) {
    const k = missed(p, W);
    console.log(row([W, k.length, new Set(k.map((c) => c.secret)).size, k.length * REQ_PER_MIN,
      W * yearly]));
  }
}

model (seed 20260415): 41 secrets, 83 client instances — 27 read every request, 38 cached, 18 read only at startup

current state: cache lifetime 240 min, rollout interval 4320 min; latest pickup 3624 min

window(min)        missed clients  secrets affected    broken req/min   yearly exposure (min)
0                              83                40               996                       0
1                              56                34               672                     166
30                             53                32               636                    4980
60                             44                30               528                    9960
120                            38                29               456                   19920
240                            18                17               216                   39840
720                            14                14               168                  119520
1440                           10                10               120                  239040
2880                            3                 3                36                  478080
3624                            0                 0                 0                  601584

The two ends of the table frame the decision. At the top, the window is zero: the old value is rejected instantly, all 83 client instances miss, and 996 requests a minute drop. At the bottom, the window is 3,624 minutes — sixty hours — and no client misses. Between the two is not a choice; it is a trade-off curve.

The middle rows show the shape of the trade-off. A one-hour window misses 44 clients: it affects 30 secrets and cuts 528 requests a minute. Four hours (240 minutes) drops the missed count to 18, because the cache lifetime is 240 minutes and at that boundary every cached client has already refreshed the value. The remaining 18 are clients that read only at startup, and they do not read at four hours — only at the next rollout. Even a one-day window still misses 10.

The last column counts the cost in the opposite direction. 41 secrets, on a 90-day period, rotate 166 times a year; on every rotation the old value stays valid for the length of the window. At the zero-miss window, the yearly total comes to 601,584 minutes — equivalent to 417 days, meaning that at any moment in the year, an average of 1.14 secrets are still also working with their old value. A leaked secret is accepted for that whole span regardless of when the leak is noticed. At a one-hour window, the same total drops to 9,960 minutes — one sixtieth.

Neither number is acceptable: not the 44 broken clients, and not the old secret staying open for 417 days a year. The window is not a knob to turn on its own; no window value is good unless the client side changes.

The Client Side and Secrets That Are Never Rotated

// rotation.mjs — the client side of auto-rotation, never-rotated secrets, and read surface.
import { SECRET, CLIENT, PROFILE, pickup, missed, REQ_PER_MIN, PERIOD } from "./secret.mjs";

const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))
  .join("");
const rowP = gen([21, 10, 14, 24, 15, 17]);
const median = (l) => l.length ? [...l].sort((a, b) => a - b)[Math.floor(l.length / 2)] : 0;

console.log(rowP(["profile", "changing", "window(min)", "yearly exposure (min)", "60 min missed",
  "60 min req/min"]));
for (const [name, p] of Object.entries(PROFILE)) {
  const latest = Math.max(...CLIENT.map((c) => pickup(c, p)));
  const changing = CLIENT.filter((c) => p.changed.includes(c.mode)).length;
  const m60 = missed(p, 60).length;
  console.log(rowP([name, changing, latest, latest * Math.round((365 / PERIOD) * SECRET.length), m60,
    m60 * REQ_PER_MIN]));
}

const never = SECRET.filter((s) => s.last === null);
const auto = SECRET.filter((s) => s.auto);
const manual = SECRET.filter((s) => s.last !== null && !s.auto);
const rowS = gen([24, 9, 15, 12, 20]);
console.log("\n" + rowS(["rotation class", "secrets", "latest (days)", "median", "left/over period"]));
const emit = (name, l, field) => {
  const v = l.map((s) => s[field]);
  const latest = Math.max(...v, 0);
  console.log(rowS([name, l.length, latest, median(v),
    latest <= PERIOD ? `${PERIOD - latest} days left` : `${latest - PERIOD} days over`]));
};
emit("auto-rotated", auto, "last");
emit("manually rotated", manual, "last");
emit("never rotated", never, "age");
for (const s of [...never].sort((a, b) => b.age - a.age).slice(0, 4))
  console.log(`  ${s.name.padEnd(38)} ${String(s.age).padStart(4)} days, ` +
    `${CLIENT.filter((c) => c.secret === s.name).length} clients, ` +
    `${s.human} human reads in the last 90 days`);

const humanTouched = SECRET.filter((s) => s.human > 0);
const intersection = humanTouched.filter((s) => s.last === null);
console.log(`\nread surface: ${humanTouched.length}/${SECRET.length} secrets were read by a human in ` +
  `the last 90 days (${SECRET.reduce((a, s) => a + s.human, 0)} reads, all in the audit log)`);
console.log(`${intersection.length} of these are never-rotated secrets; ` +
  `${CLIENT.filter((c) => intersection.some((s) => s.name === c.secret)).length} client instances ` +
  `read these ${intersection.length} secrets`);
const widest = SECRET.map((s) => [s.name, CLIENT.filter((c) => c.secret === s.name).length])
  .sort((a, b) => b[1] - a[1])[0];
console.log(`widest secret: ${widest[0]} — ${widest[1]} client instances; ` +
  `when it rotates, all ${widest[1]} of them must pick up the new value`);

profile                changing   window(min)   yearly exposure (min)  60 min missed   60 min req/min
current state                 0          3624                  601584             44              528
cache 60 min                 38          3624                  601584             18              216
startup readers              18           227                   37682             26              312
everyone re-reads            56            57                    9462              0                0

rotation class            secrets  latest (days)      median    left/over period
auto-rotated                   16             88          64         2 days left
manually rotated                8            371         209       281 days over
never rotated                  17            826         442       736 days over
  production/work-order/maintenance-identity  826 days, 5 clients, 1 human reads in the last 90 days
  staging/archivist/maintenance-identity  789 days, 1 clients, 1 human reads in the last 90 days
  org/partner-feed                        760 days, 1 clients, 0 human reads in the last 90 days
  production/validator/signing-key        748 days, 6 clients, 2 human reads in the last 90 days

read surface: 34/41 secrets were read by a human in the last 90 days (68 reads, all in the audit log)
15 of these are never-rotated secrets; 34 client instances read these 15 secrets
widest secret: production/collector/database-password — 6 client instances; when it rotates, all 6 of them must pick up the new value

The profile table delivers this lesson’s most counterintuitive result. Pulling the cache lifetime from 240 minutes down to 60 touches 38 clients and does not shorten the window at all: the zero-miss window stays at 3,624 minutes, and the yearly exposure stays at 601,584 minutes. The only thing it gains is dropping the one-hour-window miss count from 44 to 18. The reason is plain: the binding constraint is not the cache — it is the 18 clients that read only at startup; they never listen to the cache lifetime at all.

The third row confirms this. Making just those 18 clients re-readable — with less than half the touches — drops the window from 3,624 minutes to 227 minutes, and the yearly exposure from 601,584 to 37,682 minutes: one sixteenth. The fourth row combines both: 56 clients change, the window drops to 57 minutes, the yearly exposure becomes 9,462 minutes, and the one-hour-window miss count is zero. This is what auto-rotation requires on the client side — the ability to re-read a secret comes first, cache lifetime second. Without re-reading, rotation waits for a rollout every single time.

The change itself is also a line item and sits in the first column of the table. Adding re-read capability asks for a code change in 18 clients; shortening the cache lifetime asks for a configuration change in 38. Both together make 56 instances, and each of those 56 instances is a separate deployment item. Sequencing matters for this reason: the 18-client work gains sixteen times more than the 38-client work, and done alone it drops the window below four hours. If nothing is done, the rotation period does not function — that is the technical counterpart of 17 secrets having never been rotated.

The lower table is the inventory face of the same problem. 16 of the 41 secrets are auto-rotated, and the most overdue one has 2 days left in its 90-day period. 8 secrets are manually rotated, and the most overdue one has run 281 days past the period. The remaining 17 secrets have never been rotated — forty percent of the inventory — and the oldest is 826 days old. Measuring this did not need a separate scan: records with an empty last-rotation field were counted.

The last three lines set the order. 34 of the 41 secrets were read by a human at least once in the last 90 days — 68 reads in total, all sitting in the audit log. 15 of those 34 secrets are never-rotated secrets, and 34 client instances read those 15. The intersection gives the rotation order: a secret that a human has touched and has never been rotated comes before the rest of the inventory. The widest secret is the production collector’s database password; 6 client instances read it, and when it rotates, all six need to pick up the new value — the window decision is made precisely for those six instances.

Summary

• The rotation window was measured in two directions: at a zero window all 83 clients miss, at a 3,624-minute window none miss; at a one-hour window it is 44 clients and 528 requests a minute.
• The cost of a long window was counted: 41 secrets rotate 166 times a year, and a 3,624-minute window leaves an old secret standing for 601,584 minutes a year; at a one-hour window the same total is 9,960 minutes.
• Cutting the cache lifetime to a quarter touches 38 clients and does not shorten the zero-miss window at all; the binding constraint is the 18 clients that read only at startup.
• Making those 18 clients re-readable drops the window to 227 minutes, and both together to 57 minutes; in the second case, the one-hour-window miss count is zero.
• 16 of the 41 secrets are auto-rotated (2 days left in the period), 8 are manual (281 days over), and 17 have never been rotated; the oldest is 826 days old.
• 34 secrets were read by a human in the last 90 days (68 reads, all in the log), and 15 of them are never-rotated secrets; the rotation order comes from that intersection.

Next Step

Encryption is in place, secrets have become rotatable, and the same sentence appeared in every measurement: “all in the audit log.” That sentence has not been tested yet. The next lesson measures the record itself: a set of events is dropped into the audit log, and how many questions it can answer is counted — who, when, which resource, from which identity. The operation type the record does not cover is named, who can delete the record is counted, and retention period and cost are scanned together.