--- title: 'Secret Management Systems' source: 'https://academia.sh/en/courses/servers-and-hardening/secret-management-systems' course: 'Servers, Networking and Hardening' language: en updated: '2026-09-13T05:08:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts copy, location, and rotation cost in the static secret arrangement, and shows why the exposure window does not close' - 'Scans lifetime: the exposure window, the number of renewals in the period, and the number of instances unable to work on a store outage are read together' - "Measures the central store's own surface — how many principals access how many secrets, and what happens when the store goes down" --- # Secret Management Systems A secret's axis is lifetime: in a static secret the exposure window is infinite, in a credential with a defined lifetime the window is as long as the lifetime. Lifetime is scanned, renewal load and resilience to store outages are counted, and the central store's own access surface is worked out item by item. The previous lesson's key was valid until revoked; that was the one item that could not be measured. The same is true of the credentials applications carry: the data-store credential, the queue key, and the mail-gateway token are written on install day, and nowhere does it say when they will change. How secrets are stored in production was measured in M22/K01, and how the deployment pipeline reaches secrets was measured in M22/K04; neither is repeated here. This lesson's only axis is lifetime — how long a credential stays valid, and how wide the window stays open if it leaks. - **HB13.** Five secrets, six services, and three environments belong to the fictional regional measurement network. No real secret was written, the names are fictional; what is measured is the model written with `node`. - **HB14.** The period was counted as thirty days, that is 43,200 minutes. - **HB15.** On a store outage, instances' remaining lifetime was spread evenly across the lifetime interval; there is no randomness. In reality the distribution depends on renewal timing, but the direction of the result does not change. - **HB16.** A credential is either valid or it is not. Partial authorization, retry on renewal failure, and cached sessions are not modeled. - **HB17.** A secret's content, how it is generated, and how it is encrypted are not measured. The only things measured are lifetime and access items. - **HB18.** The central store was modeled as a single place. A replicated store would lower the outage item, not the access item. ## Static Secret Arrangement The model holds five secrets, the six services that need secrets, three environments, and the locations a secret sits in under the static arrangement. The lifetime list gives the values to be scanned; `null` denotes no lifetime, that is, a static secret. ```js // secret.mjs — secret set, access matrix, and lifetime model; no real secret, names are fictional export const SECRET = ['data-store-credential', 'queue-key', 'mail-gateway-token', 'metric-write-key', 'archive-key']; export const SERVICE = { // service -> secrets it needs 'read-collector': ['data-store-credential', 'queue-key'], 'validator': ['data-store-credential', 'queue-key'], 'billing': ['data-store-credential', 'mail-gateway-token'], 'work-order': ['data-store-credential', 'queue-key', 'mail-gateway-token'], 'nightly-batch': ['data-store-credential', 'archive-key'], 'metric-collector': ['metric-write-key'], }; export const ENV = ['dev', 'staging', 'prod']; export const INSTANCE = Object.keys(SERVICE).flatMap((s) => ENV.map((e) => ({ service: s, env: e }))); export const STATIC_LOCATION = ['config-repo', 'env-var', 'backup-image', 'developer-laptop']; export const PERIOD_MIN = 30 * 24 * 60; // thirty-day period export const LIFETIME = [5, 15, 60, 480, 1440, null]; // minutes; null = static, no lifetime // Human and automation principals; which secrets each can access. export const PRINCIPAL = { 'op-a': SECRET, 'op-b': ['data-store-credential', 'queue-key', 'archive-key'], 'op-c': ['mail-gateway-token'], 'deploy': SECRET, 'monitoring': ['metric-write-key'], }; // Not every secret can be generated dynamically; the second field is the reason. export const DYNAMIC = { 'data-store-credential': [true, 'the store account can be opened and closed on demand'], 'queue-key': [true, 'the queue can grant temporary authorization'], 'metric-write-key': [true, 'write authorization can be granted short-term'], 'mail-gateway-token': [false, 'issued by an external party, we cannot generate it'], 'archive-key': [false, 'an encryption key; rotating it makes the old archive unreadable'], }; export const instanceItem = () => INSTANCE.reduce((a, o) => a + SERVICE[o.service].length, 0); export const principalItem = () => Object.values(PRINCIPAL).reduce((a, s) => a + s.length, 0); export const rotationEffect = (s) => INSTANCE.filter((o) => SERVICE[o.service].includes(s)).length; // Each instance's remaining lifetime on a store outage: interval split into equal steps, no randomness. export const remainingLifetime = (life, n = INSTANCE.length) => Array.from({ length: n }, (_, i) => (life * (i + 0.5)) / n); ``` ```js // run1.mjs — items and rotation cost in the static secret arrangement are counted import { SECRET, SERVICE, ENV, INSTANCE, STATIC_LOCATION, instanceItem, rotationEffect } from './secret.mjs'; for (const [label, v, note = ''] of [ ['secret', SECRET.length], ['service x env = instance', INSTANCE.length, `${Object.keys(SERVICE).length} services, ${ENV.length} envs`], ['(instance, secret) item', instanceItem()], ['distinct location a secret sits in', STATIC_LOCATION.length, STATIC_LOCATION.join(', ')], ['secret copy (secret x location)', SECRET.length * STATIC_LOCATION.length], ["secret's lifetime", 'none', 'unchanged since install'], ]) console.log(`${label.padEnd(36)}${String(v).padStart(6)} ${note}`.trimEnd()); console.log('\nrotating a secret restarts how many instances:'); for (const s of SECRET) console.log(` ${s.padEnd(24)} ${rotationEffect(s)}`); ``` ``` secret 5 service x env = instance 18 6 services, 3 envs (instance, secret) item 36 distinct location a secret sits in 4 config-repo, env-var, backup-image, developer-laptop secret copy (secret x location) 20 secret's lifetime none unchanged since install rotating a secret restarts how many instances: data-store-credential 15 queue-key 9 mail-gateway-token 6 metric-write-key 3 archive-key 3 ``` Five secrets, spread across eighteen service instances, come to thirty-six `(instance, secret)` items. On top of that, a secret sits in four separate places: the config repo, the running process's environment variable, the backup image, and the developer's laptop. Five secrets times four locations, twenty copies. Two of these locations — the backup image and the laptop — keep no access record; a secret read from there shows up nowhere as having been read. The bottom block gives the reason rotation does not happen, in numbers. The data-store credential is used in fifteen instances; changing it means restarting fifteen instances, and these fifteen instances are spread across three separate environments. Rotation's cost is exactly the reason it does not happen. The result is the "lifetime: none" row in the table — the secret has been the same since install, so if it leaked, it has been valid since the day it leaked. The exposure window here is not a duration, it is a state: open until somebody notices. ## Lifetime Scan ```js // run2.mjs — lifetime is scanned: exposure window, renewal load, and how it survives a store outage import { INSTANCE, PERIOD_MIN, LIFETIME, remainingLifetime } from './secret.mjs'; const d1 = (x) => (Math.round(x * 10) / 10).toFixed(1); console.log('lifetime exposure window in period on store outage on 30 min outage'); console.log('(min) (avg min) renewals first drop / all (min) instances unable'); for (const life of LIFETIME) { if (life === null) { console.log(`${'static'.padEnd(10)} ${'infinite'.padEnd(18)} ${String(0).padEnd(11)} ` + `${'-'.padEnd(24)} 0`); continue; } const left = remainingLifetime(life); console.log(`${String(life).padEnd(10)} ${d1(life / 2).padEnd(18)} ` + `${String(INSTANCE.length * Math.round(PERIOD_MIN / life)).padEnd(11)} ` + `${`${d1(left[0])} / ${d1(left[left.length - 1])}`.padEnd(24)} ${left.filter((k) => k < 30).length}`); } ``` ``` lifetime exposure window in period on store outage on 30 min outage (min) (avg min) renewals first drop / all (min) instances unable 5 2.5 155520 0.1 / 4.9 18 15 7.5 51840 0.4 / 14.6 18 60 30.0 12960 1.7 / 58.3 9 480 240.0 1620 13.3 / 466.7 1 1440 720.0 540 40.0 / 1400.0 0 static infinite 0 - 0 ``` The table lays three separate outcomes of one decision side by side. At a five-minute lifetime, the exposure window drops to an average of two and a half minutes — a leaked credential is almost useless. The price is in the two columns on the right: one hundred fifty-five thousand five hundred twenty renewals during the period, and once an outage starts at the store, the first instance becomes unable to work within six seconds, all of them within five minutes. The static row is the exact opposite: zero renewals, no effect from an outage at all, and an infinite window. The rows in between are where the decision actually gets made. At a sixty-minute lifetime the window drops to half an hour, renewals fall to twelve thousand nine hundred sixty, but on a thirty-minute store outage, nine of the eighteen instances become unable to work. At an eight-hour lifetime the same outage drops only one instance, and the window rises to four hours. At a twenty-four-hour lifetime the outage does not drop a single instance, and the window is twelve hours. The second column from the right is this table's real finding: lifetime sets the outage duration the store can tolerate. As long as the outage stays under the chosen lifetime, most instances stay up; once the outage exceeds the lifetime, all of them drop. So the decision to shorten the exposure window is, without anyone noticing, the decision that the store has to run without outages. When a five-minute lifetime is chosen, the store becomes a dependency that cannot disappear for longer than five minutes for all eighteen instances at once — and that introduces a new availability item that no secret used to carry. ## Secrets Whose Lifetime Cannot Be Shortened The lifetime scan assumes every secret's lifetime can be chosen. How many of the five secrets actually have a choosable lifetime is a separate number. ```js // run4.mjs — not every secret can be generated dynamically; items outside the lifetime axis are counted import { SECRET, SERVICE, INSTANCE, DYNAMIC, instanceItem } from './secret.mjs'; console.log('secret dynamic reason'); for (const s of SECRET) console.log(` ${s.padEnd(24)} ${(DYNAMIC[s][0] ? 'yes' : 'no').padEnd(9)} ${DYNAMIC[s][1]}`); const static_ = (s) => !DYNAMIC[s][0]; const item = INSTANCE.reduce((a, o) => a + SERVICE[o.service].filter(static_).length, 0); const inst = INSTANCE.filter((o) => SERVICE[o.service].some(static_)); console.log(''); for (const [label, v] of [ ['secret whose lifetime can be shortened', `${SECRET.filter((s) => !static_(s)).length} / ${SECRET.length}`], ['secret that must stay static', `${SECRET.filter(static_).length} / ${SECRET.length}`], ['static (instance, secret) item', `${item} / ${instanceItem()}`], ['instance carrying at least one static secret', `${inst.length} / ${INSTANCE.length}`], ['exposure window in these instances', 'infinite'], ]) console.log(`${label.padEnd(46)}${v}`); console.log(`\ncarrying services: ${[...new Set(inst.map((o) => o.service))].join(', ')}`); ``` ``` secret dynamic reason data-store-credential yes the store account can be opened and closed on demand queue-key yes the queue can grant temporary authorization mail-gateway-token no issued by an external party, we cannot generate it metric-write-key yes write authorization can be granted short-term archive-key no an encryption key; rotating it makes the old archive unreadable secret whose lifetime can be shortened 3 / 5 secret that must stay static 2 / 5 static (instance, secret) item 9 / 36 instance carrying at least one static secret 9 / 18 exposure window in these instances infinite carrying services: billing, work-order, nightly-batch ``` Dynamic generation is possible for three secrets: the data-store account can be opened and closed on demand, the queue can grant temporary authorization, metric write authorization can be defined short-term. It is not possible for two secrets, and their reasons are different in kind. The mail-gateway token is issued by an external party; we are not the party that sets its lifetime, so there is nothing to do on the lifetime axis. The archive key, on the other hand, is an encryption key; when it is rotated the old archive becomes unreadable, meaning shortening its lifetime breaks the secret's function. The numbers narrow the scan table's scope. Nine of the thirty-six `(instance, secret)` items fall outside the lifetime axis, and nine of the eighteen instances — billing, work-order, and nightly-batch — keep carrying at least one static secret. For these nine instances, every column of the previous section is void: the exposure window is not as long as the lifetime, it is infinite. Even if a five-minute lifetime is chosen, half the system carries a static secret. The lifetime axis is a real axis, but it cannot be applied to every secret, and the ones it cannot be applied to are exactly the two secrets that face outward: the token going to the mail gateway and the key that opens the archive. ## The Central Store's Own Surface ```js // run3.mjs — the central store's own surface: how many principals can access how many secrets import { SECRET, INSTANCE, STATIC_LOCATION, PRINCIPAL, instanceItem, principalItem } from './secret.mjs'; console.log('principal secret accessible'); for (const [o, s] of Object.entries(PRINCIPAL)) console.log(` ${o.padEnd(12)} ${String(s.length).padEnd(3)} ${s.length === SECRET.length ? '(all)' : s.join(', ')}`); console.log(` ${'service'.padEnd(12)} ${instanceItem()} (${INSTANCE.length} instances, avg ${instanceItem() / INSTANCE.length} secrets)`); const total = principalItem() + instanceItem(); console.log(`\nmeasure static central store`); for (const [label, a, b] of [ ['distinct location a secret sits in', STATIC_LOCATION.length, 1], ['secret copy', SECRET.length * STATIC_LOCATION.length, SECRET.length], ['point carrying every secret', STATIC_LOCATION.length, 1], ['(principal, secret) item', total, total], ['logged and revocable', 0, total], ['principal that can access every secret', Object.values(PRINCIPAL).filter((s) => s.length === SECRET.length).length, Object.values(PRINCIPAL).filter((s) => s.length === SECRET.length).length], ['instance that cannot renew if the store goes down', 0, INSTANCE.length], ]) console.log(`${label.padEnd(51)} ${String(a).padEnd(8)} ${b}`); ``` ``` principal secret accessible op-a 5 (all) op-b 3 data-store-credential, queue-key, archive-key op-c 1 mail-gateway-token deploy 5 (all) monitoring 1 metric-write-key service 36 (18 instances, avg 2 secrets) measure static central store distinct location a secret sits in 4 1 secret copy 20 5 point carrying every secret 4 1 (principal, secret) item 51 51 logged and revocable 0 51 principal that can access every secret 2 2 instance that cannot renew if the store goes down 0 18 ``` The top block is the store's own access surface. Two of the five principals can access all five of the five secrets at once: one operator and the deployment automation. The matrix is written, genuinely narrowed for three principals, not narrowed at all for two. Service instances add thirty-six items with an average of two — services are narrow, people are broad. There are fifty-one `(principal, secret)` items in total, and each one is a secret that opens if that principal is compromised. The bottom table lays the two arrangements side by side, and the first three rows are the central store's gain: the location a secret sits in drops from four to one, copies from twenty to five, the point carrying every secret from four to one. The fourth row is not a gain: the `(principal, secret)` item stays at fifty-one. The central store does not reduce the access count. The static arrangement had the same fifty-one access items; the difference was that none of them were written down anywhere. The fifth row is this lesson's real gain. In the static arrangement, logged and revocable access was zero — a secret read from the backup image had neither a record nor an individually revocable state. In the central store, fifty-one of the fifty-one accesses are both loggable and revocable. What the central store gives is not less access, it is access that can be counted and revoked. The last row repeats the price: when the store goes down, eighteen instances cannot renew, whereas zero instances would have been affected in the static arrangement. This is also where the item that cannot be closed sits — the single point carrying every secret. The static arrangement had four of these points, the central store has one. Going from four to one is a real gain, but it does not reach zero, and that single point now carries both every secret and every service's ability to keep working. ## Summary - In the static arrangement, 5 secrets spread across 18 instances come to 36 `(instance, secret)` items and sit as 20 copies total across 4 separate locations; two of these locations keep no access record. Rotation cost is the reason rotation does not happen: changing the most-shared secret requires restarting 15 instances, so the secret has been the same since install and the exposure window is a state, not a duration — open until somebody notices. - The lifetime scan gives three outcomes together: at a 5-minute lifetime the window is 2.5 minutes, but the period has 155,520 renewals and an outage drops all 18 instances within 5 minutes; in the static row the window is infinite, renewals are 0, and an outage has no effect at all. - Lifetime sets the outage duration the store can tolerate: at a 60-minute lifetime a 30-minute outage drops 9 of 18 instances, at 480 minutes it drops 1, at 1440 minutes it drops none. - The lifetime axis cannot be applied to every secret: 2 of the 5 secrets must stay static (one is issued by an external party, the other is an encryption key), and 9 of the 36 items, 9 of the 18 instances, fall outside the lifetime axis and keep an infinite window. - The central store drops the location a secret sits in from 4 to 1 and copies from 20 to 5, but the `(principal, secret)` item stays at 51; 2 of the 5 principals access every secret. The gain is not less access, it is loggable and revocable access going from 0 to 51. - The item that cannot be closed is the single point carrying every secret: there were 4 in the static arrangement, 1 in the central store — but that single point now also carries the working ability of 18 instances. ## Next Step These three lessons left three separate points of concentration: the management node that faces outward, the bastion host that all access passes through, and the store that carries every secret. All three share one trait — all three are software, and all three run software that ages. A hardening step holds only as long as the code it runs has no known vulnerability. The next lesson's question is therefore patches: between the moment a fix is published and the moment it is applied to machines, how many machines stay open for how long; how many restarts and how many minutes of outage does narrowing the patch window cost; and what does the difference between an automatically applied patch and an approved-then-applied patch come to, in days and in machines.