--- title: 'Secret Store Architecture' source: 'https://academia.sh/en/courses/runtime-security/secret-store-architecture' course: 'Runtime Security and Secrets Management' language: en updated: '2026-10-07T04:31:06+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the secret store as an access matrix: how many items the policy opens, how many the read log confirms, and how large the gap between them is' - 'Counts the items that dropping to least privilege closes together with the rare legitimate reads it cuts' - 'Derives the single point that centralization opens as a startup dependency and writes down at which earlier stage the same item could have been closed' --- # Secret Store Architecture The secret store is measured in terms of access control: the path-prefix policy opens 79 (app, secret) items to nine apps, the 90-day read log says 31, and 48 items sit there never used. No app reads three of the secrets; six apps can reach the widest data secret. The price of centralization is a single point — none of the nine apps can start without the store, and 41 start attempts a day pass through that one point. The previous lesson closed the request layer: it counted what the web application firewall and behavior scoring catch, what they miss, and what they cut wrongly. Behind all of those layers stood one silent assumption — that the keys the application carries to connect to the data store, the queue, and the external gateways are **in place and held only by the application itself**. This lesson removes that assumption and measures the place where the keys sit, the **secret store**, along a single axis: who can reach what. Where a secret is kept, how many copies of it exist, and how a central store reduces the copies were measured earlier and are not repeated here. The question here begins **after** the store is built: the store exists, the secrets are in it, and a policy has been written for each app. How many items does that policy allow, how many do the apps actually read, and how much does the difference expose from a compromised app. - **SM1.** The production environment of the subscriber portal and the field app is **fictional**: nine apps, twenty secrets, and one secret store. What is measured is a **model** written in `node`. - **SM2.** The model holds **no real secret value**; it carries only secret paths, app names, and numbers. A secret path has the form `domain/item` and the policy is written with a **path prefix**. - **SM3.** A policy rule opens every secret path that starts with it. The `data/` rule gives all five data secrets at once; a rule that spells out a full path gives one secret. - **SM4.** The read log is read in two windows: the last thirty days and the last ninety days. The ninety-day window counts as real need in this model; in practice no window shows the whole of the need. - **SM5.** There is no randomness; every number is derived from the model tables and the output is the same from run to run. Restart counts are fictional units. - **SM6.** The store's own outage is modeled as single-valued: the store is either up or down. Partial degradation and a replicated store are not modeled; replication reduces the outage item, not the access item. ## Access Matrix ```js // store.mjs — secret store inventory, policy rules, and read log. A MODEL. // The production environment of the subscriber portal and field app is FICTIONAL; nine apps, one store. // THERE ARE NO REAL SECRETS: only secret PATHS, app names, and numbers are modeled. export const SECRETS = Object.entries({ data: ["portal-primary", "portal-read", "field-primary", "billing-primary", "archive"], queue: ["measurement-intake", "notification-queue", "work-order-queue"], external: ["payment-gateway", "map-service", "sms-gateway", "email-gateway", "identity-provider"], signing: ["session-signing", "document-signing", "internal-call-signing"], infra: ["object-store", "cache", "metrics-write", "log-write"], }).flatMap(([d, l]) => l.map((k) => `${d}/${k}`)); // rules: the path prefixes the policy grants. reads: secrets in the read log of the last 30 days. // format/life: how the secret is obtained. restarts: restarts per day (deploys + scaling). export const APPS = { "portal-frontend": { rules: ["external/identity-provider", "signing/"], format: "at-startup", life: 0, restarts: 6, reads: ["signing/session-signing"] }, "portal-api": { rules: ["data/", "signing/", "infra/"], format: "cached", life: 60, restarts: 12, reads: ["data/portal-primary", "data/portal-read", "signing/session-signing", "signing/internal-call-signing", "infra/cache"] }, "subscriber-signup": { rules: ["data/", "external/"], format: "at-startup", life: 0, restarts: 3, reads: ["data/portal-primary", "external/email-gateway", "external/sms-gateway"] }, "billing-service": { rules: ["data/", "external/", "signing/document-signing"], format: "cached", life: 240, restarts: 2, reads: ["data/billing-primary", "data/portal-read", "external/payment-gateway", "signing/document-signing"] }, "payment-bridge": { rules: ["external/"], format: "per-request", life: -1, restarts: 2, reads: ["external/payment-gateway"] }, "field-api": { rules: ["data/", "queue/", "external/map-service", "signing/"], format: "cached", life: 60, restarts: 8, reads: ["data/field-primary", "queue/work-order-queue", "external/map-service", "signing/internal-call-signing"] }, "work-order": { rules: ["data/", "queue/"], format: "at-startup", life: 0, restarts: 4, reads: ["data/field-primary", "queue/work-order-queue", "queue/notification-queue"] }, "notification": { rules: ["queue/", "external/"], format: "cached", life: 15, restarts: 3, reads: ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"] }, "report-generator": { rules: ["data/", "infra/"], format: "at-startup", life: 0, restarts: 1, reads: ["data/portal-read", "data/archive", "infra/object-store"] }, }; export const NAMES = Object.keys(APPS); // Legitimate reads that never show in the 30-day window but do show in 90 days. export const RARE = [["billing-service", "data/archive"], ["work-order", "external/map-service"], ["portal-api", "infra/log-write"], ["report-generator", "external/email-gateway"]]; export const reaches = (a, s) => APPS[a].rules.some((k) => s.startsWith(k)); export const reads90 = (a) => [...APPS[a].reads, ...RARE.filter((n) => n[0] === a).map((n) => n[1])]; export const sum = (f) => NAMES.reduce((t, a) => t + f(a), 0); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([18, 8, 14, 12, 12, 11]); console.log(`model: ${SECRETS.length} secrets, ${NAMES.length} apps, 1 store (fictional production environment)\n`); console.log(row(["app", "rules", "can reach", "30d reads", "90d reads", "excess"])); for (const a of NAMES) { const e = SECRETS.filter((s) => reaches(a, s)).length; console.log(row([a, APPS[a].rules.length, e, APPS[a].reads.length, reads90(a).length, e - reads90(a).length])); } const E = sum((a) => SECRETS.filter((s) => reaches(a, s)).length); const R30 = sum((a) => APPS[a].reads.length), R90 = sum((a) => reads90(a).length); console.log(row(["total", sum((a) => APPS[a].rules.length), E, R30, R90, E - R90])); const bySecret = SECRETS.map((s) => [s, NAMES.filter((a) => reaches(a, s)).length, NAMES.filter((a) => reads90(a).includes(s)).length]).sort((x, y) => y[1] - x[1]); console.log("\nwidest four secrets (how many apps can reach / how many actually read):"); for (const [s, e, o] of bySecret.slice(0, 4)) console.log(` ${s.padEnd(24)} ${e} / ${o}`); const none = bySecret.filter((t) => t[2] === 0), shared = bySecret.filter((t) => t[2] > 1); console.log(`secrets no app reads: ${none.length} (${none.map((t) => t[0]).join(", ")})`); console.log(`secrets read by more than one app: ${shared.length} / ${SECRETS.length}; ` + `when one of them changes, on average ` + `${(shared.reduce((t, x) => t + x[2], 0) / shared.length).toFixed(1)} apps are affected`); console.log(`\nsecrets exposed by a compromised app — with the policy on average ` + `${(E / NAMES.length).toFixed(1)}, widest ${Math.max(...NAMES.map((a) => SECRETS.filter((s) => reaches(a, s)).length))}; with least privilege on average ` + `${(R90 / NAMES.length).toFixed(1)}, widest ${Math.max(...NAMES.map((a) => reads90(a).length))}`); console.log(`least-privilege policy: ${R90} lines (today ${sum((a) => APPS[a].rules.length)}), ` + `items closed ${E - R90}; if written from 30 days of observation it would cut ${R90 - R30} legitimate reads`); } ``` ``` model: 20 secrets, 9 apps, 1 store (fictional production environment) app rules can reach 30d reads 90d reads excess portal-frontend 2 4 1 1 3 portal-api 3 12 5 6 6 subscriber-signup 2 10 3 3 7 billing-service 3 11 4 5 6 payment-bridge 1 5 1 1 4 field-api 4 12 4 4 8 work-order 2 8 3 4 4 notification 2 8 3 3 5 report-generator 2 9 3 4 5 total 21 79 27 31 48 widest four secrets (how many apps can reach / how many actually read): data/portal-primary 6 / 2 data/portal-read 6 / 3 data/field-primary 6 / 2 data/billing-primary 6 / 1 secrets no app reads: 3 (external/identity-provider, queue/measurement-intake, infra/metrics-write) secrets read by more than one app: 12 / 20; when one of them changes, on average 2.2 apps are affected secrets exposed by a compromised app — with the policy on average 8.8, widest 12; with least privilege on average 3.4, widest 6 least-privilege policy: 31 lines (today 21), items closed 48; if written from 30 days of observation it would cut 4 legitimate reads ``` Twenty-one rule lines open seventy-nine `(app, secret)` items. The items that appear in the ninety-day read log number thirty-one. The **forty-eight items** in between, sixty-one percent of the access opened, are granted on secrets that no app ever touched. This gap does not come from laziness but from the shape of the rule: writing `data/` is one line and gives five secrets; spelling out which of the five secrets is really needed takes five lines. Looking row by row, the two most expensive items sit in two different apps. `field-api` can reach twelve secrets with four rule lines and reads four of them; eight items are unused grants. `portal-api` reaches twelve secrets and reads six. The largest single source of unused grants is the `data/` prefix: **six apps** can reach each of the five data secrets, yet the most-read one is read by three apps and the least-read one by a single app. Six apps can get to the primary credential of the billing database; only one app reads it. Three secrets do not appear in the list at all: the identity provider secret, the key of the measurement intake queue, and the metrics write key were not read by any app for ninety days. These are the cheapest items in the access matrix — a secret nobody reads is closed without cutting anyone. The next row reads the matrix in the opposite direction and gives the one number in this lesson that reaches beyond it: twelve of the twenty secrets are read by more than one app, by 2.2 apps on average. Even if the access matrix is narrowed, this sharing remains — because sharing comes not from the policy but from the architecture; if two apps write to the same database, they read the same credential. When any one of these twelve secrets is changed, on average two apps must move to the new value at the same time, and how that move is made is not a question access control can solve. The last two rows give the verdict. A compromised app opens 8.8 secrets on average under the current policy, twelve in the worst case. Dropping to least privilege takes the average to 3.4 and the worst case to six. The cost is in the same row: the policy grows from twenty-one lines to thirty-one. **Narrowing does not shorten the policy, it lengthens it.** And if the narrowing is written from thirty days of observation, it cuts four legitimate reads — the year-end archive read, the bulk route recalculation, the log write in diagnostic mode, and the monthly summary delivery. None of the four ever shows in the thirty-day window; when they are cut, they are noticed not at deploy time but on the first day they run. ## The Single Point of Centralization ```js // outage.mjs — the single-point cost of centralization: who stops and who cannot start when the store goes down import { NAMES, APPS, sum, gen } from "./store.mjs"; const row = gen([14, 14, 17, 16, 20]); const RESTARTS = sum((a) => APPS[a].restarts); // daily restarts (fictional) const immediate = NAMES.filter((a) => APPS[a].format === "per-request"); const expired = (D) => NAMES.filter((a) => APPS[a].format === "cached" && APPS[a].life < D); console.log(`daily restarts ${RESTARTS} (deploys + scaling); apps that can start without the ` + `store 0 / ${NAMES.length}\n`); console.log(row(["outage (min)", "down at once", "cache expired", "apps down", "failed starts"])); for (const D of [1, 15, 30, 60, 120, 240, 480, 1440]) console.log(row([D, immediate.length, expired(D).length, immediate.length + expired(D).length, Math.round((D / 1440) * RESTARTS)])); console.log(`\nformat distribution:`); for (const b of ["per-request", "cached", "at-startup"]) { const l = NAMES.filter((a) => APPS[a].format === b); console.log(` ${b.padEnd(13)} ${l.length} ${l.join(", ")}`); } const running = NAMES.filter((a) => APPS[a].format === "at-startup"); console.log(`\nat a 1440 min outage: ${immediate.length + expired(1440).length} apps go down, ` + `${running.length} apps keep running but ${RESTARTS} start attempts fail`); console.log(`apps not affected by the outage at all: 0 / ${NAMES.length}`); ``` ``` daily restarts 41 (deploys + scaling); apps that can start without the store 0 / 9 outage (min) down at once cache expired apps down failed starts 1 1 0 1 0 15 1 0 1 0 30 1 1 2 1 60 1 1 2 2 120 1 3 4 3 240 1 3 4 7 480 1 4 5 14 1440 1 4 5 41 format distribution: per-request 1 payment-bridge cached 4 portal-api, billing-service, field-api, notification at-startup 4 portal-frontend, subscriber-signup, work-order, report-generator at a 1440 min outage: 5 apps go down, 4 apps keep running but 41 start attempts fail apps not affected by the outage at all: 0 / 9 ``` What was received in exchange for closing forty-eight items is in this table. A secret gathered in one place is a **dependency** gathered in one place. Even in a one-minute outage of the store, one app goes down: the payment bridge reads its secret on every request, so the moment it cannot reach the store it takes no payments. This is the natural consequence of the most sensitive secret being read most strictly — strict reading is the way of being most exposed to an outage. The following rows show that cache life is a delayed buffer. In a thirty-minute outage the notification service goes down (its cache life is fifteen minutes), in two hours the portal API and the field API go down too, and at eight hours all four cached apps are down. The cache does not prevent the outage, it **postpones** it; the time it postpones is exactly the cache life. The right-hand column is the real finding of the table. The four apps that read at startup keep running throughout the outage and at first glance look immune. They are not immune: **none** of the nine apps can start without the store, and in this environment there are forty-one restarts a day — deploys, scale-ups, and replacement of unhealthy instances. A four-hour outage drops seven start attempts, a one-day outage forty-one. In a store outage that coincides with a load spike, scale-up does not work; the system is left under the load with the instances it has. The last row of the table is the real statement of centralization: the number of apps not affected by the outage at all is zero. Building the store makes forty-eight unused access grants closable; the same step writes a **shared startup dependency** onto all nine apps. The second item is the bill of the first, and it can be reduced with replication, not zeroed. ## A Cheaper Stage The forty-eight unused grants were measured at runtime, but they were not **born** at runtime. Each item was born when a policy was written for an app: the `data/` prefix was written because it was not known which data secret that app needed. The same item could have been closed at the deploy stage — if each app had **declared explicitly** in its deployment definition the secret paths it needs, the policy would be generated from that declaration and no item beyond the thirty-one would ever have been opened. The measure of the work differs too: writing nine declarations for nine apps versus reviewing a seventy-nine-item matrix by hand and deciding which are unnecessary differs by roughly a factor of nine — and the second is redone every time a new secret is added, while the first is written once. The only item that needs to be measured at runtime is whether the declaration itself is correct: if the read log shows more than what was declared, the declaration is incomplete; if it shows less, the declaration is excessive. This comparison is an audit, not a defense. ## Summary - Twenty-one rule lines open 79 `(app, secret)` items to nine apps; the ninety-day read log says 31. Forty-eight items, 61% of the access opened, are never used. - The source of the unused grants is the path prefix rule: `data/` is one line and gives five secrets; six apps can reach each of the five data secrets, and the most-read one is read by three apps. - No app reads three of the secrets; these are the cheapest items that can be closed without cutting anyone. - Least privilege takes the secrets exposed by a compromised app from 8.8 on average to 3.4, and from 12 in the worst case to 6. Its cost is that the policy grows from 21 lines to 31; written from thirty days of observation it cuts four legitimate reads, and the cut shows only when that job first runs. - The price of centralization is a single point: a one-minute outage stops one app, an eight-hour outage five; none of the nine apps can start without the store, and 41 start attempts a day pass through this point. - The forty-eight items were measured at runtime but born at the deploy stage; had the secret paths been declared in the deployment definition they would never have been opened — nine declarations against a 79-item matrix review is a difference of roughly a factor of nine. ## Next Step This lesson narrowed access but never touched the secrets themselves. What sits in each of the thirty-one items is still **a value with no lifetime**: written on installation day, the same from that day to this. When the access matrix is narrowed, the secrets exposed by one app fall to six, but the question of how long those six secrets would work once leaked was never asked. The next lesson measures that time: when a static secret leaks, how many days the exposure window lasts, how many times a short-lived credential produced on request narrows the same window, and in return for that narrowing how many extra requests a day are added to the store and how many new dependencies to the system.