Key Rotation

The previous lesson showed that the exposure window cannot be shortened in fifteen items: signing keys, external gateway keys, and the cache server do not generate credentials on request. For those items one tool is left — changing the secret at set intervals. Rotation bounds the exposure window by its period; the window of a key rotated every ninety days cannot be longer than ninety days.

But rotation does not happen in an instant. At the moment the new value is written to the store, no app knows about it; each app moves over in its own way, at its own speed. Until that move is complete, both values must stay valid. This lesson measures that mandatory window: how many minutes it is, what sets it, what is cut when it is kept short, and what happens when the order is reversed.

• SM13. The model of the previous two lessons continues: nine apps, seventeen read secrets. The environment is fictional, what is measured is a model written in node, and it carries no real secret value.
• SM14. The switch time is derived from the read pattern. An app that reads on every request switches immediately; a cached app switches after at most the cache life; an app that reads only at startup switches only when it restarts. For the last one the model uses the mean instance lifetime: the number of instances divided by the number of restarts per day.
• SM15. The cut requests are counted as the worst case: all instances of an app switch at the end of the switch time. In reality the switches are spread across instances and the cut requests stay below this number; the ranking does not change.
• SM16. The seventeen secrets are rotated one by one and the results are summed. The store is assumed to revoke the old value instantly; the propagation time of the revocation itself is not modeled.
• SM17. There is no randomness; every number is derived from the model tables and the output is the same from run to run.

The Slowest Reader Sets the Window

// switchover.mjs — the minimum time two secrets must stay valid together during rotation. A MODEL.
// The environment is FICTIONAL; THERE ARE NO REAL SECRET VALUES, only secret paths and switch times are modeled.
export const READS = {
  "portal-frontend": ["signing/session-signing"],
  "portal-api": ["data/portal-primary", "data/portal-read", "signing/session-signing",
    "signing/internal-call-signing", "infra/cache", "infra/log-write"],
  "subscriber-signup": ["data/portal-primary", "external/email-gateway", "external/sms-gateway"],
  "billing-service": ["data/billing-primary", "data/portal-read", "data/archive",
    "external/payment-gateway", "signing/document-signing"],
  "payment-bridge": ["external/payment-gateway"],
  "field-api": ["data/field-primary", "queue/work-order-queue", "external/map-service",
    "signing/internal-call-signing"],
  "work-order": ["data/field-primary", "queue/work-order-queue", "queue/notification-queue",
    "external/map-service"],
  "notification": ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"],
  "report-generator": ["data/portal-read", "data/archive", "infra/object-store",
    "external/email-gateway"],
};
// format/life: how the secret is read. instances: running instances. restarts: restarts per day.
// requests: daily requests (fictional unit).
export const APPS = {
  "portal-frontend": { format: "at-startup", life: 0, instances: 4, restarts: 6, requests: 84000 },
  "portal-api": { format: "cached", life: 60, instances: 8, restarts: 12, requests: 96000 },
  "subscriber-signup": { format: "at-startup", life: 0, instances: 2, restarts: 3, requests: 1400 },
  "billing-service": { format: "cached", life: 240, instances: 2, restarts: 2, requests: 6200 },
  "payment-bridge": { format: "per-request", life: -1, instances: 3, restarts: 2, requests: 3400 },
  "field-api": { format: "cached", life: 60, instances: 5, restarts: 8, requests: 42000 },
  "work-order": { format: "at-startup", life: 0, instances: 3, restarts: 4, requests: 9600 },
  "notification": { format: "cached", life: 15, instances: 2, restarts: 3, requests: 12000 },
  "report-generator": { format: "at-startup", life: 0, instances: 1, restarts: 1, requests: 320 },
};
export const NAMES = Object.keys(APPS);
export const SECRETS = [...new Set(Object.values(READS).flat())].sort();
export const readersOf = (s) => NAMES.filter((a) => READS[a].includes(s));
// switchTime: the worst-case time (min) for an app to move to the new value once it is written.
// a per-request reader switches immediately; a cached one after at most its cache life; an
// at-startup reader only on restart, that is, after the mean instance lifetime.
export const switchTime = (a) => APPS[a].format === "per-request" ? 0
  : APPS[a].format === "cached" ? APPS[a].life : Math.round(1440 * APPS[a].instances / APPS[a].restarts);
export const overlap = (s) => Math.max(...readersOf(s).map(switchTime));
export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0])).join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  const row = gen([23, 9, 10, 8, 24]), row0 = gen([19, 13, 11, 10, 13]);
  console.log(`switch time — worst-case time to move to the new value (min)`);
  console.log(row0(["app", "format", "instances", "restarts", "switch(min)"]));
  for (const a of NAMES) console.log(row0([a, APPS[a].format, APPS[a].instances, APPS[a].restarts, switchTime(a)]));

  console.log(`\n${row(["secret", "readers", "overlap", "= hours", "determining app"])}`);
  const sorted = [...SECRETS].sort((x, y) => overlap(y) - overlap(x) || x.localeCompare(y));
  for (const s of sorted)
    console.log(row([s.split("/")[1], readersOf(s).length, overlap(s), (overlap(s) / 60).toFixed(1),
      readersOf(s).filter((a) => switchTime(a) === overlap(s)).join(",")]));
  const O = SECRETS.map(overlap);
  console.log(`\n${SECRETS.length} secrets; longest minimum overlap ${Math.max(...O)} min, shortest ` +
    `${Math.min(...O)} min, mean ${(O.reduce((t, x) => t + x, 0) / O.length).toFixed(0)} min`);
  const slow = NAMES.filter((a) => APPS[a].format === "at-startup");
  console.log(`overlap needed if rotated in a single shared window: ${Math.max(...O)} min`);
  console.log(`what sets the overlap: ${slow.length} at-startup apps, alone in ` +
    `${SECRETS.filter((s) => readersOf(s).some((a) => APPS[a].format === "at-startup" &&
      switchTime(a) === overlap(s))).length} / ${SECRETS.length} secrets`);
}

switch time — worst-case time to move to the new value (min)
app                       format  instances  restarts  switch(min)
portal-frontend       at-startup          4         6          960
portal-api                cached          8        12           60
subscriber-signup     at-startup          2         3          960
billing-service           cached          2         2          240
payment-bridge       per-request          3         2            0
field-api                 cached          5         8           60
work-order            at-startup          3         4         1080
notification              cached          2         3           15
report-generator      at-startup          1         1         1440

secret                   readers   overlap = hours         determining app
archive                        2      1440    24.0        report-generator
portal-read                    3      1440    24.0        report-generator
email-gateway                  3      1440    24.0        report-generator
object-store                   1      1440    24.0        report-generator
field-primary                  2      1080    18.0              work-order
map-service                    2      1080    18.0              work-order
notification-queue             2      1080    18.0              work-order
work-order-queue               2      1080    18.0              work-order
portal-primary                 2       960    16.0       subscriber-signup
sms-gateway                    2       960    16.0       subscriber-signup
session-signing                2       960    16.0         portal-frontend
billing-primary                1       240     4.0         billing-service
payment-gateway                2       240     4.0         billing-service
document-signing               1       240     4.0         billing-service
cache                          1        60     1.0              portal-api
log-write                      1        60     1.0              portal-api
internal-call-signing          2        60     1.0    portal-api,field-api

17 secrets; longest minimum overlap 1440 min, shortest 60 min, mean 815 min
overlap needed if rotated in a single shared window: 1440 min
what sets the overlap: 4 at-startup apps, alone in 11 / 17 secrets

The table above shows where the switch time comes from. The payment bridge reads its secret on every request, so it switches in zero minutes; the notification service, with its fifteen-minute cache, switches in fifteen minutes at the most. In the four apps that read only at startup, however, the switch time is not a setting but a statistic: the report generator runs with a single instance and restarts once a day, so it sees the new value after a mean of one thousand four hundred forty minutes. This number is written in no setting of secrets management; it comes from the deployment layout.

The lower table gives the real measure. The minimum overlap of a secret is the switch time of the slowest of its readers; put another way, the overlap is not a mean but a maximum. This is why the object store key, which has a single reader, and the email gateway key, which has three, sit on the same line: in both the slowest reader is the report generator, and both need twenty-four hours of overlap. The number of readers does not set the overlap; the worst of the readers does.

The three secrets that need the shortest overlap are those read only by cached apps: the internal call signing key is read by two apps and can turn over in sixty minutes. Between one thousand four hundred forty minutes and sixty minutes there is a factor of twenty-four, and the whole of that difference comes from the read pattern — there is no difference in the sensitivity or the policy of the secrets.

The last line gathers the finding in one sentence: four apps that read at startup set the overlap alone in eleven of the seventeen secrets. These are the smallest apps in the system, the ones that receive the fewest requests — the report generator sees three hundred twenty requests a day. The whole rotation calendar depends on the restart habits of these four apps.

The Bill for Shortening the Window

A twenty-four-hour overlap works against the purpose of rotation: a leaked old value keeps working for those twenty-four hours too. Shortening the window has a measurable counterpart.

// order.mjs — what is cut when the overlap window is kept short and when the order is reversed
import { NAMES, READS, SECRETS, APPS, switchTime, readersOf, overlap, gen } from "./switchover.mjs";

// With an overlap of T minutes, an app stays on the old value for switchTime(a) - T minutes and
// drops the requests that depend on that secret for that long. The seventeen secrets are rotated
// one by one and the results are summed.
const cutRequests = (T, g = switchTime) => SECRETS.reduce((t, s) => t + readersOf(s)
  .reduce((x, a) => x + APPS[a].requests * Math.max(0, g(a) - T) / 1440, 0), 0);
const missedClients = (T, g = switchTime) => SECRETS.reduce((t, s) => t + readersOf(s)
  .reduce((x, a) => x + (g(a) > T ? APPS[a].instances : 0), 0), 0);

const row = gen([14, 18, 16, 20]);
console.log(row(["overlap(min)", "missed clients", "cut requests", "old value valid"]));
for (const T of [0, 15, 60, 240, 960, 1080, 1440])
  console.log(row([T, missedClients(T), Math.round(cutRequests(T)), `${T} min`]));

const OVERLAPS = SECRETS.map(overlap), overlapSum = OVERLAPS.reduce((t, x) => t + x, 0);
console.log(`\nif each secret uses its own minimum overlap: missed clients 0, cut requests 0,`);
console.log(`the old value stays valid ${(overlapSum / SECRETS.length).toFixed(0)} min ` +
  `(${(overlapSum / SECRETS.length / 60).toFixed(1)} hours) longer on average`);

// reverse order: the old value is revoked first, the new one is written WRITE minutes later, then it propagates.
const WRITE = 5;
console.log(`\nreverse order (revoke first, new value after ${WRITE} min): missed clients ` +
  `${missedClients(-1)}, cut requests ${Math.round(cutRequests(0) +
    SECRETS.reduce((t, s) => t + readersOf(s).reduce((x, a) => x + APPS[a].requests * WRITE / 1440, 0), 0))}`);
console.log(`in the correct order the only app that is never cut is the per-request reader payment-bridge; ` +
  `in reverse order it too loses ${Math.round(READS["payment-bridge"].length * APPS["payment-bridge"].requests *
    WRITE / 1440)} requests`);

// refresh on auth failure: an at-startup app refreshes the secret when it gets an authentication error.
const REFRESH = 5;
const switchWithRefresh = (a) => APPS[a].format === "at-startup" ? REFRESH : switchTime(a);
const OVERLAPS2 = SECRETS.map((s) => Math.max(...readersOf(s).map(switchWithRefresh)));
console.log(`\nif refresh-on-auth-failure is added (${NAMES.filter((a) =>
  APPS[a].format === "at-startup").length} at-startup apps, switch ${REFRESH} min):`);
console.log(`  longest minimum overlap ${Math.max(...OVERLAPS)} -> ${Math.max(...OVERLAPS2)} min, ` +
  `mean ${(overlapSum / SECRETS.length).toFixed(0)} -> ` +
  `${(OVERLAPS2.reduce((t, x) => t + x, 0) / SECRETS.length).toFixed(0)} min`);
console.log(`  cut requests at 60 min overlap ${Math.round(cutRequests(60))} -> ` +
  `${Math.round(cutRequests(60, switchWithRefresh))}, missed clients ${missedClients(60)} -> ` +
  `${missedClients(60, switchWithRefresh)}`);

overlap(min)      missed clients    cut requests     old value valid
0                            110          125422               0 min
15                           104          115642              15 min
60                            36           87427              60 min
240                           26           67567             240 min
960                           16            3627             960 min
1080                           4             320            1080 min
1440                           0               0            1440 min

if each secret uses its own minimum overlap: missed clients 0, cut requests 0,
the old value stays valid 815 min (13.6 hours) longer on average

reverse order (revoke first, new value after 5 min): missed clients 113, cut requests 128693
in the correct order the only app that is never cut is the per-request reader payment-bridge; in reverse order it too loses 12 requests

if refresh-on-auth-failure is added (4 at-startup apps, switch 5 min):
  longest minimum overlap 1440 -> 240 min, mean 815 -> 102 min
  cut requests at 60 min overlap 87427 -> 3875, missed clients 36 -> 10

The table is a tuning curve and should be read from both ends. The top row is the case where the overlap is zero: as soon as the new value is written, the old one is revoked. One hundred ten clients — that is, one hundred ten items when counted over the secrets read by the thirty running instances of the nine apps — stay on the old value, and in the total of seventeen rotations one hundred twenty-five thousand four hundred twenty-two legitimate requests are cut. The bottom row is the opposite: at an overlap of one thousand four hundred forty minutes no client is missed, no request is cut, and the old value stays valid one full day longer.

The middle rows show what the distribution looks like. An overlap of sixty minutes brings the missed clients down from one hundred ten to thirty-six, but the cut requests only from one hundred twenty-five thousand to eighty-seven thousand. The mismatch between the ratios is meaningful: most of the missed clients are the eight instances of the portal API, and those instances switch in sixty minutes; most of the cut requests, however, come from a single app. The portal frontend sees eighty-four thousand requests a day and switches after nine hundred sixty minutes; this one app alone produces fifty-two thousand five hundred of the eighty-seven thousand requests cut at an overlap of sixty minutes.

The next two rows write the correct procedure. If each secret is rotated with its own minimum overlap, the cut requests are zero; the cost is that the old value stays valid a mean of eight hundred fifteen minutes longer. In other words, the overlap is not a security setting but a fixed duration added to the exposure window: the real worst-case window of a key rotated on a ninety-day period is not ninety days but ninety days plus thirteen and a half hours.

The reverse-order row is a separate class. When the old value is revoked first and the new one is written five minutes later, the missed clients rise to one hundred thirteen and the cut requests to one hundred twenty-eight thousand six hundred ninety-three. The difference is not large — three clients, three thousand requests — but what is decisive is who it falls on: the payment bridge, which is not affected at all in the correct order, loses twelve payment requests in the reverse order. The only way to cut an app that switches in zero minutes is for the value it will switch to not to have been written yet. That is the symptom of the reverse order, and in the dashboard only these twelve requests show.

Where the Window Shrinks

The last two lines answer the real question of the course. A twenty-four-hour overlap is not a secrets management problem; it is the problem of an app that reads at startup never reading the secret again. When the behavior of refreshing the secret on an authentication error is added to the apps, the minimum overlap falls from one thousand four hundred forty minutes to two hundred forty minutes and the mean from eight hundred fifteen minutes to one hundred two minutes; at an overlap of sixty minutes the cut requests fall from eighty-seven thousand four hundred twenty-seven to three thousand eight hundred seventy-five, that is, by a factor of twenty-three.

Where this change is made matters. It is made not in the secret store, in the rotation calendar, or in the runtime defense, but in the application code: the authentication error is caught, the secret is read again, and the request is tried once more. A behavior of a few dozen lines shortens the rotation window by a factor of twenty-four, and once written it holds for all secrets. The way to get the same result at runtime is to keep a separate overlap calendar for each secret, to track at every rotation which app has switched, and to restart by hand the one that has not.

What remains, two hundred forty minutes, truly belongs to the runtime: the four-hour cache life of the billing service. It is a setting and can be lowered; when it is lowered, the store requests measured in the previous lesson increase. Overlap, cache life, and store load are three views of a single number.

Summary

• The minimum overlap of a secret is the switch time of the slowest of its readers. In the seventeen secrets this time lies between 60 and 1,440 minutes, with a mean of 815 minutes; with a single shared window it is 1,440 minutes.
• What sets the overlap is not the sensitivity of the secret or the number of readers but the read format. Four apps that read at startup set the window alone in 11 of the 17 secrets.
• When the overlap is reduced to zero, 110 clients stay on the old value and 125,422 legitimate requests are cut; at 60 minutes, 36 clients and 87,427 requests; at 1,440 minutes, zero and zero. 52,500 of the cut requests come from a single app, the portal frontend.
• If each secret is rotated with its own minimum overlap the cut is zero; the cost is that the old value stays valid a mean of 815 minutes longer. The real window of a ninety-day period is ninety days plus thirteen and a half hours.
• The reverse order (revoke first, write after) raises the cut requests to 128,693. Its real symptom is not the volume but the payment bridge, which is not affected at all in the correct order, losing 12 requests.
• The change that shortens the overlap belongs to the code stage: refresh-on-auth-failure takes the minimum overlap from 1,440 to 240 minutes and reduces the requests that are cut at a 60-minute window by a factor of 23.

Next Step

What was measured over two lessons was two properties of the same object: the life of the secret and the cost of changing it. Both assumed that the secret exists. That assumption can be removed. The value an app carries to connect to a database is a way of proving who that app is; if the proof is given by the platform, there is no value left to carry. The next lesson counts this along a single axis: how many of the thirty-one items workload identity removes, how many items and which systems remain, how many apps come to carry no static secret at all — and how many lines the trust relation that takes the place of the secret is itself.