Certificate Automation

The previous lesson counted how many places the certificate sits in and took the renewal-touch-point count as equal to that number. Renewal itself stayed a black box: it happens some day, it finishes, the certificate is refreshed.

Inside that box is a state machine. A request is sent, control of the domain is validated, a signature is obtained, the new certificate is distributed to the copies. Each of the four steps takes time, and the second one can fail. Two settings sit outside this machine and decide the outcome: the certificate’s lifetime and the threshold that says how many hours before expiry renewal starts.

The measurement runs on the fictional regional measurement network’s twelve certificates, over a three-year period. There is no CA; the steps, their durations, and the certificate list are fictional.

• WS19. Step durations are fixed in hours: request 1, validation 2, signing 1, distribution 2. If validation fails, the backoff sequence is 1, 2, 4, 8, 16 hours, and automation gives up after the fifth attempt.
• WS20. When automation gives up, a human takes over and finishes the job in eight hours; every manually performed renewal also takes eight hours.
• WS21. The time to notice a forgotten manual renewal depends on visibility: an external-facing item is noticed in two hours, an internal item in ninety-six hours.
• WS22. Forgetting and validation error are produced by our own generator; the seed 4711 is visible and gives the same sequence run to run. The forget rate is taken as 0.25, the validation error rate as 0.30.
• WS23. The model does not measure time; every hour value is derived arithmetically from the constants above.
• WS24. The exposure window is the number of hours by which the renewal flow’s duration exceeds the threshold; the certificate stands expired for those hours.

// renewal/flow.mjs — model: the certificate renewal flow is a state machine. Duration unit is
// hours, the period is three years. There is no CA; the steps and durations are fictional, every number is derived from them.
const PERIOD = 26280;                      // 3 years, hours
const MANUAL = 8;                          // human takeover and finish (hours)
const BACKOFF = [1, 2, 4, 8, 16];          // wait after a validation failure (hours)
const STEP = { request: 1, validate: 2, sign: 1, distribute: 2 };

// State machine: request -> validate -> (fail -> wait -> validate)* -> sign -> distribute.
// "bad" says which attempt succeeds on; BACKOFF.length means it never succeeds.
function run(bad) {
  let t = STEP.request;
  for (let a = 0; a < BACKOFF.length; a += 1) {
    t += STEP.validate;
    if (a === bad) return { duration: t + STEP.sign + STEP.distribute, result: "done" };
    t += BACKOFF[a];
  }
  return { duration: t + MANUAL, result: "manual takeover" };
}

const DURATION = [0, 4, BACKOFF.length].map((k) => run(k).duration);   // 0 failures, 4 failures, gave up
const window = (threshold, duration) => Math.max(0, duration - threshold);
const renewalCount = (lifetime, threshold) => Math.floor(PERIOD / (lifetime - threshold));
const required = (lifetime) => Math.ceil(PERIOD / lifetime) - 1;

const E = [13, 14, 17, 10, 12, 12, 19, 15];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["lifetime(h)", "threshold(h)", "renewals/period", "unneeded", "window(0)",
  "window(4)", "window(gaveup)", "alert"]));
for (const lifetime of [2160, 8760])
  for (const threshold of [1080, 720, 336, 48, 12]) {
    const n = renewalCount(lifetime, threshold);
    console.log(write([lifetime, threshold, n, n - required(lifetime), ...DURATION.map((s) => window(threshold, s)),
      DURATION[2] > threshold ? "at first fail" : "at giveup"]));
  }

// --- Manual renewal vs. automatic renewal, same period --------------------------
// The fictional measurement network's certificates: [name, lifetime(h), visibility]. Visibility
// decides the time to notice a miss: an external item in hours, an internal item days later.
const NOTICE = { external: 2, internal: 96 };
const CERT = [
  ["edge-proxy-external", 2160, "external"], ["internal-proxy-staging", 2160, "internal"],
  ["internal-proxy-production", 2160, "internal"], ["measurement-app", 8760, "external"],
  ["collector-client", 8760, "internal"], ["billing-client", 8760, "internal"],
  ["mgmt-listener", 8760, "internal"], ["notification-gateway", 2160, "internal"],
  ["backup-target", 8760, "internal"], ["work-order-app", 2160, "external"],
  ["meter-network-gateway", 17520, "internal"], ["dashboard", 2160, "external"],
];
// Our own generator; the seed is visible and gives the same sequence run to run.
const rng = (seed) => { let s = seed >>> 0; return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; };
const SEED = 4711, FORGET_RATE = 0.25, ERROR_RATE = 0.30;

function manual() {
  const r = rng(SEED);
  let events = 0, missed = 0, hours = 0; const items = new Set();
  const breakdown = { external: [0, 0], internal: [0, 0] };  // [missed events, open hours]
  for (const [name, lifetime, vis] of CERT)
    for (let k = 0; k < Math.floor(PERIOD / lifetime); k += 1) {
      events += 1;
      if (r() < FORGET_RATE) {
        missed += 1; hours += NOTICE[vis] + MANUAL; items.add(name);
        breakdown[vis][0] += 1; breakdown[vis][1] += NOTICE[vis] + MANUAL;
      }
    }
  return { events, missed, hours, items: items.size, breakdown };
}

function automatic(threshold) {
  const r = rng(SEED);
  let events = 0, missed = 0, hours = 0; const items = new Set();
  for (const [name, lifetime] of CERT)
    for (let k = 0; k < renewalCount(lifetime, threshold); k += 1) {
      events += 1;
      let a = 0;
      while (a < BACKOFF.length && r() < ERROR_RATE) a += 1;      // which attempt succeeds
      const p = window(threshold, run(a).duration);
      if (a === BACKOFF.length) missed += 1;
      if (p > 0) { hours += p; items.add(name); }
    }
  return { events, missed, hours, items: items.size };
}

const Y = [27, 8, 22, 12, 16];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](Y[i])).join("");
console.log("\n" + write2(["method", "events", "missed/gaveup", "open hours", "affected items"]));
const E0 = manual();
for (const [name, r] of [["manual (no threshold)", E0], ["automatic (threshold 168)", automatic(168)],
  ["automatic (threshold 12)", automatic(12)]])
  console.log(write2([name, r.events, r.missed, r.hours, r.items]));
console.log(`\nbreakdown of manual misses: external ${E0.breakdown.external[0]} events / ${E0.breakdown.external[1]} hours` +
  `   internal ${E0.breakdown.internal[0]} events / ${E0.breakdown.internal[1]} hours`);

lifetime(h)    threshold(h)  renewals/period  unneeded   window(0)   window(4)     window(gaveup)          alert
2160                   1080               24        12           0           0                  0      at giveup
2160                    720               18         6           0           0                  0      at giveup
2160                    336               14         2           0           0                  0      at giveup
2160                     48               12         0           0           0                  2  at first fail
2160                     12               12         0           0          17                 38  at first fail
8760                   1080                3         1           0           0                  0      at giveup
8760                    720                3         1           0           0                  0      at giveup
8760                    336                3         1           0           0                  0      at giveup
8760                     48                3         1           0           0                  2  at first fail
8760                     12                3         1           0          17                 38  at first fail

method                       events         missed/gaveup  open hours  affected items
manual (no threshold)            88                    29        1700               9
automatic (threshold 168)        94                     0           0               0
automatic (threshold 12)         88                     0          56               8

breakdown of manual misses: external 14 events / 140 hours   internal 15 events / 1560 hours

The Two Ends of the Threshold

The two ends of the first table produce two separate kinds of failure, and neither looks like a failure.

The early end. For a certificate with a ninety-day lifetime, setting the threshold to forty-five days starts renewal while the certificate is only halfway through its life. Over the period, twenty-four renewals run; the required number is twelve. Twelve renewals are unneeded: none of them close an exposure window, and each one produces a request, a validation, a signature, and a distribution. When the threshold drops to twenty-one days, six unneeded renewals remain; at fourteen days, two remain.

The cost of an unneeded renewal is paid in two places. The first is the validation step itself; every renewal is an outside call and an outside dependency. The second is distribution: every renewal touches every place the certificate sits in, and in the six-copy arrangement the previous lesson counted, every touch carries a risk of partial state. Twenty-four renewals are twelve unneeded chances at partial state.

The late end. When the threshold drops to forty-eight hours, zero unneeded renewals remain. This column of the measurement is now clean; the other columns open up. When validation fails four times, the flow takes twenty-nine hours and fits inside the forty-eight-hour threshold — window zero. When automation fails all five of five attempts and hands off to a human, the total takes fifty hours and opens a window of two hours.

When the threshold drops to twelve hours, two columns open at once: seventeen hours at four failures, thirty-eight hours at giveup. Thirty-eight hours is how long the certificate stands expired, and every connecting client gets a warning or a rejection during that time.

How Lifetime Devalues the Threshold

The table’s second half repeats the same five thresholds at a one-year lifetime, and the unneeded column stops moving: one, at all five of the five thresholds. As lifetime stretches out, the threshold loses its effect on the unneeded-renewal count.

The reason is ratio: a forty-five-day threshold is half of a ninety-day lifetime, an eighth of a one-year lifetime. At a short lifetime the threshold splits the cycle in two; at a long lifetime it just clips the end. This does not mean a long lifetime is easier for automation — the window columns are identical across the two halves. The exposure window does not depend on lifetime, only on the threshold and the flow’s duration; a twelve-hour threshold on a one-year certificate leaves the same thirty-eight-hour exposure window as on a ninety-day one.

The real difference between the two halves is in the renewal count: at a ninety-day lifetime, twelve to twenty-four renewals run over the period; at a one-year lifetime, three. The rarer renewal is, the less the flow gets tested. A flow that runs three times in three years is only found broken on its fourth run, and that run is twelve hours before the certificate expires.

When the Alert Must Fire

The last column gives not a number but a rule, and the rule is decided by the threshold itself.

Automation’s worst-case total duration is forty-two hours: five validation attempts and thirty-one hours of backoff between them. A human taking over and finishing takes eight hours. The two together are fifty hours.

If the threshold is greater than fifty hours, automation can try all the way to the end; even when it gives up, enough time remains for a human, and the alert can fire at the moment of giveup. The first three threshold values are in this situation. If the threshold is smaller than fifty hours, the situation flips: waiting for automation to try to the end pushes the moment a human takes over past the certificate’s expiry. In this case the alert must fire at the first validation failure.

The rule comes down to this: the alert threshold is not a preference, it is comparing the sum of worst-case automatic duration + human duration against the threshold. At the forty-eight- and twelve-hour thresholds, automation staying silent is itself a failure — automation runs, tries, backs off, and no one knows. Silent automation is noticed later than manual renewal is.

Manual Renewal’s Bill for the Period

The second table counts the same twelve certificates as if renewed manually for three years.

There are eighty-eight renewal events over the period. Twenty-nine were forgotten — one in four. The misses spread across nine separate items: nine of the twelve certificates expired at least once. The total exposure window is seventeen hundred hours, more than seventy days over three years.

The breakdown row gives the real finding. External-facing items had fourteen misses, leaving a total of one hundred forty hours of exposure window. Internal items had fifteen misses and left fifteen hundred sixty hours. The event counts are nearly equal; the hours differ by a factor of eleven.

The difference comes from a single place: time to notice. When an external-facing certificate expires, field crews call within two hours. When an internal certificate expires, what happens — the collector cannot reach the verifier, the nightly batch job writes an alert, the notification gateway cannot send email — and none of that calls anyone. Ninety-six hours is how long it takes for the failure to reach a human’s eyes.

This shows where the “manual renewal is good enough” defense breaks. The defense is made by looking at the external-facing certificate, and it is right there: a two-hour window is acceptable. When the same discipline is applied to internal items, the window stretches forty-eight times longer, and because no one measures it, that stretch never shows up in any report.

What Automation Closes and Introduces

The last two rows apply automation to the same period.

Automation with a threshold of one hundred sixty-eight hours runs ninety-four renewals — six more than manual, because the threshold shortens the cycle. No renewal is given up on, the exposure window is zero, affected items zero. The seventeen-hundred-hour window dropped to zero.

Automation with a threshold of twelve hours runs the same eighty-eight events and leaves fifty-six hours of exposure window; eight items are affected. Again, no renewal is given up on — all five of five attempts failing at once is a rare event — but renewals left to two or three attempts do not fit inside the twelve-hour threshold. Automation itself is working correctly; the threshold is wrong.

Read together, the two rows make clear what automation closed: it closed forgetting, not error. In manual renewal, all seventeen hundred hours came from forgetting. In automation, all of the remaining fifty-six hours come from validation error, and that error can be absorbed with a threshold setting.

The new surface closing introduces can also be counted. Automation is a component: it runs with an identity, it has write authority to every place the certificate sits in, it depends on an outside validation step, and it carries a threshold in its own configuration. Four items. Manual renewal had none of these four items; in their place it had a habit that got forgotten on nine items. Automation did not change the item count, it changed the item’s type: it turned a task that had to be remembered into a component that has to be monitored.

Summary

• When the threshold was set early, twenty-four renewals ran over the period and twelve were unneeded; dropping the threshold to fourteen days brought unneeded renewals to two, and to forty-eight hours brought them to zero.
• When the threshold was set late, the window opened: at a twelve-hour threshold, four validation failures left a seventeen-hour window, and automation giving up left a thirty-eight-hour window.
• The alert’s moment is derived from the threshold: if the sum of the forty-two-hour worst-case automatic duration and the eight-hour human duration exceeds the threshold, the alert must fire at the first validation failure; if it is smaller, the moment of giveup is enough.
• Over the three-year period, twenty-nine of manual renewal’s eighty-eight events were forgotten, nine of the twelve certificates expired, and a total of seventeen hundred hours of exposure window remained.
• Half the misses were on external-facing items, but only one hundred forty of the open hours came from there; the internal items’ fifteen misses left fifteen hundred sixty hours — the difference is time to notice.
• Automation with a threshold of one hundred sixty-eight hours brought the exposure window to zero and introduced four new items in its place: an identity, a write authority, an outside dependency, and a threshold setting.

Next Step

This lesson and the two before it built and encrypted the request’s path: who arrived was relayed, encryption terminated somewhere, the certificate refreshes itself. The whole path now runs start to finish for every request — for every request.

The measurement network’s dashboard asks for the same regional map, the same meter list, and the same nightly summary file every time it opens. Most of these requests produce the same response, and each one travels from the edge proxy to the internal proxy, on to the app, and down to the data layer. The response can be held somewhere — but “somewhere” is two separate places: at the edge, close to the client; at the origin, close to the app. When both are put in place, how the hit rate splits, how many more seconds the old response stays at the edge after content changes, and how the freshness setting changes that window are separate numbers. The next lesson measures a two-layer cache with these three numbers and names the surface bad caching opens.