Patch Management

The previous lesson narrowed access and shortened secrets’ lifetime. But the software using that credential — the kernel, the system library, the language runtime, the reverse proxy — ages right where it stands. The moment a vulnerability is announced, every machine that has that component installed turns into an open-surface item, and the item stays open until the patch is applied. What the default leaves here is not a setting, it is a gap: nobody knows which version is running on which machine, and the patch gets applied when something breaks. The measure is the exposure window — the time between announcement and application; how many items stay open for how long is open item-days.

The Exposure Window Has Four Parts

The exposure window is not a single number, it is the sum of four independent parts; on a buggy patch the retry time is added on top of these four, which is why the table’s total column is larger than the sum of the four. Notice is the time between the announcement and seeing that the component is installed on the fleet, and it depends on the inventory scan interval. Test is trying the patch in a non-production environment. Window wait is the ready patch waiting for the next maintenance window. Apply is the install and restart.

The setup below is a model. The fictional regional measurement network’s machines and components are written as a data structure; no real fleet is scanned.

• HB19 — The fleet is fictional: twelve machines, eight components, and sixty (machine, component) items.
• HB20 — Fourteen announcements happen over the ninety-day window; their days and components are written in the model.
• HB21 — The inventory scan interval sets notice time: an announcement is seen at the next scan.
• HB22 — The maintenance window is fixed-interval and fixed-hour; window: 0 is a wait-free regime.
• HB23 — Whether a patch is buggy is decided by our own generator, the seed is visible. If testing catches the bug, the fixed patch arrives three days later; if it misses it, the patch goes to production, is seen half a day later, and is rolled back.
• HB24 — Three items cannot be patched in any regime; their reasons are written in the model.

// patch/model.mjs — the patch flow is a MODEL: twelve fictional machines, eight components, fourteen announcements.
// No real fleet is scanned, no real patch is downloaded; every number derives from this file.
const GROUP = { collector: 3, validator: 2, billing: 1, "work-order": 1, data: 2, gateway: 2, queue: 1 };
export const MACHINES = Object.entries(GROUP).flatMap(([a, n]) =>
  Array.from({ length: n }, (_, i) => `${a}-${i + 1}`));

export const COMPONENT = {                          // test and apply durations in days
  kernel:                { test: 2.0, apply: 0.20, group: "all" },
  "system-library":      { test: 1.0, apply: 0.15, group: "all" },
  "monitoring-agent":    { test: 0.25, apply: 0.05, group: "all" },
  scheduler:             { test: 0.25, apply: 0.05, group: "all" },
  "language-runtime":    { test: 1.5, apply: 0.10, group: ["collector", "validator", "billing", "work-order"] },
  "database-server":     { test: 2.0, apply: 0.25, group: ["data"] },
  "reverse-proxy":       { test: 0.5, apply: 0.05, group: ["gateway"] },
  "message-queue":       { test: 1.0, apply: 0.10, group: ["queue"] },
};
export const machinesFor = (b) => (COMPONENT[b].group === "all" ? MACHINES
  : MACHINES.filter((m) => COMPONENT[b].group.includes(m.slice(0, m.lastIndexOf("-")))));
export const ITEM = Object.keys(COMPONENT).reduce((a, b) => a + machinesFor(b).length, 0);

export const UNPATCHABLE = [                      // item = (machine, component); reason given by name
  ["billing-1", "language-runtime", "application compatibility"],
  ["data-1", "database-server", "service that must run continuously"],
  ["queue-1", "message-queue", "single-copy queue that cannot be drained"],
];
export const patchable = (b) =>
  machinesFor(b).filter((m) => !UNPATCHABLE.some((k) => k[0] === m && k[1] === b));

export const ANNOUNCEMENT = [                           // fourteen vulnerability announcements over a 90-day window
  [3.2, "system-library"], [8.6, "language-runtime"], [12.1, "kernel"], [19.4, "monitoring-agent"],
  [23.0, "reverse-proxy"], [27.8, "system-library"], [34.5, "database-server"], [39.9, "scheduler"],
  [45.3, "kernel"], [51.7, "message-queue"], [58.2, "language-runtime"], [63.5, "system-library"],
  [71.1, "monitoring-agent"], [78.4, "kernel"],
];
export const HORIZON = 90, FIX_DELAY = 3.0, NOTICE_DELAY = 0.5, SEED = 4127;

export function generator(seed) {                   // we write the generator ourselves, the seed is visible
  let x = seed >>> 0;
  return () => { x = (Math.imul(x, 1103515245) + 12345) >>> 0; return x / 4294967296; };
}
const r = generator(SEED);                          // independent of the draw order: all regimes see the same patches
export const BUGGY = ANNOUNCEMENT.map(() => r() < 0.20);
export const CAUGHT = ANNOUNCEMENT.map(() => r() < 0.75);

export const REGIME = {                            // scan = inventory interval, window = maintenance interval
  default: { scan: 30, test: false, window: 30, offset: 5.0 },
  weekly: { scan: 1, test: true, window: 7, offset: 2.0 },
  nightly: { scan: 1, test: true, window: 1, offset: 2 / 24 },
  continuous: { scan: 0.25, test: true, window: 0, offset: 0 },
  automatic: { scan: 0.25, test: false, window: 0, offset: 0 },
};
const firstWindow = (reg, ready) => (reg.window === 0 ? ready
  : Math.ceil((ready - reg.offset) / reg.window) * reg.window + reg.offset);

// One announcement's flow: notice, test, window wait, apply.
export function flow(i, name, buggy = BUGGY[i]) {
  const reg = REGIME[name], [day, b] = ANNOUNCEMENT[i], comp = COMPONENT[b], events = [];
  const notice = Math.ceil(day / reg.scan) * reg.scan;
  const test = reg.test ? comp.test : 0;
  let ready = notice + test, rollback = 0;
  if (buggy && reg.test && CAUGHT[i]) ready += FIX_DELAY;      // test caught it: the patch is delayed
  else if (buggy) {                                            // untested or missed: it went to production
    const first = firstWindow(reg, ready);
    events.push(first, first + NOTICE_DELAY);                  // apply, then rollback
    rollback = 1;
    ready = first + NOTICE_DELAY + FIX_DELAY;
  }
  const applyAt = firstWindow(reg, ready);
  events.push(applyAt);
  return { notice: notice - day, test, wait: applyAt - ready, applyDur: comp.apply,
    close: applyAt + comp.apply, window: applyAt + comp.apply - day, rollback, events };
}

export function scan(name, buggyOverride = null) {
  const outage = new Set();
  let avg = 0, rollback = 0, itemDays = 0, open = 0;
  for (let i = 0; i < ANNOUNCEMENT.length; i += 1) {
    const f = flow(i, name, buggyOverride === null ? BUGGY[i] : buggyOverride);
    const p = patchable(ANNOUNCEMENT[i][1]);
    avg += f.window / ANNOUNCEMENT.length;
    rollback += f.rollback;
    itemDays += f.window * p.length;
    if (f.close > HORIZON) open += p.length;
    for (const t of f.events) for (const m of p) outage.add(`${m}@${t.toFixed(4)}`);
  }
  return { avg, rollback, itemDays, outage: outage.size,
    open: open + UNPATCHABLE.length };
}

Five Regimes Scanned

// patch/window.mjs — the exposure window's four parts and their costs are scanned across five regimes
import { ANNOUNCEMENT, REGIME, MACHINES, COMPONENT, ITEM, BUGGY, SEED, flow, scan } from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
console.log(`${MACHINES.length} machines, ${Object.keys(COMPONENT).length} components, ${ITEM} items; ` +
  `90 days, ${ANNOUNCEMENT.length} announcements, ${BUGGY.filter(Boolean).length} buggy patches (seed ${SEED}).`);
console.log("\nregime     |   scan |   test |  window |  avg open |  open item |  outage | rollback");
console.log("-----------|--------|--------|---------|-----------|------------|---------|----------");
for (const [name, d] of Object.entries(REGIME)) {
  const t = scan(name);
  console.log(`${name.padEnd(10)} | ${g(d.scan, 6)} | ${(d.test ? "yes" : "no").padStart(6)}` +
    ` | ${(d.window ? d.window + " days" : "no").padStart(7)} | ${g(t.avg, 9)}` +
    ` | ${String(t.open).padStart(10)} | ${String(t.outage).padStart(7)} | ${String(t.rollback).padStart(8)}`);
}

console.log("\nThe exposure window's four parts (days), averaged over fourteen announcements:");
console.log("regime     |    notice |   test |     window wait |    apply |  total");
console.log("-----------|-----------|--------|-----------------|----------|-------");
for (const name of Object.keys(REGIME)) {
  const p = { notice: 0, test: 0, wait: 0, applyDur: 0, window: 0 };
  for (let i = 0; i < ANNOUNCEMENT.length; i += 1)
    for (const k of Object.keys(p)) p[k] += flow(i, name)[k] / ANNOUNCEMENT.length;
  console.log(`${name.padEnd(10)} | ${g(p.notice, 9)} | ${g(p.test, 6)} | ${g(p.wait, 15)}` +
    ` | ${g(p.applyDur, 8)} | ${g(p.window, 6)}`);
}

12 machines, 8 components, 60 items; 90 days, 14 announcements, 4 buggy patches (seed 4127).

regime     |   scan |   test |  window |  avg open |  open item |  outage | rollback
-----------|--------|--------|---------|-----------|------------|---------|----------
default    |  30.00 |     no | 30 days |     28.94 |         45 |      54 |        4
weekly     |   1.00 |    yes |  7 days |      6.79 |          3 |     126 |        1
nightly    |   1.00 |    yes |  1 days |      3.09 |          3 |     147 |        1
continuous |   0.25 |    yes |      no |      2.28 |          3 |     147 |        1
automatic  |   0.25 |     no |      no |      1.22 |          3 |     205 |        4

The exposure window's four parts (days), averaged over fourteen announcements:
regime     |    notice |   test |     window wait |    apply |  total
-----------|-----------|--------|-----------------|----------|-------
default    |     15.24 |   0.00 |           11.14 |     0.13 |  28.94
weekly     |      0.52 |   1.16 |            3.89 |     0.13 |   6.79
nightly    |      0.52 |   1.16 |            0.33 |     0.13 |   3.09
continuous |      0.09 |   1.16 |            0.00 |     0.13 |   2.28
automatic  |      0.09 |   0.00 |            0.00 |     0.13 |   1.22

The second table shows exactly where the default loses: 15.24 of the 28.94 days is notice alone. This time has nothing to do with the patch itself or fear of outages; its only cause is that there is no record of which version is running where. Once the inventory scan moves to nightly, this part drops to 0.52 days — 14.72 days in a single step, and without restarting a single machine.

Test is the one item that works in the opposite direction: it rises from zero to 1.16 days, meaning it lengthens the window. Its payoff is in the rollback column: four of the fourteen patches are buggy, and in the three regimes that test, three of these were caught before they reached production, bringing rollback to 1 instead of 4.

Outage, on the other hand, works opposite to the window interval. The default regime is the quietest with 54 outages, because patches that accumulate over a monthly window get bundled into a single restart — and that same bundling is what pushes the window out to 28.94 days. In the nightly regime, outage rises from 126 to 147, and wait drops from 3.89 days to 0.33. Outage count and exposure window are the two ends of the same lever. The day-ninety count is the outcome: 45 items open in the default regime, 3 in the other four.

The Day Automation Buys and the Risk It Produces

// patch/automation.mjs — the day automation buys, the risk it produces, and the new surface it introduces
import { ANNOUNCEMENT, REGIME, MACHINES, UNPATCHABLE, ITEM, HORIZON, BUGGY, scan } from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const base = scan("weekly");
console.log("regime     |   avg open |   gain vs weekly |    applied |   rollback |   rollback added |    extra");
console.log("           |     (days) |           (days) |   untested |    in prod |           (days) |   outage");
console.log("-----------|------------|------------------|------------|------------|------------------|---------");
for (const name of Object.keys(REGIME)) {
  const t = scan(name), s = scan(name, false);          // s: if no patch is buggy
  console.log(`${name.padEnd(10)} | ${g(t.avg, 10)} | ${g(base.avg - t.avg, 16)}` +
    ` | ${String(REGIME[name].test ? 0 : ANNOUNCEMENT.length).padStart(10)} | ${String(t.rollback).padStart(10)}` +
    ` | ${g(t.avg - s.avg, 16)} | ${String(t.outage - s.outage).padStart(8)}`);
}

console.log("\nUnpatchable items (announcement arrives, application never happens in any regime):");
let unpatchableDays = 0;
for (const [m, b, gerekce] of UNPATCHABLE) {
  const first = ANNOUNCEMENT.find(([, x]) => x === b)[0];
  unpatchableDays += HORIZON - first;
  console.log(`${(m + " / " + b).padEnd(34)} | ${gerekce.padEnd(42)} | announced day ${g(first, 4, 1)}` +
    ` | ${g(HORIZON - first, 5, 1)} days open`);
}
const itemDays = scan("nightly").itemDays;
console.log(`patchable in the nightly regime ${ITEM - UNPATCHABLE.length} items : ${g(itemDays, 7)} item-days`);
console.log(`unpatchable ${UNPATCHABLE.length} items${" ".repeat(21)} : ${g(unpatchableDays, 7)} item-days ` +
  `(${g(100 * unpatchableDays / (itemDays + unpatchableDays), 5)}% of total)`);

// the new surface closing introduces: one patch agent per machine, one package source, one inventory record
const added = MACHINES.length + 3;
console.log(`\nnew surface: ${MACHINES.length} patch agents + package source + inventory record + maintenance window`);
console.log(`${ITEM} items -> ${ITEM + added} items (+${added}); exposure window dropped from ` +
  `${g(scan("default").avg, 5)} days to ${g(scan("automatic").avg, 5)} days, ` +
  `rollback ${scan("automatic").rollback} instead of ${base.rollback}, ` +
  `outage ${scan("automatic").outage} instead of ${base.outage}.`);
console.log(`buggy patch ${BUGGY.filter(Boolean).length}/${ANNOUNCEMENT.length}; ` +
  `testing caught ${BUGGY.filter(Boolean).length - base.rollback} of these before production.`);

regime     |   avg open |   gain vs weekly |    applied |   rollback |   rollback added |    extra
           |     (days) |           (days) |   untested |    in prod |           (days) |   outage
-----------|------------|------------------|------------|------------|------------------|---------
default    |      28.94 |           -22.14 |         14 |          4 |             8.57 |       18
weekly     |       6.79 |             0.00 |          0 |          1 |             1.50 |       17
nightly    |       3.09 |             3.70 |          0 |          1 |             0.93 |       24
continuous |       2.28 |             4.52 |          0 |          1 |             0.89 |       24
automatic  |       1.22 |             5.57 |         14 |          4 |             1.00 |       82

Unpatchable items (announcement arrives, application never happens in any regime):
billing-1 / language-runtime       | application compatibility                  | announced day  8.6 |  81.4 days open
data-1 / database-server           | service that must run continuously         | announced day 34.5 |  55.5 days open
queue-1 / message-queue            | single-copy queue that cannot be drained   | announced day 51.7 |  38.3 days open
patchable in the nightly regime 57 items :  407.50 item-days
unpatchable 3 items                      :  175.20 item-days (30.07% of total)

new surface: 12 patch agents + package source + inventory record + maintenance window
60 items -> 75 items (+15); exposure window dropped from 28.94 days to  1.22 days, rollback 4 instead of 1, outage 205 instead of 126.
buggy patch 4/14; testing caught 3 of these before production.

The automatic regime buys 5.57 days over weekly and has the shortest window. The same row has three more numbers: all fourteen of the fourteen patches were applied untested, four were rolled back in production, and rollback produced 82 extra outages. The continuous regime buys 4.52 days — 81 percent of the gain — but zero patches are applied untested and extra outage is 24. The gap between them is 1.06 days. The argument is not automation itself, it is whether testing gets skipped. The rollback-added-days column shows that a buggy patch lengthens its own window: 1.00 day in the automatic regime, 8.57 days in the default — because the fixed version waits for the next monthly window.

The second table names the item that cannot be closed. The language runtime on the billing machine cannot be upgraded because the end-of-day batch job depends on the old version; the database server on the primary data machine cannot be restarted because there is no room for a write outage in the nightly batch window; the single-copy queue cannot be stopped because it cannot be drained. These three items carry 175.20 item-days — 5 percent of the sixty items, 30.07 percent of the open item-days.

Closing this in itself brings surface. Patch automation puts a process running code with elevated privilege on each of the twelve machines; the package source is the single point of trust for an incoming binary; the inventory record is a complete vulnerability map of the fleet for anyone who can read it; the maintenance window is a known interval of disruption repeated at the same hour every night.

Summary

• The exposure window has four parts: notice, test, window wait, apply. In the default regime, 15.24 of the 28.94 days is notice alone, and the inventory scan drops it to 0.52 days.
• Test lengthens the window by 1.16 days, and in return drops rollback in production from 4 to 1.
• Outage count and exposure window work in opposite directions: 54 outages and 28.94 days in a monthly window, 147 outages and 3.09 days in a nightly one. Open items at day ninety drop from 45 to 3.
• Automatic patching buys 5.57 days but applies all fourteen patches untested, raises rollback to 4, and produces 82 extra outages; the continuous regime gives 81 percent of the gain without dropping testing.
• The three unpatchable items — application compatibility, a service that must run continuously, a single-copy queue — carry 30.07 percent of the open item-days despite being 5 percent of the sixty items.
• The new surface closing introduces is fifteen items: the patch agent, the package source, the inventory record, and the maintenance window; the total rises from 60 to 75.

Next Step

This lesson never tested one assumption: that a prior state exists to return to when a patch is rolled back. The fix for the unpatchable items leaned on the same assumption. Recovery objectives were measured in the Resilience and Reliability course and are taken here as input; the next lesson does not measure objectives, it measures the backup itself: how many of production’s items are actually backed up, which item is left out by name, and where frequency meets the data-loss window and retention meets the point you can go back to. The backup itself is also a surface: how many copies, where, and who can read them.