Rebuild-Based Update

The previous lesson split the findings of continuous auditing by owner, and in the early-stage column thirty-two findings read build: a mutable image tag, an unsigned image and an outdated runtime version. None of the three can be fixed in place; to close them, a new image has to be built and the thirty-one replicas redeployed.

This lesson measures that distance, and the unit of measurement is the day: after a fix is published, how many days does it take to start running on the last replica of the fleet, and how many replicas stay open in the meantime. The case for immutable infrastructure — machines drifting apart, the rollback round, local state that cannot be moved — was measured in the Infrastructure as Code course and is taken here as an input. The question is a different one: how many days does patch lag drop to, and how many restarts are demanded in return.

• IP23. The environment is the same: nine services, thirty-one replicas, one cluster. No real build or deployment is run; what is measured is a model built with node. All stage durations are given in days in the model and the numbers are derived from them; wall-clock time is not measured.
• IP24. Deployments per service are evenly spaced and the weekly release total is the eighteen used in the second lesson of the topic. In reality the interval scatters; the distribution lengthens the tail and does not change the ranking.
• IP25. An in-place patch is applied to the file system of the running pod and is not written to the image; the next deployment brings back the unpatched image. This is the deciding assumption of the model.
• IP26. In a rolling deployment at most one replica is down at a time and the readiness time is fixed per replica; the restart’s own failure probability is not modeled.

Three Paths and Patch Lag

// update.mjs — thirty-one replicas, three update paths and a thirty-day window. IT IS A MODEL.
// [service, replicas, weekly releases]; the release total is 18, a distribution of the value in the second lesson of the topic.
export const SERVICES = [
  ['portal-frontend', 6, 5], ['subscriber-service', 4, 4], ['billing', 3, 3], ['work-order', 3, 2],
  ['notification', 2, 1], ['report', 2, 1], ['measurement-collector', 6, 1], ['validator', 3, 0.5],
  ['field-sync', 2, 0.5],
];
export const WINDOW = 30, READY = 45; // days; readiness time per replica (s)
export const interval = (s) => 7 / s[2];
export const replicas = SERVICES.reduce((a, s) => a + s[1], 0);

// Stage durations (days): in-place patch = package repository 0.5 + approval 1 + maintenance window 3.5 +
// application 0.25; rebuild = propagation to the registry 0.5 + build and pipeline 0.25; scheduled
// adds the nightly trigger (1) to these.
const PATCH = 0.5 + 1 + 3.5 + 0.25, BUILD = 0.25, REGISTRY = 0.5, TRIGGER = 1;
const nextDeploy = (s, t) => Math.ceil((t + 1e-9) / interval(s)) * interval(s);

// An in-place patch is not written to the image: the next deployment brings back the unpatched image.
export function protectedSpans(path, s) {
  const spans = [];
  if (path === 'in-place patch')
    for (let w = PATCH; w < WINDOW; w += 7) spans.push([w, Math.min(nextDeploy(s, w), WINDOW)]);
  else if (path === 'code-change rebuild')
    spans.push([Math.min(nextDeploy(s, REGISTRY) + BUILD, WINDOW), WINDOW]);
  else spans.push([Math.min(REGISTRY + TRIGGER + BUILD, WINDOW), WINDOW]);
  return spans.filter(([b, e]) => e > b);
}

export const PATHS = ['in-place patch', 'code-change rebuild', 'scheduled rebuild'];
export const protectedDays = (path, s) => protectedSpans(path, s).reduce((a, [b, e]) => a + e - b, 0);
export const firstSpan = (path, s) => protectedSpans(path, s)[0] ?? [WINDOW, WINDOW];
export const openDays = (path) => SERVICES.reduce((a, s) => a + s[1] * (WINDOW - protectedDays(path, s)), 0);
export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join('');

// run1.mjs — same patch, three paths: the day the last replica closes and open replica-days
import { SERVICES, PATHS, WINDOW, replicas, interval, protectedDays, firstSpan, openDays, gen } from './update.mjs';

console.log(`${SERVICES.length} services, ${replicas} replicas, ${SERVICES.reduce((a, s) => a + s[2], 0)} releases per week; ` +
  `deployment interval between ${Math.min(...SERVICES.map(interval)).toFixed(2)} and ${Math.max(...SERVICES.map(interval))} days\n`);

const fmtMain = gen([32, 14, 14, 19, 11]);
console.log(fmtMain(['path', 'first replica', 'last replica', 'open replica-days', 'open share']));
for (const p of PATHS) {
  const g = SERVICES.map((s) => firstSpan(p, s)[0]);
  console.log(fmtMain([p, Math.min(...g).toFixed(2), Math.max(...g).toFixed(2), openDays(p).toFixed(1),
    `${(100 * openDays(p) / (replicas * WINDOW)).toFixed(1)}%`]));
}

const fmtSvc = gen([23, 10, 10, 14, 17]);
console.log('\n' + fmtSvc(['service', 'replicas', 'interval', 'patch holds', 'rebuild 1st day']));
for (const s of SERVICES) {
  const [b, e] = firstSpan('in-place patch', s);
  console.log(fmtSvc([s[0], s[1], interval(s).toFixed(2), (e - b).toFixed(2),
    firstSpan('code-change rebuild', s)[0].toFixed(2)]));
}
const holds = SERVICES.reduce((a, s) => { const [b, e] = firstSpan('in-place patch', s); return a + s[1] * (e - b); }, 0) / replicas;
console.log(`in-place patch holds for a replica-weighted mean of ${holds.toFixed(2)} days; over ${WINDOW} days ` +
  `the share of time spent protected is ${(100 * SERVICES.reduce((a, s) => a + s[1] * protectedDays('in-place patch', s), 0) / (replicas * WINDOW)).toFixed(1)}%`);

9 services, 31 replicas, 18 releases per week; deployment interval between 1.40 and 14 days

path                             first replica  last replica  open replica-days open share
in-place patch                            5.25          5.25              676.6      72.8%
code-change rebuild                       1.65         14.25              180.6      19.4%
scheduled rebuild                         1.75          1.75               54.3       5.8%

service                  replicas  interval   patch holds  rebuild 1st day
portal-frontend                 6      1.40          0.35             1.65
subscriber-service              4      1.75          1.75             2.00
billing                         3      2.33          1.75             2.58
work-order                      3      3.50          1.75             3.75
notification                    2      7.00          1.75             7.25
report                          2      7.00          1.75             7.25
measurement-collector           6      7.00          1.75             7.25
validator                       3     14.00          8.75            14.25
field-sync                      2     14.00          8.75            14.25
in-place patch holds for a replica-weighted mean of 2.61 days; over 30 days the share of time spent protected is 27.2%

The first two columns of the upper table make in-place patching look like the best option: the whole fleet closes in five point two five days, because the patch is applied to all replicas in the same maintenance window. The last replica of the rebuild tied to a code change, on the other hand, closes in fourteen point two five days.

The third column reverses the decision. Over the thirty-day window, in-place patching leaves six hundred seventy-six point six replica-days open (seventy-two point eight percent of the fleet), the rebuild one hundred eighty point six replica-days (nineteen point four percent). Same patch, same fleet, same thirty days: the path that does the first closing three times faster makes the open time four times larger.

Where the Patch Does Not Hold

The lower table gives the reason service by service. Because the in-place patch is not written to the image, the next deployment brings back the unpatched image; how long the patch holds therefore depends not on the patch itself but on the deployment frequency of the service.

The ranking is the reverse of what one would expect. The portal frontend is deployed more often than once in two days and its patch holds for zero point three five days, less than nine hours: a six-replica service sitting at the edge is the service in the fleet that loses the patch fastest. The validator and field sync are deployed once in two weeks, so they carry the patch for eight point seven five days. In-place patching works best on the service that is deployed least and usually faces the least exposed traffic.

The replica-weighted mean is two point six one days; only twenty-seven point two percent of the thirty days passes protected, and even if the weekly maintenance window is repeated the table does not change, because the window comes once a week while deployment is more frequent than once a day for some services. The scheduled rebuild column, on the other hand, carries the same number for all nine services: one point seven five days. The trigger is the base image changing, not a code change in the service; deployment frequency drops out of the equation.

The Price of Disruption

// run2.mjs — the disruption rebuilding brings and the window the runtime layer holds
import { SERVICES, PATHS, WINDOW, READY, replicas, openDays, gen } from './update.mjs';

const fmtRestart = gen([10, 9, 15, 17, 18]);
console.log(fmtRestart(['replicas', 'services', 'window (s)', 'capacity loss', 'weekly restarts']));
for (const k of [...new Set(SERVICES.map((s) => s[1]))].sort((a, b) => b - a)) {
  const g = SERVICES.filter((s) => s[1] === k);
  console.log(fmtRestart([k, g.length, k * READY, `${(100 / k).toFixed(1)}%`, g.reduce((a, s) => a + s[1] * s[2], 0).toFixed(1)]));
}
const current = SERVICES.reduce((a, s) => a + s[1] * s[2], 0), added = (3 / 4) * replicas;
const sorted = SERVICES.map((s) => [s[0], s[1] * s[2]]).sort((a, b) => b[1] - a[1]);
console.log(`fleet window: nine services in parallel ${(Math.max(...SERVICES.map((s) => s[1])) * READY / 60).toFixed(2)} min, ` +
  `sequentially ${(replicas * READY / 60).toFixed(2)} min; most restarts ${sorted[0][0]} (${sorted[0][1]}/week), ` +
  `fewest ${sorted[8][0]} (${sorted[8][1]}/week)`);
console.log(`weekly replica restarts ${current.toFixed(1)}; scheduled rebuild (3 base images per month) ` +
  `adds ${added.toFixed(2)}: ${(100 * added / current).toFixed(1)}% increase. In-place patch adds zero restarts, ` +
  `but in return requires 4 maintenance windows per month and ${4 * replicas} replica touches`);

// The class that can only be stopped at runtime: a component vulnerability with no patch yet, 40 requests
// per day (from the first lesson of this course). Patch lag is the width of the net that holds that class.
const DAILY = 40, fmtNet = gen([32, 16, 24, 18]);
console.log('\n' + fmtNet(['path', 'open fleet-days', 'requests the net holds', 'multiple of best']));
const best = openDays('scheduled rebuild') / replicas;
for (const p of PATHS) {
  const g = openDays(p) / replicas;
  console.log(fmtNet([p, g.toFixed(2), Math.round(g * DAILY), `${(g / best).toFixed(1)}x`]));
}
console.log(`a total of ${WINDOW * DAILY} requests arrive in the thirty-day window`);

replicas   services     window (s)    capacity loss   weekly restarts
6                 2            270            16.7%              36.0
4                 1            180            25.0%              16.0
3                 3            135            33.3%              16.5
2                 3             90            50.0%               5.0
fleet window: nine services in parallel 4.50 min, sequentially 23.25 min; most restarts portal-frontend (30/week), fewest field-sync (1/week)
weekly replica restarts 73.5; scheduled rebuild (3 base images per month) adds 23.25: 31.6% increase. In-place patch adds zero restarts, but in return requires 4 maintenance windows per month and 124 replica touches

path                             open fleet-days  requests the net holds  multiple of best
in-place patch                             21.83                     873             12.5x
code-change rebuild                         5.83                     233              3.3x
scheduled rebuild                           1.75                      70              1.0x
a total of 1200 requests arrive in the thirty-day window

The cost of the scheduled rebuild is that weekly replica restarts rise from seventy-three point five to ninety-six point seven five: a thirty-one point six percent increase. This increase is not evenly distributed. The portal frontend already restarts thirty times a week and the extra load makes no difference; field sync restarts once a week and one and a half times that is added on top. The load falls on the least-deployed services, and those were the services that carried the patch the longest.

The capacity loss column gives the second cost. If one replica is down at a time in a rolling deployment, the loss is sixteen point seven percent for a six-replica service and fifty percent for a two-replica service. Three services run at half capacity for ninety seconds on every round; if the nine services are deployed in parallel, the fleet turns over in four and a half minutes. The real decision is whether those minutes are acceptable, and that question cannot be asked separately from patch lag.

The Width of the Net

The lower table ties the whole measurement of the course to a single number. In the first lesson, of the six hundred sixty-six requests the web application firewall stopped, only forty — six percent — belonged to the class that could not be closed at an earlier stage: a vulnerability in a component with no patch yet, forty requests a day, waiting for the patch to arrive.

Patch lag is exactly the length of time the net has to stay stretched. Under the scheduled rebuild the runtime layer holds seventy requests on its own; under the code-change rebuild, two hundred thirty-three; under in-place patching, eight hundred seventy-three — twelve and a half times. Same rule set, same threshold; the only thing that changes is the update path behind it. The value of runtime defense is measured not by the quality of the rule set but by how long the item behind it stays open.

Summary

• On the three paths the day the last replica closes is 5.25 / 14.25 / 1.75; the thirty-day open replica-days are 676.6 / 180.6 / 54.3, that is 72.8% / 19.4% / 5.8%.
• In-place patching does the first closing fastest but, because it is not written to the image, it falls at the next deployment; protection holds for 2.61 days on a replica-weighted average. The most frequently deployed service loses the patch in 0.35 days, the least frequently deployed carries it for 8.75.
• The scheduled rebuild closes all nine services in 1.75 days: the trigger is the base image changing, and deployment frequency drops out of the equation.
• Its price is weekly replica restarts rising from 73.5 to 96.75 (31.6%), and the load falls on the least-deployed services; the three services with two replicas run at half capacity for 90 seconds on every round.
• The class that can only be stopped at runtime is 40 requests a day; the requests the layer holds on its own are 70 / 233 / 873. Shortening patch lag cuts the layer’s load by a factor of 12.5 without writing a rule.

Course Wrap-Up

Lesson	What the layer catches	What it misses / falsely blocks	Stage that would close it earlier
Web Application Firewall	666 bad requests/day	1,460 missed, 8,000 cut at threshold 3	626 code, only 40 here
Rate Limiting Windows	5,760 requests (sliding window)	6,900 missed, 880 legitimate peaks	3,910 here, the rest code and design
Credential Stuffing Defense	5,512 attempts, 13 takeovers	0 blocked in stuffing, 91 account lockouts	code from 70 to 5, design 0
Security Headers Coverage	55 cells, 63.4% of requests	3 endpoints outside, 24 cells ineffective	3 headers code, 3 headers here
Application-Level Anomaly Detection	3 bad sessions (threshold 20)	2/4 legitimate signals, 7,106 records	1 code, 1 design, none for 1
Secret Store Architecture	31 items in use	48 unused grants (61%)	deployment manifest, 9 times cheaper
Short-Lived Credentials	2,033 times on 16 items	15 items fixed, 99.96% of exposure	design, 8 secrets
Key Rotation	0 cuts at minimum overlap	87,427 requests cut in 60 minutes	code, from 1,440 to 240 minutes
Workload Identity	20 static secrets removed	11 items remain, 7 applications with secrets	design, 3 of 4 gates
Leak Response	early revocation 150,934 calls	224,171 calls, 68,503 requests cut	build, 2,880 minutes latency
Network Segmentation and Service Zero Trust	35 pairs closed, 7.1 down to 4.1	1 to 2 hops to the store, 3.8 pairs of disruption	design, call contract
Container Runtime Detection	12 of 21 events	9 missed, 4 false positives	9 events build, only 3 here
Admission Controller	22 defects, first 6 rules 19 of them	5 legitimate rejections, 5/5 teams	19 pipeline gate, only 3 here
Continuous Configuration Audit	107 findings find an owner	40 unowned, 112 in 12 weeks	122 earlier, none for 25
Rebuild-Based Update	last replica in 1.75 days	5.8% open, +31.6% restarts	build, base image

The third column is the rule of this course. In none of the fifteen lessons could the runtime layer claim the whole of the volume it caught: of every hundred requests the firewall stopped, ninety-four could have been closed in code; of the twenty-two defects the admission controller shut, nineteen sat in the pipeline gate; of the one hundred forty-seven findings continuous auditing found, one hundred twenty-two could have been closed at an earlier stage. The layer’s own share is small in every lesson and real in every lesson — forty requests, three events, three defects, twenty-five unowned findings. Runtime defense is not a patch but a net stretched for the item that cannot be closed earlier. When the question “where would this have closed more cheaply” is not asked for each item, the layer stops being a defense and turns into an excuse: the big number on the dashboard is nothing but an unclosed code defect being caught again every day.

The second column must be read with the same insistence. Eight thousand falsely blocked requests, ninety-one locked accounts, five rejected legitimate deployments, eighty-seven thousand four hundred twenty-seven requests cut during rotation — these are not side effects of the layer but its price. A runtime defense that does not write down its false positives counts as unmeasured.

The next course — M25/K05 Detection, Response and Compliance — takes up the question this table does not answer. The layers were built and what each one misses was written down one by one; but which record would show that what was missed was missed was never asked. Who will do what when a breach occurs, and how all of this practice will be proven in an audit, were not asked either. A measured defense is true only in its own table unless it leaves a record that can be read at the moment of the incident and evidence that can be verified from the outside.