--- title: 'Runtime Security and Secrets Management' source: 'https://academia.sh/en/courses/runtime-security' language: en updated: '2026-10-07T04:31:05+00:00' license: 'CC BY-SA 4.0' lessons: 15 outcomes: - 'Builds application-level defense layers in production and measures their false positives alongside them' - 'Manages the secret lifecycle automatically and narrows the exposure window by a number' - 'Limits lateral movement paths with workload identity and network policies, counting them' --- # Runtime Security and Secrets Management The defense layers built at the stage where catching an item is most expensive, and where each layer could have been closed earlier: the request a web application firewall catches, misses and falsely blocks, rate limiting windows cutting off legitimate peak load, threshold selection in credential stuffing defense, the number of applications covered by managing security headers from the server, application-level anomaly detection, secret store architecture and access control, short-lived credentials narrowing the exposure window, the period during which two secrets are valid together in key rotation, the static secret that workload identity eliminates, the revocation flow after a leak, limiting lateral movement with service-to-service authentication, abnormal process behavior in the container runtime, admission controllers, continuous configuration auditing and rebuild-based updates. ## Application Defense Counting, in the same run, what defenses placed at the request layer at runtime catch, miss, and falsely block: the joint movement of three numbers against the anomaly score threshold of a ready-made rule set, the requests three rate limiting windows cut from legitimate peak load, the attack shape that account and address thresholds each see separately against credential stuffing, the number of applications covered and the endpoints left outside when security headers are managed from the server, and the delay in minutes with which application-level anomaly scoring flags a session that deviates from its baseline, together with the earlier stage at which each item could have been closed. ### Web Application Firewall Source: https://academia.sh/en/courses/runtime-security/web-application-firewall Running a ready-made rule set over a fictional daily request set and writing three counts separately: bad requests blocked, bad requests missed, and legitimate requests falsely blocked; how the three move together when the anomaly score threshold is lowered, what price writing exclusions pays to bring the same three counts back, and how much of the blocked volume could have been closed at a cheaper stage. The previous course scanned the code, inventoried the components, and signed the build output; all of those checks stopped before deployment. What protects the application from the moment it starts running was never asked. This topic answers that question at the request layer, and the first lesson sets up the measure itself. Runtime is the most expensive stage at which to catch an item: the item is in production, a real subscriber is sending the request, and the defense has milliseconds to decide. For that reason the sentence "a web application firewall should be installed" carries no decision by itself. What carries the decision is three counts, and all three must be read from **the same run**: how many bad requests were blocked, how many bad requests got through, how many legitimate requests were falsely blocked. When the third is not written down, the first two measure nothing. #### What the Rule Set Looks At The regional measurement network's subscriber portal and field application stand in the same production environment. A web application firewall is placed in front of them and started with a ready-made rule set: each rule looks for a particular **signal** in the request and writes a score when it matches, and if the request's total score reaches the threshold, the request is blocked. Everything below is a model built with `node`; no real traffic is generated, and no real payload or evasion string is written. - **AD1.** The eleven request classes, their daily counts, the five endpoints, and the ten rules belong to the fictional measurement network. - **AD2.** The true class of a request (legitimate or bad) is known to the model. In production that class is known only after review; the model assumes it is known so that falsely blocked requests can be counted. - **AD3.** The set of signals a class carries is fixed; every request in the same class gets the same score. In reality scores spread across individual requests, and this spread softens the choice of threshold without changing its direction. - **AD4.** A rule matches when **all** the tags it looks for are present. Signal tags name a surface feature the defense sees, not a syntax. - **AD5.** The "earlier closing stage" column was marked by hand for each bad class: whether the item could have been closed in design, in code, or at no earlier stage at all. ```js // requests.mjs — one day of requests and a ready-made rule set; a model, not real traffic // [name, true class, daily count, signal tags, endpoint, earlier closing stage] export const CLASSES = [ ['subscriber query form', 'legit', 84000, [], '/portal/query', '-'], ['field app bulk upload', 'legit', 5400, ['long-body', 'encoded-field'], '/field/upload', '-'], ['report filter, free text', 'legit', 2600, ['quote', 'keyword'], '/portal/report', '-'], ['invoice attachment file name', 'legit', 900, ['path-separator'], '/portal/bill', '-'], ['internal monitoring probe', 'legit', 1440, ['tool-signature'], '/health', '-'], ['concatenation in query field', 'bad', 320, ['quote', 'keyword', 'comment-marker'], '/portal/query', 'code'], ['script tag in form field', 'bad', 210, ['tag-opening', 'event-attribute'], '/portal/query', 'code'], ['traversal in attachment name', 'bad', 96, ['path-separator', 'parent-dir'], '/portal/bill', 'code'], ['quote sequence under cover', 'bad', 260, ['quote', 'keyword'], '/portal/report', 'code'], ['component flaw with no patch', 'bad', 40, ['malformed-body'], '/field/upload', 'none'], ['out-of-workflow bulk download', 'bad', 1200, [], '/portal/report', 'design'], ]; // [rule name, tags looked for (all must be present), score] export const RULES = [ ['quote and keyword', ['quote', 'keyword'], 3], ['comment marker', ['comment-marker'], 2], ['tag opening', ['tag-opening'], 3], ['event attribute', ['event-attribute'], 2], ['path separator', ['path-separator'], 2], ['parent directory', ['parent-dir'], 3], ['malformed body', ['malformed-body'], 4], ['known tool signature', ['tool-signature'], 2], ['encoded field', ['encoded-field'], 2], ['long body', ['long-body'], 1], ]; // exclusions = [endpoint, rule name] pairs; at that endpoint that rule writes no score at all export const score = (s, exclusions = []) => RULES.reduce((t, [name, find, p]) => t + (find.every((e) => s[3].includes(e)) && !exclusions.some(([u, k]) => u === s[4] && k === name) ? p : 0), 0); export function measure(threshold, exclusions = []) { let blocked = 0, missed = 0, falsely = 0; for (const s of CLASSES) { const stopped = score(s, exclusions) >= threshold; if (s[1] === 'bad') stopped ? (blocked += s[2]) : (missed += s[2]); else if (stopped) falsely += s[2]; } return { blocked, missed, falsely }; } ``` A rule set arriving ready-made does not mean the rules know this application. The rules look at a syntactic surface, and that surface is one that legitimate work also passes through: a subscriber typing free text into a report filter and a request attempting concatenation in a query field can carry the same two signals. The rule cannot tell them apart, because the difference is not in the shape of the request but in its **intent and context**; the only thing the rule sees is shape. #### The Three Counts Move Together The threshold is the only real setting of the rule set. In the run below, the rule set, the request set, and the scores stay fixed; the only thing that changes is the threshold. ```js // run.mjs — the same daily set, only the threshold changes import { CLASSES, RULES, score, measure } from './requests.mjs'; const s = (x, n) => String(x).padStart(n); const bad = CLASSES.filter((k) => k[1] === 'bad'); const T = (l) => l.reduce((t, k) => t + k[2], 0); console.log(`${CLASSES.length} request classes, ${RULES.length} rules; ${T(CLASSES)} requests per day` + ` (${T(bad)} bad / ${T(CLASSES) - T(bad)} legit)\n`); console.log(`${'request class'.padEnd(31)}${s('class', 6)}${s('daily', 7)}${s('score', 6)}` + `${s('endpoint', 15)}${s('early stage', 13)}`); for (const k of CLASSES) console.log(`${k[0].padEnd(31)}${s(k[1], 6)}${s(k[2], 7)}${s(score(k), 6)}${s(k[4], 15)}${s(k[5], 13)}`); console.log(`\n${'threshold'.padEnd(11)}${s('blocked', 11)}${s('missed', 8)}${s('falsely blocked', 16)}` + `${s('false/blocked', 19)}`); for (const e of [6, 5, 4, 3, 2]) { const r = measure(e); console.log(`${String(e).padEnd(11)}${s(r.blocked, 11)}${s(r.missed, 8)}${s(r.falsely, 16)}` + `${s(r.blocked ? (r.falsely / r.blocked).toFixed(1) : '-', 19)}`); } ``` ``` 11 request classes, 10 rules; 96466 requests per day (2126 bad / 94340 legit) request class class daily score endpoint early stage subscriber query form legit 84000 0 /portal/query - field app bulk upload legit 5400 3 /field/upload - report filter, free text legit 2600 3 /portal/report - invoice attachment file name legit 900 2 /portal/bill - internal monitoring probe legit 1440 2 /health - concatenation in query field bad 320 5 /portal/query code script tag in form field bad 210 5 /portal/query code traversal in attachment name bad 96 5 /portal/bill code quote sequence under cover bad 260 3 /portal/report code component flaw with no patch bad 40 4 /field/upload none out-of-workflow bulk download bad 1200 0 /portal/report design threshold blocked missed falsely blocked false/blocked 6 0 2126 0 - 5 626 1500 0 0.0 4 666 1460 0 0.0 3 926 1200 8000 8.6 2 926 1200 10340 11.2 ``` The five rows are a tuning curve, and each row has to be read on its own. At threshold 6 the rule set blocks nothing: no class reaches a total score of 6, which means a layer that is installed but inert cannot be told apart from zero in day-to-day measurement. At threshold 5 three classes are blocked and 626 bad requests are cut; falsely blocked requests are zero. Lowering the threshold to 4 adds only 40 requests, and falsely blocked stays zero; four is the setting that costs nothing in this set. The real decision is in the third row. Lowering the threshold to 3 blocks 260 more bad requests and, in exchange, cuts **8,000 legitimate requests**. The ratio is in the last column: 8.6 legitimate requests cut for every bad request blocked. Nor are the ones cut random; they are the field application's bulk upload and the free text typed into the report filter, which means two real functions of the system. At threshold 2 the blocked bad requests do not increase at all, and the cut requests increase by 2,340: past a certain point, tightening only starts to cut legitimate work. The missed column never drops below 1,200 at any setting. The reason is not a choice of threshold but a **limit of sight**: the out-of-workflow bulk download carries no signal at all. The requests are flawless in shape, authenticated, and authorized; the only thing wrong is volume and context. A rule set that looks at syntax cannot see this at any threshold. #### The Price of Writing Exclusions Holding the threshold at 3 and closing the false blocking one case at a time is the most common answer in production: the endpoint that was cut and the rule that cut it are written into an exclusion list, and at that endpoint that rule writes no score. Three exclusions are enough: two rules at the field upload endpoint and the quote rule at the report endpoint. ```js // exclusions.mjs — the price of closing false blocking at threshold 3 with exclusions import { CLASSES, score, measure } from './requests.mjs'; const s = (x, n) => String(x).padStart(n); const EXCLUSIONS = [['/field/upload', 'long body'], ['/field/upload', 'encoded field'], ['/portal/report', 'quote and keyword']]; const SETTINGS = [['threshold 4, no exclusions', 4, []], ['threshold 3, no exclusions', 3, []], ['threshold 3, 3 exclusions', 3, EXCLUSIONS]]; console.log(`${'setting'.padEnd(27)}${s('blocked', 11)}${s('missed', 8)}${s('falsely blocked', 16)}${s('tracked items', 15)}`); for (const [name, e, m] of SETTINGS) { const r = measure(e, m); console.log(`${name.padEnd(27)}${s(r.blocked, 11)}${s(r.missed, 8)}${s(r.falsely, 16)}${s(10 + m.length, 15)}`); } console.log(`\nthe surface each exclusion opens at its endpoint:`); for (const [u, k] of EXCLUSIONS) { const passed = CLASSES.filter((x) => x[4] === u && x[1] === 'bad' && score(x) >= 3 && score(x, EXCLUSIONS) < 3); console.log(` ${u.padEnd(15)}${k.padEnd(26)}${passed.length ? `passed: ${passed[0][0]} (${passed[0][2]})` : 'no bad request passed'}`); } ``` ``` setting blocked missed falsely blocked tracked items threshold 4, no exclusions 666 1460 0 10 threshold 3, no exclusions 926 1200 8000 10 threshold 3, 3 exclusions 666 1460 0 13 the surface each exclusion opens at its endpoint: /field/upload long body no bad request passed /field/upload encoded field no bad request passed /portal/report quote and keyword passed: quote sequence under cover (260) ``` The first and third rows are identical in all three counts: 666, 1,460, 0. In other words, the threshold tightened with three exclusions gives exactly the same result as the loose threshold with no exclusions written at all. The difference is only in the last column: the number of tracked items has gone from ten to thirteen. Three more items, three separate justifications, three lines to be reviewed again at every endpoint change. The sublist gives the reason. An exclusion switches off one rule at one endpoint and covers **every request** that passes through that endpoint. The quote exclusion at the report endpoint rescued the legitimate filter text, and in the same stroke it rescued the 260 bad requests that pass through the same endpoint: the three-point rule no longer writes anything there. The two exclusions at the field upload endpoint did not produce this side effect, because the only bad class at that endpoint is caught by a different rule. The difference between the two cannot be known in advance; it becomes visible only when the set of requests the exclusion covers is counted. An exclusion list is therefore not a setting but an inventory: unless it is tracked separately which rule each line silences at which endpoint, and what else passes through that endpoint, the list becomes, over time, more decisive than the rule set itself. #### Where the Blocked Volume Belongs The first two columns say what this layer does. The third column is the real question of the course: could the blocked volume already have been closed at a cheaper stage. ```js // stage.mjs — how much of the blocked item could have been closed at a cheaper stage import { CLASSES, score } from './requests.mjs'; const s = (x, n) => String(x).padStart(n); const bad = CLASSES.filter((k) => k[1] === 'bad'); const STAGES = ['code', 'design', 'none']; console.log(`${'threshold'.padEnd(11)}${STAGES.map((a) => s(a, 10)).join('')}${s('only here %', 17)}`); for (const e of [5, 4, 3]) { const d = bad.filter((k) => score(k) >= e); const t = d.reduce((x, k) => x + k[2], 0); const share = STAGES.map((a) => d.filter((k) => k[5] === a).reduce((x, k) => x + k[2], 0)); console.log(`${String(e).padEnd(11)}${share.map((p) => s(p, 10)).join('')}` + `${s(t ? (100 * share[2] / t).toFixed(1) : '-', 17)}`); } console.log(`\nbad classes never blocked at any threshold:`); for (const k of bad.filter((x) => score(x) < 2)) console.log(` ${k[0].padEnd(31)}${s(k[2], 7)} no signal, earlier stage: ${k[5]}`); console.log(`\nclasses that can be closed only at runtime:`); for (const k of bad.filter((x) => x[5] === 'none')) console.log(` ${k[0].padEnd(31)}${s(k[2], 7)} score ${score(k)}, stops at threshold 4`); ``` ``` threshold code design none only here % 5 626 0 0 0.0 4 626 0 40 6.0 3 886 0 40 4.3 bad classes never blocked at any threshold: out-of-workflow bulk download 1200 no signal, earlier stage: design classes that can be closed only at runtime: component flaw with no patch 40 score 4, stops at threshold 4 ``` The table shows the layer's real share. Of the 666 requests blocked at threshold 4, 626, or 94%, come from classes that can be closed at the code stage: query concatenation closes with a parameterized query, the script tag with output encoding, directory traversal with path validation, and each of the three is written once and holds for all requests. This layer catches the same items again every day, on every request. The remaining 6% is something else. For a flaw in a component that has no patch yet, there is no earlier stage: the item was not in the design, it was not written into the code, and at the moment it showed up in the dependency inventory no fixed release existed. The only thing that blocks those 40 requests is this layer, and that is exactly the reason the rule set exists. The measure of the layer is not the total number of blocks but the number **blocked only here**. The sublist tells the opposite direction. The out-of-workflow bulk download is not blocked at any threshold; this item is caught neither by the rules nor by the threshold, because it is an authorization and workflow decision, which means it belongs to the design stage. Even though the rule set never sees this item, the daily dashboard will still show 666 blocks. Having the layer installed does not widen the class of items it covers. #### Summary - A runtime defense is measured with three counts, and all three are read from the same run: bad requests blocked, bad requests missed, legitimate requests falsely blocked. When the third is not written down, the first two measure nothing. - Lowering the threshold from 5 to 4 raised blocked requests from 626 to 666 and left falsely blocked at zero; going down to 3 blocked 260 more requests and cut 8,000 legitimate requests (8.6 cut per blocked request), and going down to 2 added 2,340 more cuts without adding any gain. - Threshold 3 with three exclusions gave the same three counts as threshold 4 with no exclusions (666, 1,460, 0); the only difference is that the tracked items went from 10 to 13. One of the exclusions also covered the 260 bad requests passing through the same endpoint. - Missed requests never dropped below 1,200 at any threshold: the out-of-workflow bulk download carries no signal. A rule that looks at syntax sees shape, not context. - Of the 666 requests blocked at threshold 4, 626 come from classes that can be closed once at the code stage. The part that can be closed only at runtime is 40 requests per day, which is 6% — a component flaw with no patch. #### Next Step This lesson's rule set looked at a single request and decided: is there a signal, does the score reach the threshold. The 1,200 missed requests were missed for exactly this reason: none of them was wrong on its own. The wrongness of the out-of-workflow bulk download is not inside the request but in **the number and timing of requests coming from the same client**. To see it, the defense has to read not a request but a counter: how many requests per window, tied to which key, under which threshold. The next lesson runs that counter in three different window shapes and asks the same three counts there, especially of the second, because the most expensive side of a rate limit is not the bad request it blocks but the request it cuts from legitimate peak load. ### Rate Limiting Windows Source: https://academia.sh/en/courses/runtime-security/rate-limiting-windows Running the same five-minute load through three different rate limit windows and how the window shape shifts the three counts: the window boundary effect of the fixed window, the request the sliding window cuts from legitimate peak load, the token bucket saving the peak share while also letting the attack peak through, how many bad requests the limit that saves the legitimate peak lets go, and which part of the cut volume could have been closed at an earlier stage. The previous lesson had a rule set look at requests one by one, and it showed that the 1,200 missed requests were missed for exactly this reason: none of them was wrong on its own. The wrongness was not inside the request but in **the number and timing of requests** from the same client. Rate limiting is the defense that looks at this axis, and its measure is not the bad requests it stops but what it cuts from legitimate peak load. Throttling algorithms were measured earlier and are not repeated here. This lesson asks a single question: when the same nominal limit is applied in three different window shapes, where do the three counts shift. The window shape looks like an implementation detail; once measured, it turns out to be the main decision that determines whether legitimate work is cut. #### The Same Load, Three Window Shapes A five-minute slice is taken from the regional measurement network's production environment. Everything below is a model built with `node`; no real traffic is generated, and no request body is written. - **AD6.** The seven client classes, their arrival series, and their requests per second belong to the fictional measurement network. Two of the legitimate classes have a peak load that is the work itself: the sessions the field application opens at shift start, and the bulk query that runs in the billing period. - **AD7.** The rate limit is tied to a single key per client, and in this lesson the key is fixed. Whether the key is an account or an address is a separate axis; the next lesson measures it. - **AD8.** Arrival rate is spread evenly within a second. In reality clustering within a second raises the amount the sliding window cuts somewhat, without changing its direction. - **AD9.** The three shapes carry **the same nominal limit**: 600 requests per minute. The token bucket adds 10 tokens per second and fills at 900 tokens; the average rate is the same, the peak share differs. - **AD10.** The "earlier closing stage" column was marked by hand for each bad client. ```js // load.mjs — five-minute arrival series and three window shapes; a model, not real traffic // [client, true class, [[t0, t1, requests per second], ...], earlier closing stage] export const DURATION = 300; export const CLIENTS = [ ['subscriber portal, steady', 'legit', [[0, 300, 4]], '-'], ['field app, shift start', 'legit', [[0, 110, 2], [110, 150, 30], [150, 300, 2]], '-'], ['billing period bulk query', 'legit', [[0, 45, 1], [45, 65, 40], [65, 300, 1]], '-'], ['internal monitoring poll', 'legit', [[0, 300, 1]], '-'], ['out-of-workflow bulk download', 'bad', [[0, 300, 9]], 'design'], ['page crawling scraper', 'bad', [[0, 300, 26]], 'none'], ['login attempt series', 'bad', [[40, 100, 18], [160, 220, 18]], 'code'], ]; export const series = (profile) => { const a = new Array(DURATION).fill(0); for (const [t0, t1, r] of profile) for (let t = t0; t < t1; t++) a[t] = r; return a; }; // fixed window: the counter resets at the window boundary export function fixed(a, limit, win) { let passed = 0, counter = 0; for (let t = 0; t < a.length; t++) { if (t % win === 0) counter = 0; const allow = Math.min(a[t], Math.max(0, limit - counter)); counter += allow; passed += allow; } return passed; } // sliding window: requests accepted in the last `win` seconds are counted export function sliding(a, limit, win) { let passed = 0; const accepted = new Array(a.length).fill(0); for (let t = 0; t < a.length; t++) { let recent = 0; for (let u = Math.max(0, t - win + 1); u < t; u++) recent += accepted[u]; accepted[t] = Math.min(a[t], Math.max(0, limit - recent)); passed += accepted[t]; } return passed; } // token bucket: `rate` tokens are added per second, the bucket fills at `cap` tokens export function bucket(a, rate, cap) { let passed = 0, tokens = cap; for (let t = 0; t < a.length; t++) { tokens = Math.min(cap, tokens + rate); const allow = Math.min(a[t], tokens); tokens -= allow; passed += allow; } return passed; } export const SHAPES = [ ['fixed window 60 s', (a) => fixed(a, 600, 60)], ['sliding window 60 s', (a) => sliding(a, 600, 60)], ['token bucket 10/s', (a) => bucket(a, 10, 900)], ]; export function measure(shape) { let caught = 0, missed = 0, falsely = 0, legit = 0; for (const [, cls, profile] of CLIENTS) { const a = series(profile); const total = a.reduce((x, y) => x + y, 0); const cut = total - shape(a); if (cls === 'bad') { caught += cut; missed += total - cut; } else { falsely += cut; legit += total; } } return { caught, missed, falsely, legit }; } ``` The only difference between the three shapes is how the counter keeps its history. The fixed window resets the counter at the start of the minute; it remembers no request from the previous minute. The sliding window remembers every request it accepted in the last sixty seconds. The token bucket keeps no window at all: it accumulates the unused share and allows it to be spent in one go. #### Requests Cut from Legitimate Peak Load ```js // run-windows.mjs — the same five-minute load; the only thing that changes is the window shape import { CLIENTS, SHAPES, series, measure } from './load.mjs'; const s = (x, n) => String(x).padStart(n); const T = (f) => CLIENTS.filter(f).reduce((t, k) => t + series(k[2]).reduce((x, y) => x + y, 0), 0); console.log(`${CLIENTS.length} clients, 300 seconds; total ${T(() => true)} requests` + ` (${T((k) => k[1] === 'bad')} bad / ${T((k) => k[1] === 'legit')} legit)\n`); console.log(`${'client'.padEnd(36)}${s('class', 6)}${s('total', 7)}` + SHAPES.map(([name]) => s(name.split(' ')[0], 9)).join('') + s('earlier stage', 15)); for (const [name, cls, profile, stage] of CLIENTS) { const a = series(profile), total = a.reduce((x, y) => x + y, 0); console.log(`${name.padEnd(36)}${s(cls, 6)}${s(total, 7)}` + SHAPES.map(([, f]) => s(total - f(a), 9)).join('') + s(stage, 15)); } console.log(`\n(three columns: requests cut by that shape)\n`); console.log(`${'window shape'.padEnd(20)}${s('caught', 11)}${s('missed', 8)}` + `${s('falsely blocked', 16)}${s('legit cut %', 15)}`); for (const [name, f] of SHAPES) { const r = measure(f); console.log(`${name.padEnd(20)}${s(r.caught, 11)}${s(r.missed, 8)}${s(r.falsely, 16)}` + `${s((100 * r.falsely / r.legit).toFixed(1), 15)}`); } ``` ``` 7 clients, 300 seconds; total 16960 requests (12660 bad / 4300 legit) client class total fixed sliding token earlier stage subscriber portal, steady legit 1200 0 0 0 - field app, shift start legit 1720 360 640 0 - billing period bulk query legit 1080 45 240 0 - internal monitoring poll legit 300 0 0 0 - out-of-workflow bulk download bad 2700 0 0 0 design page crawling scraper bad 7800 4800 4800 3910 none login attempt series bad 2160 240 960 0 code (three columns: requests cut by that shape) window shape caught missed falsely blocked legit cut % fixed window 60 s 5040 7620 405 9.4 sliding window 60 s 5760 6900 880 20.5 token bucket 10/s 3910 8750 0 0.0 ``` The three rows carry the same nominal limit, and in all three the three counts differ. The sliding window stops the most bad requests: 5,760. In the same row **20.5%** of legitimate load is cut, and what is cut is not random requests but the 640 sessions the field application opens at shift start and the 240 queries of the billing period. The fixed window cuts less because it resets the counter at the minute boundary; the shift peak is split across two windows, so it escapes with 360 requests. The same reset works on the bad side too: only 240 requests are cut from the login attempt series, because both sixty-second peaks fall on the two sides of the boundary. The window boundary effect does not choose whom it will favor. The token bucket row is the most striking one. **Nothing** is cut from legitimate requests: the accumulation of 900 tokens covers both the shift peak and the billing peak in full. The same accumulation covers the login attempt series in full too, all 2,160 requests getting through. The bucket does not ask whether a peak is legitimate; it looks only at whether there is accumulated share. A defense that wants to recognize peak share has to recognize the peak-shaped attack as well. The out-of-workflow bulk download of 2,700 requests is not cut in any of the three shapes. Nine requests per second is below the limit of 600 per minute; as long as the client works at a steady rate just under the limit, no window shape sees it. The item that slipped past the previous lesson's rule set slips past the rate limit as well. #### The Price of Raising the Limit The 880 legitimate requests the sliding window cuts mean a disruption in production: part of the field crew cannot get into the application at shift start. The first answer given is to raise the limit. ```js // limit.mjs — the price of raising the limit to save the legitimate peak load import { CLIENTS, series, sliding, bucket } from './load.mjs'; const s = (x, n) => String(x).padStart(n); const BAD = CLIENTS.filter((k) => k[1] === 'bad'); const LEGIT = CLIENTS.filter((k) => k[1] === 'legit'); const attempts = CLIENTS.find((k) => k[0].startsWith('login')); const cut = (list, f) => list.reduce((t, k) => { const a = series(k[2]); return t + a.reduce((x, y) => x + y, 0) - f(a); }, 0); console.log(`sliding window, 60 s; only the limit changes`); console.log(`${'limit'.padEnd(7)}${s('falsely blocked', 16)}${s('caught', 11)}${s('missed', 8)}` + `${s('login attempts passed', 23)}`); for (const L of [600, 800, 1000, 1240, 1500]) { const f = (a) => sliding(a, L, 60); console.log(`${String(L).padEnd(7)}${s(cut(LEGIT, f), 16)}${s(cut(BAD, f), 11)}` + `${s(12660 - cut(BAD, f), 8)}${s(f(series(attempts[2])), 23)}`); } console.log(`\ntoken bucket, 10/s; only the bucket capacity changes`); console.log(`${'capacity'.padEnd(10)}${s('falsely blocked', 16)}${s('caught', 11)}${s('missed', 8)}` + `${s('login attempts passed', 23)}`); for (const K of [200, 400, 600, 900, 1200]) { const f = (a) => bucket(a, 10, K); console.log(`${String(K).padEnd(10)}${s(cut(LEGIT, f), 16)}${s(cut(BAD, f), 11)}` + `${s(12660 - cut(BAD, f), 8)}${s(f(series(attempts[2])), 23)}`); } ``` ``` sliding window, 60 s; only the limit changes limit falsely blocked caught missed login attempts passed 600 880 5760 6900 1200 800 480 4360 8300 1600 1000 240 2960 9700 2000 1240 0 1600 11060 2160 1500 0 300 12360 2160 token bucket, 10/s; only the bucket capacity changes capacity falsely blocked caught missed login attempts passed 200 1020 5190 7470 1580 400 620 4590 8070 1980 600 220 4210 8450 2160 900 0 3910 8750 2160 1200 0 3610 9050 2160 ``` The two tables say the same thing through two different settings. In the sliding window, falsely blocked requests fall to zero only when the limit reaches 1,240; the busiest sixty-second slice of the shift peak is exactly that much. In the same row, caught requests have dropped from 5,760 to 1,600, and missed requests have risen from 6,900 to 11,060. The last column writes the real price of the decision: **all** of the login attempt series starts to pass. The token bucket table repeats the same curve on the capacity axis. The first capacity at which the legitimate peak passes with zero cuts is 900; the first capacity at which the whole login series passes is 600. That is, the bucket has already saved the attack peak **before** it is loosened enough to save the legitimate peak. The order cannot be reversed, because the two peaks are the same thing: a short-lived high rate. What comes out of this is not a setting recommendation but a limit: in a system whose peak load is the work itself, a single-threshold rate limit tied to a single key has to let the attack peak through in order to let the legitimate peak through. The distinction is not in the rate but in **what** the request is made to; a rate counter does not see that. #### Where What Is Cut Belongs ```js // stage-share.mjs — how much of the cut bad requests could have been closed at a cheaper stage import { CLIENTS, SHAPES, series, sliding } from './load.mjs'; const s = (x, n) => String(x).padStart(n); const BAD = CLIENTS.filter((k) => k[1] === 'bad'); const LEGIT = CLIENTS.filter((k) => k[1] === 'legit'); const STAGES = ['design', 'code', 'none']; const cut = (k, f) => { const a = series(k[2]); return a.reduce((x, y) => x + y, 0) - f(a); }; console.log(`${'window shape'.padEnd(20)}${STAGES.map((a) => s(a, 10)).join('')}${s('only here %', 17)}`); for (const [name, f] of SHAPES) { const share = STAGES.map((a) => BAD.filter((k) => k[3] === a).reduce((x, k) => x + cut(k, f), 0)); const t = share.reduce((x, y) => x + y, 0); console.log(`${name.padEnd(20)}${share.map((p) => s(p, 10)).join('')}${s((100 * share[2] / t).toFixed(1), 17)}`); } console.log(`\nsliding window; limit needed to catch the out-of-workflow bulk download (9/s)`); console.log(`${'limit'.padEnd(7)}${s('cut from bulk download', 26)}${s('legit cut', 15)}${s('legit cut %', 15)}`); const download = BAD.find((k) => k[3] === 'design'); for (const L of [600, 540, 480, 360]) { const f = (a) => sliding(a, L, 60); const m = LEGIT.reduce((x, k) => x + cut(k, f), 0); console.log(`${String(L).padEnd(7)}${s(cut(download, f), 26)}${s(m, 15)}${s((100 * m / 4300).toFixed(1), 15)}`); } ``` ``` window shape design code none only here % fixed window 60 s 0 240 4800 95.2 sliding window 60 s 0 960 4800 83.3 token bucket 10/s 0 0 3910 100.0 sliding window; limit needed to catch the out-of-workflow bulk download (9/s) limit cut from bulk download legit cut legit cut % 600 0 880 20.5 540 0 1000 23.3 480 300 1120 26.0 360 900 1360 31.6 ``` The upper table gives the layer's real share. The page crawling scraper cannot be closed at any earlier stage: its requests are individually legitimate, authenticated, and authorized, and only their number per unit of time is meaningful. The 4,800 requests cut in all three shapes (3,910 in the bucket) belong to this class, and this is exactly the reason this layer exists. The login attempt series belongs somewhere else: once a second factor is required at the code stage, the series loses its value, because even if the right password is found, no session is opened. The 240 or 960 requests the rate limit cuts from that series are the repeated catching, every day, of an item that can be closed. The lower table shows the item that belongs to the design stage. To see the out-of-workflow bulk download with a rate limit, the limit has to be lowered to 480; at that setting 300 requests are cut from this client and 1,120 from legitimate load, which is **a quarter** of the legitimate work. When the limit drops to 360, the bad requests cut rise to 900 and the legitimate cut to 31.6%. This item can be closed with a rate limit only by making the system unusable; its right place is design. Once the bulk download is moved into a queue job, there is no room left to work at a steady rate just under the limit. #### Summary - The same nominal limit (600 requests per minute) gave three different results in three window shapes: fixed window 5,040 / 7,620 / 405, sliding window 5,760 / 6,900 / 880, token bucket 3,910 / 8,750 / 0 (caught / missed / falsely blocked). - The sliding window cut 20.5% of legitimate load, and what it cut was the shift start and billing period peaks. The fixed window cutting less is not an advantage but a by-product of the window boundary effect: the same reset let the login series off with 240 requests cut instead of 960. - The token bucket cut nothing from the legitimate peak but let all 2,160 requests of the login attempt series through. By the time the bucket capacity was opened enough to save the legitimate peak (900), the attack peak had already escaped at a capacity of 600. - The limit that zeroes falsely blocked requests in the sliding window is 1,240; at that setting caught requests fall from 5,760 to 1,600 and missed requests rise to 11,060. In a system whose peak load is the work itself, a single-threshold rate limit lets the attack peak through in order to let the legitimate peak through. - The only part of the cut volume that can be closed solely here is the scraper (4,800 / 3,910). The login series closes with a second factor at the code stage, the out-of-workflow bulk download with a queue job in design; seeing the latter with a rate limit requires lowering the limit to 480, that is, cutting a quarter of legitimate work. #### Next Step This lesson's counter was tied to a single key, and what that key was never came up. The part of the login attempt series that was cut or missed stayed uncertain for exactly this reason: the series' rate per second was known, but **how many accounts** and **how many addresses** the attempts were spread across was not. The same total rate is one kind of attack when loaded onto a single account and a very different kind when distributed as a single attempt each across 10,000 accounts, and the second is invisible in every setting of this lesson. The next lesson moves the threshold off the rate axis and onto the account and address axes, separates which threshold stops which attack shape, and asks the same three counts there, especially of the legitimate account that gets locked out. ### Credential Stuffing Defense Source: https://academia.sh/en/courses/runtime-security/credential-stuffing-defense Running account and address thresholds separately over a one-hour login attempt record and separating which threshold stops which attack shape: the account threshold stops brute force and the address threshold stops concentrated-source spraying, while stuffing with a single attempt per account passes both; the gap between the number of attempts blocked and the number of account takeovers prevented, and the lockout of the shared egress address and of the legitimate account being targeted. The previous lesson tied the rate counter to a single key and never asked what that key was. The state of the login attempt series stayed uncertain for exactly this reason: the rate per second was known, but how many accounts and how many addresses the attempts were spread across was not. The same total rate is one kind of attack when loaded onto a single account and a very different kind when distributed as a single attempt each across 4,000 accounts. This lesson moves the threshold off the rate axis and onto the account and address axes. The names of the two shapes are kept apart. **Credential stuffing** is the trying, in this system, of account–password pairs that leaked from somewhere else; usually a single attempt is made per account, and the hope is that the same password was used here too. **Spraying** is the trying of a small number of common passwords across many accounts; a few attempts fall on each account. Both are sparse on the account axis, but they show different density on the address axis, and a defense can measure exactly this difference. #### Three Attack Shapes, Two Axes Everything below is a model built with `node`. No attempt is actually made, and no credential list or attempt tool is written; the attack is represented only as a record row, with target account and source address fields. - **AD11.** The 1,900 real accounts, the attack shapes, and their counts belong to the fictional measurement network. The record is generated with a visible seed; because the generator is written in the lesson itself, the run is repeatable. - **AD12.** Whether an attempt succeeds derives in the model from a property of the account: stuffing succeeds only on an account whose password belongs to a leaked pair, and spraying succeeds only on an account whose password is common. - **AD13.** The threshold looks at the number of **failed** attempts accumulated within the window; a successful attempt does not increment the counter. This is the common behavior in production, and it is the source of this lesson's main finding. - **AD14.** The legitimate failure rate is 12%, and the whole field crew comes from a single shared egress address. Thirty subscribers also forget their passwords and try one after another. - **AD15.** When the threshold is exceeded, later attempts are blocked; an account lock and an address throttle give the same result. In reality the two take different times to clear, which does not change the direction of the numbers. ```js // login.mjs — one-hour login attempt record; a model, no real attempts are generated export const SEED = 20250804; let s = SEED; const rnd = () => (s = (s * 48271) % 2147483647) / 2147483647; const int = (n) => Math.floor(rnd() * n); // real accounts: is a second factor enrolled, does the password belong to a leaked pair, is it common export const ACCOUNTS = []; for (let i = 0; i < 1900; i++) ACCOUNTS.push({ name: `account-${i}`, factor: rnd() < 0.86, reused: rnd() < 0.038, common: rnd() < 0.009 }); export const BY_NAME = new Map(ACCOUNTS.map((a) => [a.name, a])); // [minute, account, address, class, success] export function generateRecords() { const k = []; for (let i = 0; i < 500; i++) // field crew: a single shared egress address k.push([int(60), ACCOUNTS[int(380)].name, 'shared-egress', 'legit', rnd() > 0.12]); for (let i = 0; i < 1900; i++) // portal subscribers: scattered addresses k.push([int(60), ACCOUNTS[380 + int(1520)].name, `subscriber-addr-${int(900)}`, 'legit', rnd() > 0.12]); for (let i = 0; i < 30; i++) { // subscriber who forgot the password: consecutive attempts const n = 6 + int(4), t0 = int(50), addr = `subscriber-addr-${int(900)}`, acct = ACCOUNTS[380 + int(1520)].name; for (let j = 0; j < n; j++) k.push([t0 + j, acct, addr, 'legit', j === n - 1]); } for (let i = 0; i < 4000; i++) { // stuffing: a single attempt per account, 400 addresses const acct = i < 1900 ? ACCOUNTS[i] : null; k.push([int(60), acct ? acct.name : `unknown-${i}`, `stuffing-addr-${i % 400}`, 'stuffing', Boolean(acct && acct.reused)]); } for (let r = 0; r < 3; r++) // spraying: three rounds, one attempt per account for (let i = 0; i < 1900; i++) k.push([5 + r * 20 + int(5), ACCOUNTS[i].name, `spraying-addr-${i % 40}`, 'spraying', ACCOUNTS[i].common && r === i % 3]); for (let i = 0; i < 600; i++) // brute force: a single account, a single address k.push([10 + Math.floor(i / 120), ACCOUNTS[7].name, 'brute-addr', 'bruteforce', false]); return k.sort((a, b) => a[0] - b[0]); } // threshold: failed attempts accumulated within the window; once exceeded, later attempts are blocked export function measure(records, accountLimit, addressLimit) { const accountFails = new Map(), addressFails = new Map(), locked = new Set(); const byKind = {}, taken = []; let blockedLegit = 0, legitByAccount = 0, legitByAddress = 0, blockedBad = 0, passedBad = 0; for (const [, account, address, cls, success] of records) { const f = accountFails.get(account) || 0, g = addressFails.get(address) || 0; const blocked = f >= accountLimit || g >= addressLimit; if (cls === 'legit') { if (blocked) { blockedLegit++; locked.add(account); f >= accountLimit ? legitByAccount++ : legitByAddress++; } else if (!success) { accountFails.set(account, f + 1); addressFails.set(address, g + 1); } continue; } byKind[cls] = byKind[cls] || { blocked: 0, passed: 0, takenOver: 0 }; if (blocked) { blockedBad++; byKind[cls].blocked++; continue; } passedBad++; byKind[cls].passed++; if (success) { byKind[cls].takenOver++; taken.push([account, cls]); } else { accountFails.set(account, f + 1); addressFails.set(address, g + 1); } } return { blockedLegit, legitByAccount, legitByAddress, lockedAccounts: locked.size, blockedBad, passedBad, takenOver: taken, byKind }; } ``` #### Which Threshold Sees What ```js // run-thresholds.mjs — the same record; the only thing that changes is the account and address thresholds import { SEED, generateRecords, measure } from './login.mjs'; const s = (x, n) => String(x).padStart(n); const K = generateRecords(); const count = (f) => K.filter(f).length; const I = Infinity; console.log(`seed ${SEED}; 60 minutes, ${K.length} attempts` + ` (${count((r) => r[3] === 'legit')} legit / ${count((r) => r[3] !== 'legit')} bad)\n`); console.log(`${'attack kind'.padEnd(14)}${s('attempts', 9)}${s('accounts', 10)}${s('addresses', 11)}` + `${s('per account', 14)}${s('per address', 14)}${s('accounts taken over', 21)}`); for (const t of ['stuffing', 'spraying', 'bruteforce']) { const r = K.filter((x) => x[3] === t); const a = new Set(r.map((x) => x[1])).size, d = new Set(r.map((x) => x[2])).size; console.log(`${t.padEnd(14)}${s(r.length, 9)}${s(a, 10)}${s(d, 11)}` + `${s((r.length / a).toFixed(1), 14)}${s((r.length / d).toFixed(1), 14)}` + `${s(r.filter((x) => x[4]).length, 21)}`); } const SETTINGS = [['no threshold', I, I], ['account 5', 5, I], ['address 50', I, 50], ['account 5 + address 50', 5, 50], ['account 3 + address 20', 3, 20]]; console.log(`\n${'setting'.padEnd(24)}${s('blocked bad', 17)}${s('passed bad', 12)}` + `${s('taken over', 13)}${s('blocked legit', 18)}${s('locked accounts', 17)}`); for (const [name, ae, de] of SETTINGS) { const r = measure(K, ae, de); console.log(`${name.padEnd(24)}${s(r.blockedBad, 17)}${s(r.passedBad, 12)}${s(r.takenOver.length, 13)}` + `${s(r.blockedLegit, 18)}${s(r.lockedAccounts, 17)}`); } console.log(`\nblocked attempts by attack kind:`); console.log(`${'setting'.padEnd(24)}${['stuffing', 'spraying', 'bruteforce'].map((t) => s(t, 12)).join('')}`); for (const [name, ae, de] of SETTINGS.slice(1)) { const r = measure(K, ae, de); console.log(`${name.padEnd(24)}` + ['stuffing', 'spraying', 'bruteforce'].map((t) => s(r.byKind[t].blocked, 12)).join('')); } ``` ``` seed 20250804; 60 minutes, 12923 attempts (2623 legit / 10300 bad) attack kind attempts accounts addresses per account per address accounts taken over stuffing 4000 4000 400 1.0 10.0 57 spraying 5700 1900 40 3.0 142.5 13 bruteforce 600 1 1 600.0 600.0 0 setting blocked bad passed bad taken over blocked legit locked accounts no threshold 0 10300 70 0 0 account 5 669 9631 69 220 91 address 50 4244 6056 63 227 164 account 5 + address 50 4301 5999 62 364 194 account 3 + address 20 5512 4788 57 613 315 blocked attempts by attack kind: setting stuffing spraying bruteforce account 5 18 55 596 address 50 0 3694 550 account 5 + address 50 11 3694 596 account 3 + address 20 15 4899 598 ``` The first table puts the two axes side by side, and the distinction is read there. Brute force sits at the extreme on the account axis with 600 attempts; when the account threshold is set to 5, 596 of its attempts are blocked, so this shape counts as finished with a single threshold. Spraying is sparse on the account axis (3 attempts per account, under the threshold), but because it is squeezed into 40 addresses it rises to 142.5 attempts per address; when the address threshold is set to 50, 3,694 of its attempts are blocked. Stuffing stays under the threshold on both axes: one attempt per account, ten per address. The address threshold blocks **zero** attempts from this shape. The 15 and 18 seen in the stuffing column of the lower table come not from stuffing's own density but from the same accounts also being targeted by spraying: when the failures of the two shapes add up on the same account counter, a few accounts cross the threshold. That is, stuffing is seen only indirectly, and only when another shape is next to it. The last two columns of the middle table show the cost. At account threshold 5, 91 legitimate accounts are locked, and only 30 of them are subscribers who forgot their passwords; the rest were locked **because they were the target of the attack**. Failed attempts aimed at their accounts filled the counter, and the real owner could no longer get in. At address threshold 50, 227 legitimate attempts are cut; all belong to the field crew coming from the shared egress address, because the 12% failure share of the 500 attempts passing through that address crosses the threshold on its own. The address threshold cuts not an account but a team. #### The Takeover the Threshold Does Not Prevent The column most likely to be overlooked in the table above is the "taken over" column. The number that stands out in a dashboard is the blocked attempts, and as the setting tightens that number climbs from hundreds to thousands. The taken over column, by contrast, barely moves: 70 accounts are taken over in the run with no threshold, and 57 at the tightest setting. 5,512 attempts were blocked, and takeovers fell by only 13. These two numbers have to sit in the same table, because the purpose of the defense is not to reduce attempts but to prevent the account from being taken over. ```js // stage-cost.mjs — which stage closes the same account takeover earlier import { ACCOUNTS, BY_NAME, generateRecords, measure } from './login.mjs'; const s = (x, n) => String(x).padStart(n); const K = generateRecords(), I = Infinity; const legit = K.filter((r) => r[3] === 'legit').length; const base = measure(K, I, I), tight = measure(K, 3, 20); const noFactor = (r) => r.takenOver.filter(([name]) => !BY_NAME.get(name).factor).length; const setting = ACCOUNTS.filter((a) => a.reused || a.common).length; console.log(`${'measure'.padEnd(34)}${s('stage', 16)}${s('taken over', 13)}${s('legit cost', 30)}`); for (const [a, b, c, d] of [ ['no measure', '-', base.takenOver.length, '0'], ['account 3 + address 20 threshold', 'runtime', tight.takenOver.length, `${tight.blockedLegit} attempts, ${tight.lockedAccounts} accounts`], ['second factor, 86% coverage', 'code', noFactor(base), `extra step in ${legit} sessions`], ['threshold and second factor', 'both', noFactor(tight), `${tight.blockedLegit} attempts, extra step`], ['pair check at password setting', 'design', 0, `${setting} accounts change password`], ]) console.log(`${a.padEnd(34)}${s(b, 16)}${s(c, 13)}${s(d, 30)}`); console.log(`\nsecond factor coverage ${ACCOUNTS.filter((a) => a.factor).length}/${ACCOUNTS.length};` + ` the ${ACCOUNTS.filter((a) => !a.factor).length} accounts outside coverage stay vulnerable at every setting\n`); console.log(`${'setting'.padEnd(24)}${s('account threshold cut', 23)}${s('address threshold cut', 23)}` + `${s('locked legit accounts', 23)}`); for (const [name, ae, de] of [['account 5', 5, I], ['address 50', I, 50], ['account 5 + address 50', 5, 50], ['account 3 + address 20', 3, 20]]) { const r = measure(K, ae, de); console.log(`${name.padEnd(24)}${s(r.legitByAccount, 23)}${s(r.legitByAddress, 23)}${s(r.lockedAccounts, 23)}`); } console.log(`\nlegit attempts from the shared egress address: ` + `${K.filter((r) => r[3] === 'legit' && r[2] === 'shared-egress').length}`); ``` ``` measure stage taken over legit cost no measure - 70 0 account 3 + address 20 threshold runtime 57 613 attempts, 315 accounts second factor, 86% coverage code 5 extra step in 2623 sessions threshold and second factor both 4 613 attempts, extra step pair check at password setting design 0 70 accounts change password second factor coverage 1639/1900; the 261 accounts outside coverage stay vulnerable at every setting setting account threshold cut address threshold cut locked legit accounts account 5 220 0 91 address 50 0 227 164 account 5 + address 50 138 226 194 account 3 + address 20 220 393 315 legit attempts from the shared egress address: 500 ``` The upper table answers the course's main question for this lesson. The runtime threshold blocked 53% of bad attempts and brought takeovers down from 70 to 57: the share of takeovers prevented is 19%. The reason lies not in the setting of the threshold but in its definition. The threshold counts **failures**; in stuffing, the successful attempt is the first attempt the account sees, and at that moment the counter is empty. The counter fills only after failures accumulate, so the threshold structurally carries the property of being late. The same takeover falls from 70 to 5 at the code stage with a second factor. Why the 5 is not zero is also written in the table: second factor coverage is 1,639 accounts, and the 261 accounts outside coverage are protected at no threshold setting. Coverage is this measure's only real setting. Its price is an extra step in 2,623 legitimate sessions; no session is cut, though some sessions are delayed. The measure at the design stage closes it at an even lower cost. When leaked pairs and common passwords are checked as the password is set, the success condition of both stuffing and spraying disappears: the taken over count is zero. Its price is 70 accounts changing their passwords, and this is a one-time event. For the same item, the runtime threshold cuts 613 legitimate attempts every day and locks 315 accounts; the check at design closes the item once. The lower table separates where the false block lands. The account threshold puts the cut on the account owner, and the address threshold puts it on everyone behind a network location. 500 legitimate attempts sit behind the shared egress address, and when the address threshold is lowered to 20, the attempts cut from that address rise to 393. The address threshold hands out punishment according to the size of the team behind the address rather than the targeted attacker. #### Summary - The three attack shapes separate on two axes: brute force, at 600 attempts per account, stops at the account threshold (596 blocked); spraying, at 142.5 attempts per address, stops at the address threshold (3,694 blocked); stuffing, at 1.0 per account and 10.0 per address, is under both. - The address threshold blocked zero attempts from stuffing. The 15–18 attempts seen in the lower table are an indirect catch that comes from the same accounts also being targeted by spraying. - 70 accounts were taken over in the run with no threshold, and 57 at the tightest setting. While blocked attempts rose to 5,512, takeovers prevented are only 13 (19%), because the threshold counts failures and in stuffing the successful attempt is the account's first attempt. - The false block lands in two different places: of the 91 accounts locked at account threshold 5, only 30 are subscribers who forgot their passwords, and the rest were locked because they were the target of the attack. All 227 attempts cut at address threshold 50 belong to the field crew at the shared egress address. - The same takeover falls from 70 to 5 at the code stage with a second factor (the remaining 5 from the 261 accounts outside coverage), and to 0 at design with a password setting check. The price at design is a one-time password change on 70 accounts; the price of the runtime threshold is 613 legitimate attempts cut every day. #### Next Step The thresholds of this lesson, and the rules and counters of the previous two lessons, all looked at requests or attempts one by one. What the three shared was the assumption that the defense looks at information arriving with the request. Yet there is one more decision made at runtime, given by the server and having nothing to do with the content of the request: the security headers sent along with the response. They stop no request; they determine what the browser is allowed to do, and when managed from a single place they cover all the applications behind them at once. The next lesson measures this layer not by the requests it catches but by **the number of applications it covers**, and puts the main question to the end left outside coverage. ### Security Headers Coverage Source: https://academia.sh/en/courses/runtime-security/security-headers-coverage Measuring with four separate counts the endpoints, applications, and requests covered by moving security headers out of the application and managing them at the edge: the daily requests carried by browser surface that does not pass through the edge, the cells configured but ineffective on endpoints with no browser surface, the number of legitimate pages a strict content security policy breaks, and the code-stage work needed before the policy can be tightened. The previous three lessons measured defenses that look at the request: the rule set looked at the shape of the request, the rate counter at the number of requests, and the thresholds at the account and the address. All three decided whether to stop a request or let it through. There is one more decision made at runtime, and it stops no request: the security headers sent along with the response determine what the browser is allowed to do. What the headers do was measured inside the application in an earlier course and is not repeated here. This lesson's measure is different: when header configuration is taken out of the application and moved to the edge layer in front of them, **how many applications and how many endpoints** are covered at once, and what is left outside coverage. This is a defense that produces no "caught requests" count; its measure is coverage, and when coverage is not counted, a configuration file gets confused with real protection. #### Where the Header Is Written The regional measurement network's production environment is modeled with `node`; no real server is brought up, and no real response is produced. - **AD16.** The fourteen endpoints, nine applications, six headers, and daily request counts belong to the fictional measurement network. Some of the endpoints pass through the shared edge layer, and some are called directly. - **AD17.** The "browser surface" field says whether the endpoint returns to a browser a document or a resource the browser loads. Security headers do something only on browser surface; written on an API, they change no behavior. - **AD18.** Management at the edge writes all six headers to every endpoint that passes through the edge. On an endpoint that does not pass through the edge, only the header the application wrote itself remains. - **AD19.** The content security policy is run in report-only mode: the resources that would be blocked are counted, not actually blocked. Whether the resources are legitimate is known in the model. - **AD20.** The "earlier closing stage" column was marked by hand for each header. ```js // endpoints.mjs — endpoints in production, six security headers, and page resources; a model // [endpoint, application, path, browser surface?, daily requests, headers the application wrote itself] export const ENDPOINTS = [ ['subscriber portal, page', 'subscriber portal', 'edge', true, 84000, 2], ['subscriber portal, API', 'subscriber portal', 'edge', false, 41000, 1], ['subscriber portal, authentication', 'subscriber portal', 'edge', true, 6800, 2], ['field app, page', 'field app', 'edge', true, 12000, 0], ['field app, file upload', 'field app', 'edge', false, 5400, 0], ['field app, mobile backend', 'field app', 'edge', false, 22000, 0], ['billing page', 'billing interface', 'edge', true, 9000, 3], ['report page', 'reporting', 'edge', true, 2600, 1], ['health probe', 'subscriber portal', 'edge', false, 1440, 0], ['admin interface', 'admin interface', 'direct', true, 900, 0], ['legacy subscriber portal', 'legacy portal', 'direct', true, 3100, 1], ['static asset host', 'asset host', 'direct', true, 62000, 0], ['measurement collection API', 'measurement collection', 'direct', false, 150000, 0], ['billing service API', 'billing service', 'direct', false, 38000, 0], ]; // [header, class of item it stops, earlier closing stage] export const HEADERS = [ ['content security policy', 'unauthorized script execution', 'code'], ['framing protection', 'page embedding', 'none'], ['transport layer enforcement', 'plaintext connection', 'none'], ['content type pinning', 'type sniffing', 'code'], ['referrer information policy', 'identifier leaked in transit', 'code'], ['browser capability restriction', 'unwanted device access', 'none'], ]; // number of headers written on an endpoint after edge management export const written = (e) => (e[2] === 'edge' ? HEADERS.length : e[5]); // a header does something only on browser surface export const effective = (e) => (e[3] ? written(e) : 0); // [page, resource, location, legitimate?, daily loads] export const RESOURCES = [ ['subscriber portal, page', 'app script', 'same-origin', true, 84000], ['subscriber portal, page', 'stylesheet', 'asset-host', true, 84000], ['subscriber portal, page', 'inline measurement script', 'inline', true, 84000], ['subscriber portal, page', 'extra script from a dependency', 'third-party', false, 84000], ['subscriber portal, authentication', 'form script', 'same-origin', true, 6800], ['subscriber portal, authentication', 'inline field validation', 'inline', true, 6800], ['field app, page', 'map component', 'asset-host', true, 12000], ['field app, page', 'inline status script', 'inline', true, 12000], ['billing page', 'table script', 'same-origin', true, 9000], ['billing page', 'font', 'asset-host', true, 9000], ['report page', 'charting library', 'asset-host', true, 2600], ['report page', 'injected inline script', 'inline', false, 2600], ]; export const POLICIES = [ ['same origin only', (r) => r[2] === 'same-origin'], ['plus asset host', (r) => ['same-origin', 'asset-host'].includes(r[2])], ['plus inline, blanket', (r) => ['same-origin', 'asset-host', 'inline'].includes(r[2])], ['plus inline, hashed', (r) => ['same-origin', 'asset-host'].includes(r[2]) || (r[2] === 'inline' && r[3])], ]; ``` #### How Coverage Is Counted ```js // coverage.mjs — endpoints, requests, and cells covered by managing headers at the edge import { ENDPOINTS, HEADERS, written, effective } from './endpoints.mjs'; const s = (x, n) => String(x).padStart(n); const T = (f) => ENDPOINTS.filter(f).reduce((t, e) => t + e[4], 0); const N = (f) => ENDPOINTS.filter(f).length; console.log(`${ENDPOINTS.length} endpoints, ${new Set(ENDPOINTS.map((e) => e[1])).size} applications, ${HEADERS.length} headers` + ` = ${ENDPOINTS.length * HEADERS.length} cells; ${T(() => true)} requests per day\n`); console.log(`${'endpoint'.padEnd(33)}${s('path', 10)}${s('browser', 10)}${s('daily', 8)}` + `${s('own written', 15)}${s('after edge', 16)}${s('effective', 11)}`); for (const e of ENDPOINTS) console.log(`${e[0].padEnd(33)}${s(e[2], 10)}${s(e[3] ? 'yes' : 'no', 10)}${s(e[4], 8)}` + `${s(e[5], 15)}${s(written(e), 16)}${s(effective(e), 11)}`); const after = ENDPOINTS.reduce((t, e) => t + written(e), 0), eff = ENDPOINTS.reduce((t, e) => t + effective(e), 0); console.log(`\n${'measure'.padEnd(38)}${s('before', 8)}${s('after edge', 16)}`); for (const [a, b, c] of [ ['written header cells', ENDPOINTS.reduce((t, e) => t + e[5], 0), after], ['effective cells on browser surface', ENDPOINTS.reduce((t, e) => t + (e[3] ? e[5] : 0), 0), eff], ['endpoints writing headers', N((e) => e[5] > 0), N((e) => written(e) > 0)], ]) console.log(`${a.padEnd(38)}${s(b, 8)}${s(c, 16)}`); console.log(`\n${'how coverage is counted'.padEnd(38)}${s('covered', 10)}${s('total', 8)}${s('%', 8)}`); for (const [a, b, c] of [ ['by endpoint count', N((e) => e[2] === 'edge'), ENDPOINTS.length], ['by request volume', T((e) => e[2] === 'edge'), T(() => true)], ['by request volume on browser surface', T((e) => e[2] === 'edge' && e[3]), T((e) => e[3])], ['by browser cell count', eff, N((e) => e[3]) * HEADERS.length], ]) console.log(`${a.padEnd(38)}${s(b, 10)}${s(c, 8)}${s((100 * b / c).toFixed(1), 8)}`); console.log(`\nbrowser surface left outside coverage (what bringing it to the edge would gain):`); for (const e of ENDPOINTS.filter((x) => x[3] && x[2] !== 'edge')) console.log(` ${e[0].padEnd(24)}${s(e[4], 8)} requests/day, ${s(HEADERS.length - e[5], 2)} cells gained`); console.log(`\nconfigured but ineffective cells (on endpoints without a browser surface): ${after - eff}`); ``` ``` 14 endpoints, 9 applications, 6 headers = 84 cells; 438240 requests per day endpoint path browser daily own written after edge effective subscriber portal, page edge yes 84000 2 6 6 subscriber portal, API edge no 41000 1 6 0 subscriber portal, authentication edge yes 6800 2 6 6 field app, page edge yes 12000 0 6 6 field app, file upload edge no 5400 0 6 0 field app, mobile backend edge no 22000 0 6 0 billing page edge yes 9000 3 6 6 report page edge yes 2600 1 6 6 health probe edge no 1440 0 6 0 admin interface direct yes 900 0 0 0 legacy subscriber portal direct yes 3100 1 1 1 static asset host direct yes 62000 0 0 0 measurement collection API direct no 150000 0 0 0 billing service API direct no 38000 0 0 0 measure before after edge written header cells 10 55 effective cells on browser surface 9 31 endpoints writing headers 6 10 how coverage is counted covered total % by endpoint count 9 14 64.3 by request volume 184240 438240 42.0 by request volume on browser surface 114400 180400 63.4 by browser cell count 31 48 64.6 browser surface left outside coverage (what bringing it to the edge would gain): admin interface 900 requests/day, 6 cells gained legacy subscriber portal 3100 requests/day, 5 cells gained static asset host 62000 requests/day, 6 cells gained configured but ineffective cells (on endpoints without a browser surface): 24 ``` The gain of edge management is written in the first summary table, and it is not small: written header cells rise from 10 to 55, so a single configuration change fills 45 cells at once. Inside the application, this work would have been done nine times in nine separate codebases, and five applications had never done it. This is the real advantage of the edge layer: coverage is taken out of decisions the application teams would each make separately. The second table gives the same coverage with four separate counts, and the four produce different numbers: 64.3% by endpoint count, 42.0% by request volume, 63.4% by request volume on browser surface, and 64.6% by browser cell count. Four different coverage rates can be written for the same layer, and which one is reported is a choice. The request volume count comes out low because the measurement collection API carries 150,000 requests a day, yet that endpoint is not called from a browser; on that endpoint nothing changes whether a header is present or not. So the meaningful count is the third: how much of the request volume on browser surface is covered. The lower list gives the real finding. Three endpoints are left outside coverage, and one of them is the static asset host, which carries 62,000 requests a day. Because it does not appear as an "application" in the application inventory, it is never counted in coverage tables; yet the browser loads resources from it, and those resources carry no header. The admin interface and the legacy subscriber portal add up to 4,000 requests, which is 6% of the volume outside coverage. This list sets the order of the work of widening coverage, and an endpoint that never entered the inventory is the largest item in line. The last line counts the flaw in the opposite direction: 24 cells are configured but ineffective. All six headers are written on the API endpoints, they show green in the audit list, and they change no behavior. The number of configurations is not a measure of protection. #### The Page a Strict Policy Breaks Coverage produces no false positive; the **content** of the header does. When the content security policy is tightened, what gets blocked is not only the unauthorized script but the page's own resources. ```js // policy.mjs — run the content security policy in report-only mode and count the legitimate resources it breaks import { ENDPOINTS, HEADERS, RESOURCES, POLICIES, written } from './endpoints.mjs'; const s = (x, n) => String(x).padStart(n); const covered = new Set(ENDPOINTS.filter((e) => e[3] && written(e) === HEADERS.length).map((e) => e[0])); const R = RESOURCES.filter((r) => covered.has(r[0])); console.log(`policy applies only on the ${covered.size} browser endpoints managed from the edge;` + ` these endpoints hold ${R.length} resources\n`); console.log(`${'policy'.padEnd(24)}${s('blocked legit', 18)}${s('broken pages', 15)}` + `${s('blocked bad', 17)}${s('passed bad', 12)}`); for (const [name, allow] of POLICIES) { const blocked = R.filter((r) => !allow(r)), m = blocked.filter((r) => r[3]); console.log(`${name.padEnd(24)}${s(m.length, 18)}${s(new Set(m.map((r) => r[0])).size, 15)}` + `${s(blocked.filter((r) => !r[3]).length, 17)}${s(R.filter((r) => allow(r) && !r[3]).length, 12)}`); } const inline = R.filter((r) => r[2] === 'inline' && r[3]); const allow2 = POLICIES[1][1], afterCode = R.filter((r) => r[2] !== 'inline' || !r[3]); console.log(`\ninline scripts that must be written to the hash list: ${inline.length}`); console.log(`with inline scripts removed at the code stage, the "plus asset host" policy:` + ` blocked legit ${afterCode.filter((r) => r[3] && !allow2(r)).length},` + ` passed bad ${afterCode.filter((r) => !r[3] && allow2(r)).length}`); console.log(`\n${'header'.padEnd(33)}${'item stopped'.padEnd(32)}${s('earlier stage', 15)}`); for (const [name, item, stage] of HEADERS) console.log(`${name.padEnd(33)}${item.padEnd(32)}${s(stage, 15)}`); const none = HEADERS.filter((h) => h[2] === 'none'); console.log(`\nheaders that can be closed only at this layer ${none.length}/${HEADERS.length};` + ` endpoints where they are effective ${ENDPOINTS.filter((e) => e[3] && written(e) === HEADERS.length).length},` + ` ${ENDPOINTS.filter((e) => e[3] && written(e) === HEADERS.length).reduce((t, e) => t + e[4], 0)} requests per day`); ``` ``` policy applies only on the 5 browser endpoints managed from the edge; these endpoints hold 12 resources policy blocked legit broken pages blocked bad passed bad same origin only 7 5 2 0 plus asset host 3 3 2 0 plus inline, blanket 0 0 1 1 plus inline, hashed 0 0 2 0 inline scripts that must be written to the hash list: 3 with inline scripts removed at the code stage, the "plus asset host" policy: blocked legit 0, passed bad 0 header item stopped earlier stage content security policy unauthorized script execution code framing protection page embedding none transport layer enforcement plaintext connection none content type pinning type sniffing code referrer information policy identifier leaked in transit code browser capability restriction unwanted device access none headers that can be closed only at this layer 3/6; endpoints where they are effective 5, 114400 requests per day ``` The four rows are a curve of strictness. The strictest policy blocks two bad resources, and alongside them it blocks seven legitimate resources, breaking all five pages; every browser endpoint covered is broken. Once the asset host is allowed, the broken pages fall to three, and the bad resources are still two. The third row is the escape route seen most often in production: inline scripts are allowed wholesale, broken pages fall to zero, and the injected inline script on the report page passes with the same permission. This setting, which zeroes the broken pages, drops the bad resources the policy blocks from two to one. The fourth row gives both at once: when inline scripts are allowed through a hash list, broken pages are zero and the bad resources blocked are two. The price is the list itself. Three inline scripts have to be written to the hash list, and if the list is not updated when one of these scripts changes, the page breaks. The policy is no longer a header value but an inventory that has to be checked at every release of the code, a burden of the same kind as the exclusion list in the first lesson. One line below answers the course's question. When inline scripts are removed at the code stage, the "plus asset host" policy blocks zero legitimate resources and lets zero bad resources through: no hash list is needed. The header gives the same result only in return for continuous maintenance. The lower table spreads this distinction across the six headers: the item stopped by three of them can be closed at the code stage, and three have no earlier stage. These three determine the browser's behavior and cannot be stated from application code; this is the share only this layer still covers, and it holds on five endpoints, at 114,400 requests a day. #### Summary - Managing headers at the edge raised written cells from 10 to 55 and endpoints writing headers from 6 to 10; coverage no longer depends on the separate decisions of the application teams. - The same coverage gives four different rates with four separate counts: 64.3% by endpoint count, 42.0% by request volume, 63.4% by request volume on browser surface, and 64.6% by browser cell count. The meaningful one is the third; the others also count endpoints the browser never sees. - Three browser endpoints remained outside coverage, and of their 66,000 requests, 62,000 belong to the static asset host that does not appear as an "application" in the inventory. On the other side, 24 cells are configured but ineffective: headers are written on API endpoints and do nothing. - At its strictest setting, the content security policy blocked 2 bad resources and also blocked 7 legitimate resources, breaking all 5 pages. Allowing inline scripts wholesale zeroed the breakage but dropped the bad resources blocked from 2 to 1; the hash list gave both (0 broken, 2 blocked) and in return brought a maintenance inventory of 3 lines. - When inline scripts are removed at the code stage, the policy gives 0 broken and 0 bad resources passed, without a hash list. The item stopped by three of the six headers closes at the code stage; the three headers that can be closed only at this layer hold on 5 endpoints, at 114,400 requests a day. #### Next Step This lesson's measure was coverage, and coverage was provided by the values sent along with a response. The previous three lessons looked at the shape, the number, and the identity of the request. The limit shared by all four is the same: the decision is made on a single request or a single response. Yet the item that no setting has been able to stop since the first lesson, the out-of-workflow bulk download, looked flawless in each individual request; the only thing wrong was **the shape the requests of the same session drew together**. The next lesson takes the defense out of the request and looks at the behavior of the session, flags the session that deviates from the baseline with an anomaly score, and puts the main question there: how many minutes after the deviation began. ### Application-Level Anomaly Detection Source: https://academia.sh/en/courses/runtime-security/application-level-anomaly-detection Running seven sessions scored against a role baseline through a forty-five-minute behavior log and comparing the flagging threshold across four settings: the delay between the minute the deviation began and the minute the flag was raised, the number of records that left during that delay, the score gap between the slowly deviating insider threat session and the legitimate analyst session in the billing period, and how many records earlier the same leak could have been closed at the design stage. The previous four lessons looked at the request: the rule set at the shape, the rate counter at the number, the thresholds at the account and the address, the headers at the response. All four made their decision on a single request or a single response. The item that no setting has been able to stop since the first lesson slipped through for exactly this reason: the requests of the out-of-workflow bulk download were flawless one by one, authenticated, and authorized. The only thing wrong was **the shape the requests of the same session drew together**. This lesson moves the defense out of the request and into the behavior of the session. The measure changes too: instead of requests stopped, **delay** is counted: how many minutes after the deviation began the flag is raised, and how much data leaves during those minutes. This layer does not block a request; it flags a session late or early. #### Baseline and Anomaly Score Everything below is a model built with `node`; no real session log is read, and no method of taking over a session is written. Taken-over sessions are represented only by behavior vectors: requests per minute, unique endpoints, records, and the number of first-seen endpoints. - **AD21.** The seven sessions, three role baselines, and forty-five-minute window belong to the fictional measurement network. The baseline is given as the mean of thirty days of normal behavior. - **AD22.** The score derives from the ratio of each feature to its baseline, and the deviation share per feature is capped at two units. Without the cap, a single outlier would flag the session instantly; real scoring systems have the same kind of saturation. - **AD23.** The flag is raised when the score total of the last five minutes crosses the threshold. The stacking window is fixed; the only thing that changes is the threshold. - **AD24.** Two of the legitimate sessions are deliberately unusual: the analyst in the billing period has a record volume rising to two and a half times the baseline, and the new-hire field technician has a high number of first-seen endpoints. Both are candidates for a false flag. - **AD25.** The "earlier closing stage" column was marked by hand for each bad session. ```js // behavior.mjs — forty-five-minute session behavior log and anomaly scoring; a model export const DURATION = 45; export const FEATURES = ['requests/min', 'unique endpoints', 'records/min', 'first-seen endpoints']; export const WEIGHT = [1, 1.5, 2, 2.5]; export const CAP = 2; // cap on the deviation share per feature // role baseline: mean taken from thirty days of normal behavior export const BASELINE = { subscriber: [4, 3, 12, 0.2], field: [9, 5, 40, 0.4], analyst: [6, 7, 300, 0.3], }; // [session, role, true class, deviation minute, deviation multipliers, minutes to full multiplier, earlier stage] export const SESSIONS = [ ['subscriber-3391', 'subscriber', 'legit', Infinity, [1, 1, 1, 1], 1, '-'], ['field-12, shift', 'field', 'legit', Infinity, [1, 1, 1, 1], 1, '-'], ['analyst-4, billing period', 'analyst', 'legit', 0, [1.2, 1.1, 2.6, 1.3], 4, '-'], ['field-58, new hire', 'field', 'legit', 0, [0.8, 1.8, 0.9, 6.0], 3, '-'], ['subscriber-8842, taken over', 'subscriber', 'bad', 12, [3.5, 2.5, 22, 5], 3, 'code'], ['analyst-9, insider threat', 'analyst', 'bad', 8, [1.4, 1.3, 2.2, 1.5], 12, 'design'], ['field-31, stolen token', 'field', 'bad', 20, [2.2, 4.4, 3.5, 9], 2, 'none'], ]; // the minute-by-minute feature vector of a session export function trace(o) { const [, role, , onset, mult, ramp] = o, base = BASELINE[role], d = []; for (let t = 0; t < DURATION; t++) { const f = t < onset ? 0 : Math.min(1, (t - onset + 1) / ramp); d.push(base.map((b, i) => b * (1 + (mult[i] - 1) * f))); } return d; } export const score = (v, role) => v.reduce((t, x, i) => t + WEIGHT[i] * Math.min(CAP, Math.max(0, x / BASELINE[role][i] - 1)), 0); export const stack = (o, win = 5) => { const p = trace(o).map((v) => score(v, o[1])); return p.map((_, t) => p.slice(Math.max(0, t - win + 1), t + 1).reduce((x, y) => x + y, 0)); }; // the session is flagged when the score total of the last `win` minutes crosses the threshold export function flag(o, threshold, win = 5) { const d = trace(o), p = d.map((v) => score(v, o[1])); for (let t = 0; t < DURATION; t++) { const sum = p.slice(Math.max(0, t - win + 1), t + 1).reduce((x, y) => x + y, 0); if (sum >= threshold) { const start = Number.isFinite(o[3]) ? o[3] : 0; const records = d.slice(start, t + 1).reduce((x, v) => x + v[2], 0); return { minute: t, delay: t - start, records: Math.round(records) }; } } return null; } ``` #### With How Many Minutes of Delay ```js // run-flagging.mjs — the same behavior log, with the flagging threshold as the only change import { DURATION, SESSIONS, trace, score, stack, flag } from './behavior.mjs'; const s = (x, n) => String(x).padStart(n); console.log(`${SESSIONS.length} sessions, ${DURATION} minutes, 3 role baselines\n`); console.log(`${'session'.padEnd(30)}${s('role', 11)}${s('class', 7)}${s('deviation min', 15)}` + `${s('peak minute score', 20)}${s('peak stack', 17)}${s('total records', 14)}${s('earlier stage', 15)}`); for (const o of SESSIONS) { const d = trace(o), p = d.map((v) => score(v, o[1])); console.log(`${o[0].padEnd(30)}${s(o[1], 11)}${s(o[2], 7)}` + `${s(Number.isFinite(o[3]) ? o[3] : '-', 15)}${s(Math.max(...p).toFixed(2), 20)}` + `${s(Math.max(...stack(o)).toFixed(1), 17)}` + `${s(Math.round(d.reduce((x, v) => x + v[2], 0)), 14)}${s(o[6], 15)}`); } console.log(`\n${'threshold'.padEnd(11)}${s('flagged bad', 18)}${s('missed bad', 16)}` + `${s('falsely flagged', 20)}${s('mean delay min', 21)}${s('records out during delay', 26)}`); const B = SESSIONS.filter((o) => o[2] === 'bad'), L = SESSIONS.filter((o) => o[2] === 'legit'); for (const threshold of [40, 30, 20, 12]) { const fb = B.filter((o) => flag(o, threshold)), fl = L.filter((o) => flag(o, threshold)); const g = fb.map((o) => flag(o, threshold)); console.log(`${String(threshold).padEnd(11)}${s(`${fb.length}/${B.length}`, 18)}` + `${s(B.length - fb.length, 16)}${s(`${fl.length}/${L.length}`, 20)}` + `${s((g.reduce((x, r) => x + r.delay, 0) / g.length).toFixed(1), 21)}` + `${s(g.reduce((x, r) => x + r.records, 0), 26)}`); } for (const threshold of [30, 20]) { console.log(`\nthreshold ${threshold}, session by session:`); console.log(` ${'session'.padEnd(30)}${s('class', 7)}${s('flag min', 15)}${s('delay min', 12)}` + `${s('records out during delay', 26)}`); for (const o of SESSIONS) { const r = flag(o, threshold); console.log(` ${o[0].padEnd(30)}${s(o[2], 7)}${s(r ? r.minute : 'not flagged', 15)}` + `${s(r ? r.delay : '-', 12)}${s(r && o[2] === 'bad' ? r.records : '-', 26)}`); } } ``` ``` 7 sessions, 45 minutes, 3 role baselines session role class deviation min peak minute score peak stack total records earlier stage subscriber-3391 subscriber legit - 0.00 0.0 540 - field-12, shift field legit - 0.00 0.0 1800 - analyst-4, billing period analyst legit 0 4.30 21.5 34380 - field-58, new hire field legit 0 6.20 31.0 1624 - subscriber-8842, taken over subscriber bad 12 13.25 66.3 8604 code analyst-9, insider threat analyst bad 8 4.50 22.5 24840 design field-31, stolen token field bad 20 13.20 66.0 4250 none threshold flagged bad missed bad falsely flagged mean delay min records out during delay 40 2/3 1 0/4 3.0 1314 30 2/3 1 1/4 2.0 910 20 3/3 0 2/4 4.7 7106 12 3/3 0 2/4 3.3 4556 threshold 30, session by session: session class flag min delay min records out during delay subscriber-3391 legit not flagged - - field-12, shift legit not flagged - - analyst-4, billing period legit not flagged - - field-58, new hire legit 5 5 - subscriber-8842, taken over bad 14 2 540 analyst-9, insider threat bad not flagged - - field-31, stolen token bad 22 2 370 threshold 20, session by session: session class flag min delay min records out during delay subscriber-3391 legit not flagged - - field-12, shift legit not flagged - - analyst-4, billing period legit 6 6 - field-58, new hire legit 3 3 - subscriber-8842, taken over bad 13 1 276 analyst-9, insider threat bad 20 12 6600 field-31, stolen token bad 21 1 230 ``` The two rows of the first table should be read side by side. The legitimate analyst in the billing period has a peak five-minute stack of 21.5, and the insider threat session has 22.5. The gap is one unit, and everything the scoring sees is made of this gap. Both are in the same role, working at several times the baseline volume; one is producing an end-of-period report, the other is exfiltrating. The scoring cannot separate these two sessions, because the difference is not in the behavior but in **the intent**: the same limit as the rule set of the first lesson, one layer up. The threshold table gives the consequence. At threshold 40, no legitimate session is flagged and the mean delay is 3 minutes, but the insider threat is never caught: its stack never reaches 40. At threshold 30 the table looks even better, with the delay falling to 2 minutes and the records leaving during the delay to 910, but the insider threat escapes again and the new-hire field technician is falsely flagged. The first setting that catches the insider threat is 20, and there three of the four numbers get worse at once: falsely flagged sessions rise to 2, the mean delay to 4.7 minutes, and the records leaving during the delay to 7,106. The two session-by-session lists show where the delay comes from. The taken-over subscriber session and the stolen token session deviate fast and are flagged in one or two minutes; the records they let out during the delay lie between 230 and 540. The insider threat, on the other hand, is flagged after 12 minutes, and in that time it downloads 6,600 records, 93% of the total delay cost of the three bad sessions. The slowly deviating session is both the one caught last and the one that takes out the most data; delay and damage grow in the same direction. #### Where the Delay Belongs ```js // stage.mjs — at which earlier stage, and by how many records, the same leak would be closed sooner import { DURATION, SESSIONS, BASELINE, trace, flag } from './behavior.mjs'; const s = (x, n) => String(x).padStart(n); const find = (name) => SESSIONS.find((o) => o[0].startsWith(name)); const insider = find('analyst-9'), period = find('analyst-4'), T = BASELINE.analyst[2]; const cum = (o) => trace(o).reduce((a, v) => (a.push((a.at(-1) || 0) + v[2]), a), []); const excess = (o, t) => Math.round(cum(o)[t] - T * (t + 1)); const capAt = (o, cap) => { const k = cum(o), t = k.findIndex((x) => x >= cap); return t < 0 ? { label: 'not cut', t: DURATION - 1 } : { label: `min ${t}`, t }; }; console.log(`analyst role baseline ${T} records/min; ${DURATION} minutes of normal work ${T * DURATION} records`); console.log(`insider session downloaded ${Math.round(cum(insider).at(-1))} records in total,` + ` above baseline ${excess(insider, DURATION - 1)}\n`); console.log(`${'measure'.padEnd(34)}${s('stage', 16)}${s('above-baseline records', 24)}${s('legit cost', 34)}`); const e20 = flag(insider, 20), e40 = flag(insider, 40); for (const [a, b, c, d] of [ ['no measure', '-', excess(insider, DURATION - 1), '0'], ['anomaly threshold 40', 'runtime', e40 ? excess(insider, e40.minute) : excess(insider, DURATION - 1), 'no false flag'], ['anomaly threshold 20', 'runtime', excess(insider, e20.minute), `billing analyst flagged at min ${flag(period, 20).minute}`], ['per-role record cap 18000', 'design', excess(insider, capAt(insider, 18000).t), `billing analyst cut at ${capAt(period, 18000).label}`], ['bulk export moved to queue job', 'design', 0, 'bulk job goes to approval'], ]) console.log(`${a.padEnd(34)}${s(b, 16)}${s(c, 24)}${s(d, 34)}`); console.log(`\n${'cap'.padEnd(8)}${s('insider cut', 13)}${s('above-baseline records', 24)}` + `${s('billing analyst cut', 21)}${s('legit records cut', 19)}`); for (const cap of [13500, 18000, 25000]) { const a = capAt(insider, cap), b = capAt(period, cap); console.log(`${String(cap).padEnd(8)}${s(a.label, 13)}${s(excess(insider, a.t), 24)}${s(b.label, 21)}` + `${s(Math.round(cum(period).at(-1) - cum(period)[b.t]), 19)}`); } console.log(`\n${'bad session'.padEnd(30)}${s('threshold 20 delay', 20)}${s('earlier stage', 15)}${s('closes there', 15)}`); const CLOSES = { code: 'second factor', design: 'workflow', none: 'none' }; for (const o of SESSIONS.filter((x) => x[2] === 'bad')) console.log(`${o[0].padEnd(30)}${s(`${flag(o, 20).delay} min`, 20)}${s(o[6], 15)}${s(CLOSES[o[6]], 15)}`); ``` ``` analyst role baseline 300 records/min; 45 minutes of normal work 13500 records insider session downloaded 24840 records in total, above baseline 11340 measure stage above-baseline records legit cost no measure - 11340 0 anomaly threshold 40 runtime 11340 no false flag anomaly threshold 20 runtime 2700 billing analyst flagged at min 6 per-role record cap 18000 design 7740 billing analyst cut at min 23 bulk export moved to queue job design 0 bulk job goes to approval cap insider cut above-baseline records billing analyst cut legit records cut 13500 min 27 5220 min 18 20280 18000 min 34 7740 min 23 16380 25000 not cut 11340 min 32 9360 bad session threshold 20 delay earlier stage closes there subscriber-8842, taken over 1 min code second factor analyst-9, insider threat 12 min design workflow field-31, stolen token 1 min none none ``` The top table shows that a crude design measure can be **worse** than runtime scoring. A per-role record cap of 18,000 lowers the records the insider threat takes out above baseline only from 11,340 to 7,740, while the anomaly threshold of 20 stops the same item at 2,700. The middle table explains why the cap stays so weak: the cap would have to be lowered enough to cut the insider threat early, but a cap of 13,500 cuts the billing-period analyst at minute 18 and leaves 20,280 legitimate records outside. A fixed number cannot be tuned in a role whose workload is seasonal. What can be tuned is the workflow itself. When bulk export is taken off the interactive endpoint and moved to an approved queue job, the number of records a session can take out above baseline is zero: the insider threat session cannot produce that volume, because the interactive endpoint does not have that capability. The legitimate cost is also smaller: the billing analyst's bulk job is not cut, it goes to approval and is delayed. This is a decision given once in the design; it does not require adjusting a threshold every day. The bottom table separates the three bad sessions. The taken-over subscriber session is caught with a one-minute delay, but there was already a cheaper option: the third lesson had lowered the same takeover from seventy to five with the second factor. The insider threat is caught with a 12-minute delay, and the right place is the workflow. For the stolen token session, there is no earlier stage: the token is valid, authentication is successful, and no design or code decision invalidates a valid credential. The share only this layer can cover is that single row, and even that row carries one minute of delay. #### Summary - The legitimate analyst in the billing period has a peak five-minute stack of 21.5, and the insider threat session has 22.5. The difference the scoring sees between the two sessions is one unit; the separation lies not in the behavior but in the intent. - Thresholds 40 and 30 flag no legitimate session or only one, and miss the insider threat. The first setting that catches the insider threat is 20, and there falsely flagged sessions rise to 2/4, the mean delay to 4.7 minutes, and the records leaving during the delay to 7,106. - The two fast-deviating sessions were flagged in 1-2 minutes and let out 230-540 records during the delay. The slowly deviating insider threat was flagged in 12 minutes and let out 6,600 records: 93% of the total delay cost. Delay and damage grow in the same direction. - A crude design measure can be worse than runtime: a record cap of 18,000 stopped the above-baseline leak at 7,740, while anomaly threshold 20 stopped it at 2,700. A cap of 13,500 cuts the insider threat at minute 27, but it also cuts 20,280 legitimate records of the billing analyst. - Moving bulk export to an approved queue job zeroes the above-baseline records and does not cut the legitimate work, only delays it. Of the three bad sessions, one closes at the code stage (second factor) and one at the design stage (workflow); for the stolen token session, there is no earlier stage. #### Next Step This topic counted in five lessons what is caught at the request layer and what is falsely blocked: the rule set, the rate limit, the account and address thresholds, header coverage, and session behavior. In each lesson a class of items was written as "not here, it would have been closed earlier", and in each lesson there was also one of the opposite kind: a class that can be closed only here. This lesson's stolen token session is exactly of that class, and the question it leaves goes beyond the whole topic. That session was flagged, but what required it to be flagged was that the token **was still valid**. The defenses of all five lessons rest on the same assumption: the keys, tokens, and credentials behind them are valid, and as long as they stay valid, these layers have to look at behavior. Where those credentials are kept, who can read them, and for how long they stay valid was never asked. The next topic asks this, and its first measure is lifetime: the time a secret stays valid is itself the window it leaves open. ## Secrets Management How the keys and credentials an application carries are held at runtime is measured in numbers: the secret store's access matrix and the items that least privilege closes, the startup dependency the store opens as a single point, how many times a short-lived credential narrows the exposure window and the request load it brings in return, the minimum overlap during which two secrets are valid together in key rotation and the requests a wrong ordering cuts, the static secrets that workload identity eliminates and the external-system items that remain, and the minutes from detection to revocation in the leak-response flow and the calls made with the secret in that time. ### Secret Store Architecture Source: https://academia.sh/en/courses/runtime-security/secret-store-architecture The secret store is measured in terms of access control: the path-prefix policy opens 79 (app, secret) items to nine apps, the 90-day read log says 31, and 48 items sit there never used. No app reads three of the secrets; six apps can reach the widest data secret. The price of centralization is a single point — none of the nine apps can start without the store, and 41 start attempts a day pass through that one point. The previous lesson closed the request layer: it counted what the web application firewall and behavior scoring catch, what they miss, and what they cut wrongly. Behind all of those layers stood one silent assumption — that the keys the application carries to connect to the data store, the queue, and the external gateways are **in place and held only by the application itself**. This lesson removes that assumption and measures the place where the keys sit, the **secret store**, along a single axis: who can reach what. Where a secret is kept, how many copies of it exist, and how a central store reduces the copies were measured earlier and are not repeated here. The question here begins **after** the store is built: the store exists, the secrets are in it, and a policy has been written for each app. How many items does that policy allow, how many do the apps actually read, and how much does the difference expose from a compromised app. - **SM1.** The production environment of the subscriber portal and the field app is **fictional**: nine apps, twenty secrets, and one secret store. What is measured is a **model** written in `node`. - **SM2.** The model holds **no real secret value**; it carries only secret paths, app names, and numbers. A secret path has the form `domain/item` and the policy is written with a **path prefix**. - **SM3.** A policy rule opens every secret path that starts with it. The `data/` rule gives all five data secrets at once; a rule that spells out a full path gives one secret. - **SM4.** The read log is read in two windows: the last thirty days and the last ninety days. The ninety-day window counts as real need in this model; in practice no window shows the whole of the need. - **SM5.** There is no randomness; every number is derived from the model tables and the output is the same from run to run. Restart counts are fictional units. - **SM6.** The store's own outage is modeled as single-valued: the store is either up or down. Partial degradation and a replicated store are not modeled; replication reduces the outage item, not the access item. #### Access Matrix ```js // store.mjs — secret store inventory, policy rules, and read log. A MODEL. // The production environment of the subscriber portal and field app is FICTIONAL; nine apps, one store. // THERE ARE NO REAL SECRETS: only secret PATHS, app names, and numbers are modeled. export const SECRETS = Object.entries({ data: ["portal-primary", "portal-read", "field-primary", "billing-primary", "archive"], queue: ["measurement-intake", "notification-queue", "work-order-queue"], external: ["payment-gateway", "map-service", "sms-gateway", "email-gateway", "identity-provider"], signing: ["session-signing", "document-signing", "internal-call-signing"], infra: ["object-store", "cache", "metrics-write", "log-write"], }).flatMap(([d, l]) => l.map((k) => `${d}/${k}`)); // rules: the path prefixes the policy grants. reads: secrets in the read log of the last 30 days. // format/life: how the secret is obtained. restarts: restarts per day (deploys + scaling). export const APPS = { "portal-frontend": { rules: ["external/identity-provider", "signing/"], format: "at-startup", life: 0, restarts: 6, reads: ["signing/session-signing"] }, "portal-api": { rules: ["data/", "signing/", "infra/"], format: "cached", life: 60, restarts: 12, reads: ["data/portal-primary", "data/portal-read", "signing/session-signing", "signing/internal-call-signing", "infra/cache"] }, "subscriber-signup": { rules: ["data/", "external/"], format: "at-startup", life: 0, restarts: 3, reads: ["data/portal-primary", "external/email-gateway", "external/sms-gateway"] }, "billing-service": { rules: ["data/", "external/", "signing/document-signing"], format: "cached", life: 240, restarts: 2, reads: ["data/billing-primary", "data/portal-read", "external/payment-gateway", "signing/document-signing"] }, "payment-bridge": { rules: ["external/"], format: "per-request", life: -1, restarts: 2, reads: ["external/payment-gateway"] }, "field-api": { rules: ["data/", "queue/", "external/map-service", "signing/"], format: "cached", life: 60, restarts: 8, reads: ["data/field-primary", "queue/work-order-queue", "external/map-service", "signing/internal-call-signing"] }, "work-order": { rules: ["data/", "queue/"], format: "at-startup", life: 0, restarts: 4, reads: ["data/field-primary", "queue/work-order-queue", "queue/notification-queue"] }, "notification": { rules: ["queue/", "external/"], format: "cached", life: 15, restarts: 3, reads: ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"] }, "report-generator": { rules: ["data/", "infra/"], format: "at-startup", life: 0, restarts: 1, reads: ["data/portal-read", "data/archive", "infra/object-store"] }, }; export const NAMES = Object.keys(APPS); // Legitimate reads that never show in the 30-day window but do show in 90 days. export const RARE = [["billing-service", "data/archive"], ["work-order", "external/map-service"], ["portal-api", "infra/log-write"], ["report-generator", "external/email-gateway"]]; export const reaches = (a, s) => APPS[a].rules.some((k) => s.startsWith(k)); export const reads90 = (a) => [...APPS[a].reads, ...RARE.filter((n) => n[0] === a).map((n) => n[1])]; export const sum = (f) => NAMES.reduce((t, a) => t + f(a), 0); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([18, 8, 14, 12, 12, 11]); console.log(`model: ${SECRETS.length} secrets, ${NAMES.length} apps, 1 store (fictional production environment)\n`); console.log(row(["app", "rules", "can reach", "30d reads", "90d reads", "excess"])); for (const a of NAMES) { const e = SECRETS.filter((s) => reaches(a, s)).length; console.log(row([a, APPS[a].rules.length, e, APPS[a].reads.length, reads90(a).length, e - reads90(a).length])); } const E = sum((a) => SECRETS.filter((s) => reaches(a, s)).length); const R30 = sum((a) => APPS[a].reads.length), R90 = sum((a) => reads90(a).length); console.log(row(["total", sum((a) => APPS[a].rules.length), E, R30, R90, E - R90])); const bySecret = SECRETS.map((s) => [s, NAMES.filter((a) => reaches(a, s)).length, NAMES.filter((a) => reads90(a).includes(s)).length]).sort((x, y) => y[1] - x[1]); console.log("\nwidest four secrets (how many apps can reach / how many actually read):"); for (const [s, e, o] of bySecret.slice(0, 4)) console.log(` ${s.padEnd(24)} ${e} / ${o}`); const none = bySecret.filter((t) => t[2] === 0), shared = bySecret.filter((t) => t[2] > 1); console.log(`secrets no app reads: ${none.length} (${none.map((t) => t[0]).join(", ")})`); console.log(`secrets read by more than one app: ${shared.length} / ${SECRETS.length}; ` + `when one of them changes, on average ` + `${(shared.reduce((t, x) => t + x[2], 0) / shared.length).toFixed(1)} apps are affected`); console.log(`\nsecrets exposed by a compromised app — with the policy on average ` + `${(E / NAMES.length).toFixed(1)}, widest ${Math.max(...NAMES.map((a) => SECRETS.filter((s) => reaches(a, s)).length))}; with least privilege on average ` + `${(R90 / NAMES.length).toFixed(1)}, widest ${Math.max(...NAMES.map((a) => reads90(a).length))}`); console.log(`least-privilege policy: ${R90} lines (today ${sum((a) => APPS[a].rules.length)}), ` + `items closed ${E - R90}; if written from 30 days of observation it would cut ${R90 - R30} legitimate reads`); } ``` ``` model: 20 secrets, 9 apps, 1 store (fictional production environment) app rules can reach 30d reads 90d reads excess portal-frontend 2 4 1 1 3 portal-api 3 12 5 6 6 subscriber-signup 2 10 3 3 7 billing-service 3 11 4 5 6 payment-bridge 1 5 1 1 4 field-api 4 12 4 4 8 work-order 2 8 3 4 4 notification 2 8 3 3 5 report-generator 2 9 3 4 5 total 21 79 27 31 48 widest four secrets (how many apps can reach / how many actually read): data/portal-primary 6 / 2 data/portal-read 6 / 3 data/field-primary 6 / 2 data/billing-primary 6 / 1 secrets no app reads: 3 (external/identity-provider, queue/measurement-intake, infra/metrics-write) secrets read by more than one app: 12 / 20; when one of them changes, on average 2.2 apps are affected secrets exposed by a compromised app — with the policy on average 8.8, widest 12; with least privilege on average 3.4, widest 6 least-privilege policy: 31 lines (today 21), items closed 48; if written from 30 days of observation it would cut 4 legitimate reads ``` Twenty-one rule lines open seventy-nine `(app, secret)` items. The items that appear in the ninety-day read log number thirty-one. The **forty-eight items** in between, sixty-one percent of the access opened, are granted on secrets that no app ever touched. This gap does not come from laziness but from the shape of the rule: writing `data/` is one line and gives five secrets; spelling out which of the five secrets is really needed takes five lines. Looking row by row, the two most expensive items sit in two different apps. `field-api` can reach twelve secrets with four rule lines and reads four of them; eight items are unused grants. `portal-api` reaches twelve secrets and reads six. The largest single source of unused grants is the `data/` prefix: **six apps** can reach each of the five data secrets, yet the most-read one is read by three apps and the least-read one by a single app. Six apps can get to the primary credential of the billing database; only one app reads it. Three secrets do not appear in the list at all: the identity provider secret, the key of the measurement intake queue, and the metrics write key were not read by any app for ninety days. These are the cheapest items in the access matrix — a secret nobody reads is closed without cutting anyone. The next row reads the matrix in the opposite direction and gives the one number in this lesson that reaches beyond it: twelve of the twenty secrets are read by more than one app, by 2.2 apps on average. Even if the access matrix is narrowed, this sharing remains — because sharing comes not from the policy but from the architecture; if two apps write to the same database, they read the same credential. When any one of these twelve secrets is changed, on average two apps must move to the new value at the same time, and how that move is made is not a question access control can solve. The last two rows give the verdict. A compromised app opens 8.8 secrets on average under the current policy, twelve in the worst case. Dropping to least privilege takes the average to 3.4 and the worst case to six. The cost is in the same row: the policy grows from twenty-one lines to thirty-one. **Narrowing does not shorten the policy, it lengthens it.** And if the narrowing is written from thirty days of observation, it cuts four legitimate reads — the year-end archive read, the bulk route recalculation, the log write in diagnostic mode, and the monthly summary delivery. None of the four ever shows in the thirty-day window; when they are cut, they are noticed not at deploy time but on the first day they run. #### The Single Point of Centralization ```js // outage.mjs — the single-point cost of centralization: who stops and who cannot start when the store goes down import { NAMES, APPS, sum, gen } from "./store.mjs"; const row = gen([14, 14, 17, 16, 20]); const RESTARTS = sum((a) => APPS[a].restarts); // daily restarts (fictional) const immediate = NAMES.filter((a) => APPS[a].format === "per-request"); const expired = (D) => NAMES.filter((a) => APPS[a].format === "cached" && APPS[a].life < D); console.log(`daily restarts ${RESTARTS} (deploys + scaling); apps that can start without the ` + `store 0 / ${NAMES.length}\n`); console.log(row(["outage (min)", "down at once", "cache expired", "apps down", "failed starts"])); for (const D of [1, 15, 30, 60, 120, 240, 480, 1440]) console.log(row([D, immediate.length, expired(D).length, immediate.length + expired(D).length, Math.round((D / 1440) * RESTARTS)])); console.log(`\nformat distribution:`); for (const b of ["per-request", "cached", "at-startup"]) { const l = NAMES.filter((a) => APPS[a].format === b); console.log(` ${b.padEnd(13)} ${l.length} ${l.join(", ")}`); } const running = NAMES.filter((a) => APPS[a].format === "at-startup"); console.log(`\nat a 1440 min outage: ${immediate.length + expired(1440).length} apps go down, ` + `${running.length} apps keep running but ${RESTARTS} start attempts fail`); console.log(`apps not affected by the outage at all: 0 / ${NAMES.length}`); ``` ``` daily restarts 41 (deploys + scaling); apps that can start without the store 0 / 9 outage (min) down at once cache expired apps down failed starts 1 1 0 1 0 15 1 0 1 0 30 1 1 2 1 60 1 1 2 2 120 1 3 4 3 240 1 3 4 7 480 1 4 5 14 1440 1 4 5 41 format distribution: per-request 1 payment-bridge cached 4 portal-api, billing-service, field-api, notification at-startup 4 portal-frontend, subscriber-signup, work-order, report-generator at a 1440 min outage: 5 apps go down, 4 apps keep running but 41 start attempts fail apps not affected by the outage at all: 0 / 9 ``` What was received in exchange for closing forty-eight items is in this table. A secret gathered in one place is a **dependency** gathered in one place. Even in a one-minute outage of the store, one app goes down: the payment bridge reads its secret on every request, so the moment it cannot reach the store it takes no payments. This is the natural consequence of the most sensitive secret being read most strictly — strict reading is the way of being most exposed to an outage. The following rows show that cache life is a delayed buffer. In a thirty-minute outage the notification service goes down (its cache life is fifteen minutes), in two hours the portal API and the field API go down too, and at eight hours all four cached apps are down. The cache does not prevent the outage, it **postpones** it; the time it postpones is exactly the cache life. The right-hand column is the real finding of the table. The four apps that read at startup keep running throughout the outage and at first glance look immune. They are not immune: **none** of the nine apps can start without the store, and in this environment there are forty-one restarts a day — deploys, scale-ups, and replacement of unhealthy instances. A four-hour outage drops seven start attempts, a one-day outage forty-one. In a store outage that coincides with a load spike, scale-up does not work; the system is left under the load with the instances it has. The last row of the table is the real statement of centralization: the number of apps not affected by the outage at all is zero. Building the store makes forty-eight unused access grants closable; the same step writes a **shared startup dependency** onto all nine apps. The second item is the bill of the first, and it can be reduced with replication, not zeroed. #### A Cheaper Stage The forty-eight unused grants were measured at runtime, but they were not **born** at runtime. Each item was born when a policy was written for an app: the `data/` prefix was written because it was not known which data secret that app needed. The same item could have been closed at the deploy stage — if each app had **declared explicitly** in its deployment definition the secret paths it needs, the policy would be generated from that declaration and no item beyond the thirty-one would ever have been opened. The measure of the work differs too: writing nine declarations for nine apps versus reviewing a seventy-nine-item matrix by hand and deciding which are unnecessary differs by roughly a factor of nine — and the second is redone every time a new secret is added, while the first is written once. The only item that needs to be measured at runtime is whether the declaration itself is correct: if the read log shows more than what was declared, the declaration is incomplete; if it shows less, the declaration is excessive. This comparison is an audit, not a defense. #### Summary - Twenty-one rule lines open 79 `(app, secret)` items to nine apps; the ninety-day read log says 31. Forty-eight items, 61% of the access opened, are never used. - The source of the unused grants is the path prefix rule: `data/` is one line and gives five secrets; six apps can reach each of the five data secrets, and the most-read one is read by three apps. - No app reads three of the secrets; these are the cheapest items that can be closed without cutting anyone. - Least privilege takes the secrets exposed by a compromised app from 8.8 on average to 3.4, and from 12 in the worst case to 6. Its cost is that the policy grows from 21 lines to 31; written from thirty days of observation it cuts four legitimate reads, and the cut shows only when that job first runs. - The price of centralization is a single point: a one-minute outage stops one app, an eight-hour outage five; none of the nine apps can start without the store, and 41 start attempts a day pass through this point. - The forty-eight items were measured at runtime but born at the deploy stage; had the secret paths been declared in the deployment definition they would never have been opened — nine declarations against a 79-item matrix review is a difference of roughly a factor of nine. #### Next Step This lesson narrowed access but never touched the secrets themselves. What sits in each of the thirty-one items is still **a value with no lifetime**: written on installation day, the same from that day to this. When the access matrix is narrowed, the secrets exposed by one app fall to six, but the question of how long those six secrets would work once leaked was never asked. The next lesson measures that time: when a static secret leaks, how many days the exposure window lasts, how many times a short-lived credential produced on request narrows the same window, and in return for that narrowing how many extra requests a day are added to the store and how many new dependencies to the system. ### Short-Lived Credentials Source: https://academia.sh/en/courses/runtime-security/short-lived-credentials The time from the moment a secret leaks to the moment it expires on its own is measured: in 16 of 31 items the exposure window drops from 90 days to 64 minutes, a factor of 2,033; in the remaining 15 items it does not shrink at all, and after the move 99.96% of the total exposure gathers there. The cost is not the total of store requests but their distribution — the load of three apps grows by 6.3 to 14.5 times, 1,073 temporary credentials a day are born and die on the target systems, and the number of apps that go down in a two-hour store outage rises from four to eight. The previous lesson narrowed the access matrix: of seventy-nine `(app, secret)` items, thirty-one were actually read and forty-eight were unused grants. What sits in each of the remaining thirty-one items was never looked at. That thing is **a value with no lifetime**: written on installation day, the same from that day to this. This lesson measures that value along a single axis — if the secret leaks, how long does it keep working. The architecture of the secret storage service and how a dynamic credential generator is set up as a service were measured in earlier courses and are not repeated here. The measure here is a single duration: the **exposure window**. For a static secret this is the time until the next planned rotation; for a short-lived credential generated on request it is the remaining token life. The difference between the two is the subject of this lesson — and so is the set of items where that difference never arises. - **SM7.** The previous lesson's model continues: nine apps, one secret store, thirty-one read items, and seventeen distinct secrets. The environment is **fictional**, what is measured is a **model** written in `node`, and it **carries no real secret value**. - **SM8.** The exposure window does not include the detection of the leak. Detection and the revocation flow are measured separately in the last lesson of this topic; here the time until the secret expires **on its own** is counted. - **SM9.** For a static secret, the model assumes planned rotation is **done on time**. Late rotation and what happens during rotation are the subject of the next lesson. - **SM10.** The moment of the leak is assumed random within the window: the mean exposure window is half the rotation period, and the worst case is all of it. The same rule holds for token life. - **SM11.** Whether a target system can generate a credential on request is given in the model table. A signing key is not counted as generable: the party that verifies the signature must know the same key, so the key cannot be changed per request. - **SM12.** There is no randomness; every number is derived from the model tables and the output is the same from run to run. #### How Many Times the Window Shrinks ```js // lifetime.mjs — 31 (app, secret) items and the exposure window of each. A MODEL. // The production environment of the regional measurement network is FICTIONAL: nine apps, one secret store. // THERE ARE NO REAL SECRET VALUES: only secret paths, rotation periods, and token lives are modeled. export const READS = { "portal-frontend": ["signing/session-signing"], "portal-api": ["data/portal-primary", "data/portal-read", "signing/session-signing", "signing/internal-call-signing", "infra/cache", "infra/log-write"], "subscriber-signup": ["data/portal-primary", "external/email-gateway", "external/sms-gateway"], "billing-service": ["data/billing-primary", "data/portal-read", "data/archive", "external/payment-gateway", "signing/document-signing"], "payment-bridge": ["external/payment-gateway"], "field-api": ["data/field-primary", "queue/work-order-queue", "external/map-service", "signing/internal-call-signing"], "work-order": ["data/field-primary", "queue/work-order-queue", "queue/notification-queue", "external/map-service"], "notification": ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"], "report-generator": ["data/portal-read", "data/archive", "infra/object-store", "external/email-gateway"], }; // domain default [rotation (days), generates, token life (min)]; generates = whether the target // system can generate a short-lived credential on request. Exceptions are written separately. const DOMAIN_DEFAULT = { data: [180, true, 60], queue: [180, true, 120], infra: [180, true, 240], signing: [90, false, 0], external: [365, false, 0] }; const EXCEPTIONS = { "infra/log-write": [180, true, 720], "infra/cache": [180, false, 0], "external/payment-gateway": [90, false, 0] }; export const SECRETS = Object.fromEntries([...new Set(Object.values(READS).flat())].map((s) => { const [rotation, generates, life] = EXCEPTIONS[s] || DOMAIN_DEFAULT[s.split("/")[0]]; return [s, { rotation, generates, life }]; })); // format/life: how the secret is obtained today. instances: running instances. // restarts: restarts per day. requests: daily requests (fictional unit). export const APPS = { "portal-frontend": { format: "at-startup", life: 0, instances: 4, restarts: 6, requests: 84000 }, "portal-api": { format: "cached", life: 60, instances: 8, restarts: 12, requests: 96000 }, "subscriber-signup": { format: "at-startup", life: 0, instances: 2, restarts: 3, requests: 1400 }, "billing-service": { format: "cached", life: 240, instances: 2, restarts: 2, requests: 6200 }, "payment-bridge": { format: "per-request", life: -1, instances: 3, restarts: 2, requests: 3400 }, "field-api": { format: "cached", life: 60, instances: 5, restarts: 8, requests: 42000 }, "work-order": { format: "at-startup", life: 0, instances: 3, restarts: 4, requests: 9600 }, "notification": { format: "cached", life: 15, instances: 2, restarts: 3, requests: 12000 }, "report-generator": { format: "at-startup", life: 0, instances: 1, restarts: 1, requests: 320 }, }; export const NAMES = Object.keys(APPS); export const ITEMS = NAMES.flatMap((a) => READS[a].map((s) => [a, s])); export const DOMAINS = ["data", "queue", "infra", "signing", "external"]; // exposure window: time until the secret expires on its own once it leaks (min). export const exposure = (s, short) => short && SECRETS[s].generates ? SECRETS[s].life : SECRETS[s].rotation * 1440; export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); export const span = (min) => min < 1440 ? `${Math.round(min)} min` : `${(min / 1440).toFixed(1)} days`; export const mean = (k, short) => k.reduce((t, [, s]) => t + exposure(s, short) / 2, 0) / k.length; if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([10, 8, 15, 16, 20, 7]); const D = ITEMS.filter(([, s]) => SECRETS[s].generates), S = ITEMS.filter(([, s]) => !SECRETS[s].generates); console.log(`model: ${ITEMS.length} (app, secret) items, ${Object.keys(SECRETS).length} read ` + `secrets, ${NAMES.length} apps (fictional production environment)\n`); console.log(row(["domain", "items", "shortenable", "window today", "short-lived window", "fold"])); for (const d of DOMAINS) { const k = ITEMS.filter(([, s]) => s.startsWith(d + "/")); console.log(row([d, k.length, k.filter(([, s]) => SECRETS[s].generates).length, span(mean(k, false)), span(mean(k, true)), (mean(k, false) / mean(k, true)).toFixed(0)])); } console.log(row(["total", ITEMS.length, D.length, span(mean(ITEMS, false)), span(mean(ITEMS, true)), (mean(ITEMS, false) / mean(ITEMS, true)).toFixed(1)])); console.log(`\nonly the ${D.length} shortenable items: mean window ` + `${span(mean(D, false))} -> ${span(mean(D, true))}, ` + `${Math.round(mean(D, false) / mean(D, true))}-fold`); console.log(`the remaining ${S.length} items: mean window ${span(mean(S, false))}, unchanged`); console.log(`exposure share after the move to short-lived — ` + `${S.length} static items: ${(100 * mean(S, true) * S.length / (mean(ITEMS, true) * ITEMS.length)).toFixed(2)}%`); const rest = [...new Set(S.map(([, s]) => s))].sort((x, y) => SECRETS[y].rotation - SECRETS[x].rotation); console.log(`${rest.length} secrets that could never move to short-lived: ` + rest.map((s) => `${s.split("/")[1]}(${SECRETS[s].rotation}d)`).join(" ")); } ``` ``` model: 31 (app, secret) items, 17 read secrets, 9 apps (fictional production environment) domain items shortenable window today short-lived window fold data 10 10 90.0 days 30 min 4320 queue 4 4 90.0 days 60 min 2160 infra 3 2 90.0 days 30.1 days 3 signing 5 0 45.0 days 45.0 days 1 external 9 0 151.9 days 151.9 days 1 total 31 16 100.7 days 54.3 days 1.9 only the 16 shortenable items: mean window 90.0 days -> 64 min, 2033-fold the remaining 15 items: mean window 112.2 days, unchanged exposure share after the move to short-lived — 15 static items: 99.96% 8 secrets that could never move to short-lived: email-gateway(365d) sms-gateway(365d) map-service(365d) cache(180d) session-signing(90d) internal-call-signing(90d) payment-gateway(90d) document-signing(90d) ``` The first two rows of the table give the real size of the short-lived credential. In the database items the exposure window drops from ninety days to thirty minutes; the ratio is four thousand three hundred twenty. In the queue items, two thousand one hundred sixty. Most defenses improve an item by thirty or fifty percent; here the same item changes by three orders of magnitude at once — between ninety days and thirty minutes there are already four orders of magnitude. The last row reads the same table over the whole set, and the figure collapses. The mean over the thirty-one items falls from 100.7 days to 54.3 days: **only 1.9-fold.** The same change, in the same run, gives two thousand-fold and two-fold side by side. The difference is in the third column of the table: of the thirty-one items, sixteen can be shortened and fifteen cannot. The fifteen items that cannot be shortened fall into three classes. Signing keys (five items) cannot be changed per request, because the party that verifies the signature must know the same key; shortening the key's life breaks the verifying side. External gateway keys (nine items) depend on what the provider gives, and in this model all four give a fixed key — the payment, map, email, and SMS gateways. The cache server (one item) does not generate users on request. The two rows after the fourth are the decision itself. After the move, **99.96%** of the total exposure gathers in the fifteen items that cannot be shortened. In other words, moving to short lives shrinks 0.04% of the risk by three orders of magnitude, without touching at all the items that carry nearly all of it. This does not mean the work is worthless — when a database credential leaks, what is taken is subscriber data, and a window dropping from ninety days to thirty minutes is a real gain. What it means is this: **the mean exposure window is the wrong indicator.** The right indicator is the list of items that cannot be shortened, and that list consists of eight secrets. #### The Cost of Renewal As token life shortens, renewal becomes more frequent; and the place renewal goes to is the single point measured in the previous lesson. ```js // load.mjs — the cost of moving to short lives: store requests, generation on the target, outage resilience import { NAMES, READS, SECRETS, APPS, ITEMS, gen } from "./lifetime.mjs"; // daily store requests of one item. In the short-lived case the item renews on its own token life; // a static item keeps the app's read pattern of today. const requestCount = (a, s, short) => { const u = APPS[a]; if (short && SECRETS[s].generates) return Math.round(1440 / SECRETS[s].life) * u.instances + u.restarts; if (u.format === "per-request") return u.requests; if (u.format === "cached") return Math.round(1440 / u.life) * u.instances + u.restarts; return u.restarts; }; // how long an app holds out in a store outage: 0 for per-request readers, its cache life for cached // ones, unlimited for at-startup readers. In the short-lived case the shortest token life caps it too. const endurance = (a, short) => Math.min( APPS[a].format === "per-request" ? 0 : APPS[a].format === "cached" ? APPS[a].life : Infinity, ...(short ? READS[a].filter((s) => SECRETS[s].generates).map((s) => SECRETS[s].life) : [])); const row = gen([18, 8, 13, 13, 11, 6]); console.log(row(["app", "items", "reqs today", "short-lived", "generation", "fold"])); for (const a of NAMES) { const b = READS[a].reduce((t, s) => t + requestCount(a, s, false), 0); const k = READS[a].reduce((t, s) => t + requestCount(a, s, true), 0); const u = READS[a].filter((s) => SECRETS[s].generates).reduce((t, s) => t + requestCount(a, s, true), 0); console.log(row([a, READS[a].length, b, k, u, (k / b).toFixed(1)])); } const T = (short, f = () => true) => ITEMS.filter(([, s]) => f(s)) .reduce((t, [a, s]) => t + requestCount(a, s, short), 0); console.log(row(["total", ITEMS.length, T(false), T(true), T(true, (s) => SECRETS[s].generates), (T(true) / T(false)).toFixed(1)])); console.log(`\ntemporary credentials generated per day on the target system (created, then expired):`); for (const d of ["data", "queue", "infra"]) console.log(` ${d.padEnd(9)}${String(ITEMS.filter(([, s]) => s.startsWith(d + "/") && SECRETS[s].generates).reduce((t, [a, s]) => t + requestCount(a, s, true), 0)).padStart(5)}`); const row2 = gen([13, 22, 20]); console.log(`\n${row2(["outage (min)", "apps down today", "short-lived down"])}`); for (const D of [15, 60, 120, 240, 1440]) console.log(row2([D, NAMES.filter((a) => endurance(a, false) < D).length, NAMES.filter((a) => endurance(a, true) < D).length])); const resilient = NAMES.filter((a) => endurance(a, true) === Infinity); console.log(`\napps that can run indefinitely in a store outage: today ` + `${NAMES.filter((a) => endurance(a, false) === Infinity).length}, short-lived ${resilient.length}` + `${resilient.length ? ` (${resilient.join(", ")})` : ""}`); ``` ``` app items reqs today short-lived generation fold portal-frontend 1 6 6 0 1.0 portal-api 6 1224 1048 436 0.9 subscriber-signup 3 9 57 51 6.3 billing-service 5 70 178 150 2.5 payment-bridge 1 3400 3400 0 1.0 field-api 4 512 452 196 0.9 work-order 4 16 160 156 10.0 notification 3 585 417 27 0.7 report-generator 4 4 58 57 14.5 total 31 5826 5776 1073 1.0 temporary credentials generated per day on the target system (created, then expired): data 863 queue 175 infra 35 outage (min) apps down today short-lived down 15 1 1 60 2 2 120 4 8 240 4 8 1440 5 8 apps that can run indefinitely in a store outage: today 4, short-lived 1 (portal-frontend) ``` The total row at first glance says the opposite of what was expected: daily store requests fall from five thousand eight hundred twenty-six to five thousand seven hundred seventy-six, that is, they do not change at all. It cannot be concluded that moving to short lives adds no load to the store; the total hides two movements that cancel each other. The app rows separate those two movements. Of the four apps whose load grows, three are the ones that today read their secret **only at startup**: the report generator by fourteen and a half times, the work order service by ten times, subscriber signup by six point three times. Today they make four, sixteen, and nine requests a day; after the move to short lives, fifty-eight, one hundred sixty, and fifty-seven. A value with no lifetime is read once, a value with a lifetime is renewed constantly. In the three apps whose load falls, the reason is the reverse: the portal and field APIs already refresh their cache every sixty minutes, and the token life is sixty minutes too; the only thing that changes is the log write key moving to a twelve-hour life and the requests for that item dropping from two hundred four to twenty-eight. The second table shows the real new cost. The counterpart of one thousand seventy-three requests is not a read but a **generation**: on the target systems, eight hundred sixty-three database users, one hundred seventy-five queue credentials, and thirty-five infrastructure credentials are created every day and cleaned up when they expire. This is the load of the database, not of the secret store, and it is often forgotten in secrets management accounting; a flow that can create but cannot delete produces an accumulation of eight hundred sixty-three items a day. The third table is the most expensive item of the cost, and it directly worsens the finding of the previous lesson. The number of apps that can hold out two hours in a store outage is four today; after the move to short lives that number rises to eight — in other words, eight of the nine apps go down in a two-hour store outage. The number of apps that can withstand an outage indefinitely falls from four to one, and that one app is the portal frontend, which has no shortenable items at all. Cutting the exposure window from ninety days to thirty minutes means cutting the app's dependence on the store from ninety days to thirty minutes too; the two are two faces of the same setting and cannot be chosen separately. #### A Cheaper Stage The thirty-minute window is set up at runtime, but whether the window **can be shortened** is not decided at runtime. Each of the eight secrets that cannot be shortened was chosen at an earlier stage: the email and SMS gateways at the vendor selection stage, the signing keys at the design stage, the cache server at the infrastructure selection. If the question "does it generate a short-lived credential on request" was not asked when an external provider was chosen, that provider's key stays static for the lifetime of the system; this lesson can measure that key but cannot shorten it. All nine external items were born this way, and today they carry the largest share of the total exposure. The only decision that truly belongs to runtime is the choice of token life in the sixteen shortenable items. That choice is not one-directional either: lowering the life from sixty minutes to fifteen divides the window by four, multiplies the store requests by four, and divides the outage endurance by four. The three are the same number. #### Summary - The exposure window is the time from the moment a secret leaks to the moment it expires on its own: until the next planned rotation for a static secret, the remaining token life for a short-lived credential. - In the 16 shortenable items the window falls on average from 90 days to 64 minutes, a factor of 2,033; in the database items the ratio is 4,320. - Over all thirty-one items the window narrows only 1.9-fold on average. After the move, 99.96% of the total exposure gathers in the 15 items that cannot be shortened. This is why the mean exposure window is the wrong indicator; the right one is the eight-secret list of what cannot be shortened. - Daily requests to the store go from 5,826 to 5,776, that is, they do not change; but 1,073 of them are now generations rather than reads, and 863 database users are born and die every day on the target systems. The load of the three apps that read their secret only at startup grows by 6.3, 10, and 14.5 times. - The most expensive item of the cost is outage resilience: the apps that go down in a two-hour store outage rise from 4 to 8, and the ones that can hold out indefinitely fall from 4 to 1. Shortening the window and tightening the store dependency are the same setting. - The eight secrets that cannot be shortened were fixed at the vendor and design choice stages; the runtime measures them but cannot shorten them. #### Next Step This lesson showed that life cannot be shortened in fifteen items. For those items one tool is left: **rotation**. If the secret does not shorten, it is changed at set intervals and the exposure window is bounded by the rotation period. But rotation is not something that happens in an instant. Every app that will read the new value has to move over, and apps do not move at the same speed: one in sixty minutes, another only when it restarts. During that transition **two secrets must stay valid together**, or the app that has not moved is cut off. The next lesson calculates that time: for seventeen secrets, how many minutes the minimum overlap is, how many clients miss the new value when the time is kept short, and how many legitimate requests are cut. ### Key Rotation Source: https://academia.sh/en/courses/runtime-security/key-rotation Rotation is measured in terms of the window in which two secrets stay valid together: the minimum overlap of seventeen secrets lies between 60 and 1,440 minutes, averages 815 minutes, and in eleven of them four apps that read only at startup set the window alone. When the overlap is kept short the arithmetic reverses — in a 60-minute window 36 clients miss the new value and 87,427 legitimate requests are cut, and when the order is reversed even the one app that reads on every request is cut. The same window drops from 1,440 minutes to 240 minutes when refresh-on-auth-failure is added. The previous lesson showed that the exposure window cannot be shortened in fifteen items: signing keys, external gateway keys, and the cache server do not generate credentials on request. For those items one tool is left — changing the secret at set intervals. Rotation bounds the exposure window by its period; the window of a key rotated every ninety days cannot be longer than ninety days. But rotation does not happen in an instant. At the moment the new value is written to the store, no app knows about it; each app moves over in its own way, at its own speed. Until that move is complete, **both values must stay valid**. This lesson measures that mandatory window: how many minutes it is, what sets it, what is cut when it is kept short, and what happens when the order is reversed. - **SM13.** The model of the previous two lessons continues: nine apps, seventeen read secrets. The environment is **fictional**, what is measured is a **model** written in `node`, and it **carries no real secret value**. - **SM14.** The switch time is derived from the read pattern. An app that reads on every request switches immediately; a cached app switches after at most the cache life; an app that reads only at startup switches only when it restarts. For the last one the model uses the mean instance lifetime: the number of instances divided by the number of restarts per day. - **SM15.** The cut requests are counted as the worst case: all instances of an app switch at the end of the switch time. In reality the switches are spread across instances and the cut requests stay below this number; the ranking does not change. - **SM16.** The seventeen secrets are rotated **one by one** and the results are summed. The store is assumed to revoke the old value instantly; the propagation time of the revocation itself is not modeled. - **SM17.** There is no randomness; every number is derived from the model tables and the output is the same from run to run. #### The Slowest Reader Sets the Window ```js // switchover.mjs — the minimum time two secrets must stay valid together during rotation. A MODEL. // The environment is FICTIONAL; THERE ARE NO REAL SECRET VALUES, only secret paths and switch times are modeled. export const READS = { "portal-frontend": ["signing/session-signing"], "portal-api": ["data/portal-primary", "data/portal-read", "signing/session-signing", "signing/internal-call-signing", "infra/cache", "infra/log-write"], "subscriber-signup": ["data/portal-primary", "external/email-gateway", "external/sms-gateway"], "billing-service": ["data/billing-primary", "data/portal-read", "data/archive", "external/payment-gateway", "signing/document-signing"], "payment-bridge": ["external/payment-gateway"], "field-api": ["data/field-primary", "queue/work-order-queue", "external/map-service", "signing/internal-call-signing"], "work-order": ["data/field-primary", "queue/work-order-queue", "queue/notification-queue", "external/map-service"], "notification": ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"], "report-generator": ["data/portal-read", "data/archive", "infra/object-store", "external/email-gateway"], }; // format/life: how the secret is read. instances: running instances. restarts: restarts per day. // requests: daily requests (fictional unit). export const APPS = { "portal-frontend": { format: "at-startup", life: 0, instances: 4, restarts: 6, requests: 84000 }, "portal-api": { format: "cached", life: 60, instances: 8, restarts: 12, requests: 96000 }, "subscriber-signup": { format: "at-startup", life: 0, instances: 2, restarts: 3, requests: 1400 }, "billing-service": { format: "cached", life: 240, instances: 2, restarts: 2, requests: 6200 }, "payment-bridge": { format: "per-request", life: -1, instances: 3, restarts: 2, requests: 3400 }, "field-api": { format: "cached", life: 60, instances: 5, restarts: 8, requests: 42000 }, "work-order": { format: "at-startup", life: 0, instances: 3, restarts: 4, requests: 9600 }, "notification": { format: "cached", life: 15, instances: 2, restarts: 3, requests: 12000 }, "report-generator": { format: "at-startup", life: 0, instances: 1, restarts: 1, requests: 320 }, }; export const NAMES = Object.keys(APPS); export const SECRETS = [...new Set(Object.values(READS).flat())].sort(); export const readersOf = (s) => NAMES.filter((a) => READS[a].includes(s)); // switchTime: the worst-case time (min) for an app to move to the new value once it is written. // a per-request reader switches immediately; a cached one after at most its cache life; an // at-startup reader only on restart, that is, after the mean instance lifetime. export const switchTime = (a) => APPS[a].format === "per-request" ? 0 : APPS[a].format === "cached" ? APPS[a].life : Math.round(1440 * APPS[a].instances / APPS[a].restarts); export const overlap = (s) => Math.max(...readersOf(s).map(switchTime)); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([23, 9, 10, 8, 24]), row0 = gen([19, 13, 11, 10, 13]); console.log(`switch time — worst-case time to move to the new value (min)`); console.log(row0(["app", "format", "instances", "restarts", "switch(min)"])); for (const a of NAMES) console.log(row0([a, APPS[a].format, APPS[a].instances, APPS[a].restarts, switchTime(a)])); console.log(`\n${row(["secret", "readers", "overlap", "= hours", "determining app"])}`); const sorted = [...SECRETS].sort((x, y) => overlap(y) - overlap(x) || x.localeCompare(y)); for (const s of sorted) console.log(row([s.split("/")[1], readersOf(s).length, overlap(s), (overlap(s) / 60).toFixed(1), readersOf(s).filter((a) => switchTime(a) === overlap(s)).join(",")])); const O = SECRETS.map(overlap); console.log(`\n${SECRETS.length} secrets; longest minimum overlap ${Math.max(...O)} min, shortest ` + `${Math.min(...O)} min, mean ${(O.reduce((t, x) => t + x, 0) / O.length).toFixed(0)} min`); const slow = NAMES.filter((a) => APPS[a].format === "at-startup"); console.log(`overlap needed if rotated in a single shared window: ${Math.max(...O)} min`); console.log(`what sets the overlap: ${slow.length} at-startup apps, alone in ` + `${SECRETS.filter((s) => readersOf(s).some((a) => APPS[a].format === "at-startup" && switchTime(a) === overlap(s))).length} / ${SECRETS.length} secrets`); } ``` ``` switch time — worst-case time to move to the new value (min) app format instances restarts switch(min) portal-frontend at-startup 4 6 960 portal-api cached 8 12 60 subscriber-signup at-startup 2 3 960 billing-service cached 2 2 240 payment-bridge per-request 3 2 0 field-api cached 5 8 60 work-order at-startup 3 4 1080 notification cached 2 3 15 report-generator at-startup 1 1 1440 secret readers overlap = hours determining app archive 2 1440 24.0 report-generator portal-read 3 1440 24.0 report-generator email-gateway 3 1440 24.0 report-generator object-store 1 1440 24.0 report-generator field-primary 2 1080 18.0 work-order map-service 2 1080 18.0 work-order notification-queue 2 1080 18.0 work-order work-order-queue 2 1080 18.0 work-order portal-primary 2 960 16.0 subscriber-signup sms-gateway 2 960 16.0 subscriber-signup session-signing 2 960 16.0 portal-frontend billing-primary 1 240 4.0 billing-service payment-gateway 2 240 4.0 billing-service document-signing 1 240 4.0 billing-service cache 1 60 1.0 portal-api log-write 1 60 1.0 portal-api internal-call-signing 2 60 1.0 portal-api,field-api 17 secrets; longest minimum overlap 1440 min, shortest 60 min, mean 815 min overlap needed if rotated in a single shared window: 1440 min what sets the overlap: 4 at-startup apps, alone in 11 / 17 secrets ``` The table above shows where the switch time comes from. The payment bridge reads its secret on every request, so it switches in zero minutes; the notification service, with its fifteen-minute cache, switches in fifteen minutes at the most. In the four apps that read only at startup, however, the switch time is not a setting but a **statistic**: the report generator runs with a single instance and restarts once a day, so it sees the new value after a mean of one thousand four hundred forty minutes. This number is written in no setting of secrets management; it comes from the deployment layout. The lower table gives the real measure. The minimum overlap of a secret is the switch time of the **slowest** of its readers; put another way, the overlap is not a mean but a maximum. This is why the object store key, which has a single reader, and the email gateway key, which has three, sit on the same line: in both the slowest reader is the report generator, and both need twenty-four hours of overlap. The number of readers does not set the overlap; the **worst** of the readers does. The three secrets that need the shortest overlap are those read only by cached apps: the internal call signing key is read by two apps and can turn over in sixty minutes. Between one thousand four hundred forty minutes and sixty minutes there is a factor of twenty-four, and the whole of that difference comes from the read pattern — there is no difference in the sensitivity or the policy of the secrets. The last line gathers the finding in one sentence: four apps that read at startup set the overlap **alone** in eleven of the seventeen secrets. These are the smallest apps in the system, the ones that receive the fewest requests — the report generator sees three hundred twenty requests a day. The whole rotation calendar depends on the restart habits of these four apps. #### The Bill for Shortening the Window A twenty-four-hour overlap works against the purpose of rotation: a leaked old value keeps working for those twenty-four hours too. Shortening the window has a measurable counterpart. ```js // order.mjs — what is cut when the overlap window is kept short and when the order is reversed import { NAMES, READS, SECRETS, APPS, switchTime, readersOf, overlap, gen } from "./switchover.mjs"; // With an overlap of T minutes, an app stays on the old value for switchTime(a) - T minutes and // drops the requests that depend on that secret for that long. The seventeen secrets are rotated // one by one and the results are summed. const cutRequests = (T, g = switchTime) => SECRETS.reduce((t, s) => t + readersOf(s) .reduce((x, a) => x + APPS[a].requests * Math.max(0, g(a) - T) / 1440, 0), 0); const missedClients = (T, g = switchTime) => SECRETS.reduce((t, s) => t + readersOf(s) .reduce((x, a) => x + (g(a) > T ? APPS[a].instances : 0), 0), 0); const row = gen([14, 18, 16, 20]); console.log(row(["overlap(min)", "missed clients", "cut requests", "old value valid"])); for (const T of [0, 15, 60, 240, 960, 1080, 1440]) console.log(row([T, missedClients(T), Math.round(cutRequests(T)), `${T} min`])); const OVERLAPS = SECRETS.map(overlap), overlapSum = OVERLAPS.reduce((t, x) => t + x, 0); console.log(`\nif each secret uses its own minimum overlap: missed clients 0, cut requests 0,`); console.log(`the old value stays valid ${(overlapSum / SECRETS.length).toFixed(0)} min ` + `(${(overlapSum / SECRETS.length / 60).toFixed(1)} hours) longer on average`); // reverse order: the old value is revoked first, the new one is written WRITE minutes later, then it propagates. const WRITE = 5; console.log(`\nreverse order (revoke first, new value after ${WRITE} min): missed clients ` + `${missedClients(-1)}, cut requests ${Math.round(cutRequests(0) + SECRETS.reduce((t, s) => t + readersOf(s).reduce((x, a) => x + APPS[a].requests * WRITE / 1440, 0), 0))}`); console.log(`in the correct order the only app that is never cut is the per-request reader payment-bridge; ` + `in reverse order it too loses ${Math.round(READS["payment-bridge"].length * APPS["payment-bridge"].requests * WRITE / 1440)} requests`); // refresh on auth failure: an at-startup app refreshes the secret when it gets an authentication error. const REFRESH = 5; const switchWithRefresh = (a) => APPS[a].format === "at-startup" ? REFRESH : switchTime(a); const OVERLAPS2 = SECRETS.map((s) => Math.max(...readersOf(s).map(switchWithRefresh))); console.log(`\nif refresh-on-auth-failure is added (${NAMES.filter((a) => APPS[a].format === "at-startup").length} at-startup apps, switch ${REFRESH} min):`); console.log(` longest minimum overlap ${Math.max(...OVERLAPS)} -> ${Math.max(...OVERLAPS2)} min, ` + `mean ${(overlapSum / SECRETS.length).toFixed(0)} -> ` + `${(OVERLAPS2.reduce((t, x) => t + x, 0) / SECRETS.length).toFixed(0)} min`); console.log(` cut requests at 60 min overlap ${Math.round(cutRequests(60))} -> ` + `${Math.round(cutRequests(60, switchWithRefresh))}, missed clients ${missedClients(60)} -> ` + `${missedClients(60, switchWithRefresh)}`); ``` ``` overlap(min) missed clients cut requests old value valid 0 110 125422 0 min 15 104 115642 15 min 60 36 87427 60 min 240 26 67567 240 min 960 16 3627 960 min 1080 4 320 1080 min 1440 0 0 1440 min if each secret uses its own minimum overlap: missed clients 0, cut requests 0, the old value stays valid 815 min (13.6 hours) longer on average reverse order (revoke first, new value after 5 min): missed clients 113, cut requests 128693 in the correct order the only app that is never cut is the per-request reader payment-bridge; in reverse order it too loses 12 requests if refresh-on-auth-failure is added (4 at-startup apps, switch 5 min): longest minimum overlap 1440 -> 240 min, mean 815 -> 102 min cut requests at 60 min overlap 87427 -> 3875, missed clients 36 -> 10 ``` The table is a tuning curve and should be read from both ends. The top row is the case where the overlap is zero: as soon as the new value is written, the old one is revoked. One hundred ten clients — that is, one hundred ten items when counted over the secrets read by the thirty running instances of the nine apps — stay on the old value, and in the total of seventeen rotations one hundred twenty-five thousand four hundred twenty-two legitimate requests are cut. The bottom row is the opposite: at an overlap of one thousand four hundred forty minutes no client is missed, no request is cut, and the old value stays valid one full day longer. The middle rows show what the distribution looks like. An overlap of sixty minutes brings the missed clients down from one hundred ten to thirty-six, but the cut requests only from one hundred twenty-five thousand to eighty-seven thousand. The mismatch between the ratios is meaningful: most of the missed clients are the eight instances of the portal API, and those instances switch in sixty minutes; most of the cut requests, however, come from **a single app**. The portal frontend sees eighty-four thousand requests a day and switches after nine hundred sixty minutes; this one app alone produces fifty-two thousand five hundred of the eighty-seven thousand requests cut at an overlap of sixty minutes. The next two rows write the correct procedure. If each secret is rotated with its own minimum overlap, the cut requests are zero; the cost is that the old value stays valid a mean of eight hundred fifteen minutes longer. In other words, the overlap is not a security setting but **a fixed duration added to the exposure window**: the real worst-case window of a key rotated on a ninety-day period is not ninety days but ninety days plus thirteen and a half hours. The reverse-order row is a separate class. When the old value is revoked first and the new one is written five minutes later, the missed clients rise to one hundred thirteen and the cut requests to one hundred twenty-eight thousand six hundred ninety-three. The difference is not large — three clients, three thousand requests — but what is decisive is **who it falls on**: the payment bridge, which is not affected at all in the correct order, loses twelve payment requests in the reverse order. The only way to cut an app that switches in zero minutes is for the value it will switch to not to have been written yet. That is the symptom of the reverse order, and in the dashboard only these twelve requests show. #### Where the Window Shrinks The last two lines answer the real question of the course. A twenty-four-hour overlap is not a secrets management problem; it is the problem of **an app that reads at startup never reading the secret again**. When the behavior of refreshing the secret on an authentication error is added to the apps, the minimum overlap falls from one thousand four hundred forty minutes to two hundred forty minutes and the mean from eight hundred fifteen minutes to one hundred two minutes; at an overlap of sixty minutes the cut requests fall from eighty-seven thousand four hundred twenty-seven to three thousand eight hundred seventy-five, that is, by a factor of twenty-three. Where this change is made matters. It is made not in the secret store, in the rotation calendar, or in the runtime defense, but **in the application code**: the authentication error is caught, the secret is read again, and the request is tried once more. A behavior of a few dozen lines shortens the rotation window by a factor of twenty-four, and once written it holds for all secrets. The way to get the same result at runtime is to keep a separate overlap calendar for each secret, to track at every rotation which app has switched, and to restart by hand the one that has not. What remains, two hundred forty minutes, truly belongs to the runtime: the four-hour cache life of the billing service. It is a setting and can be lowered; when it is lowered, the store requests measured in the previous lesson increase. Overlap, cache life, and store load are three views of a single number. #### Summary - The minimum overlap of a secret is the switch time of the slowest of its readers. In the seventeen secrets this time lies between 60 and 1,440 minutes, with a mean of 815 minutes; with a single shared window it is 1,440 minutes. - What sets the overlap is not the sensitivity of the secret or the number of readers but the read format. Four apps that read at startup set the window alone in 11 of the 17 secrets. - When the overlap is reduced to zero, 110 clients stay on the old value and 125,422 legitimate requests are cut; at 60 minutes, 36 clients and 87,427 requests; at 1,440 minutes, zero and zero. 52,500 of the cut requests come from a single app, the portal frontend. - If each secret is rotated with its own minimum overlap the cut is zero; the cost is that the old value stays valid a mean of 815 minutes longer. The real window of a ninety-day period is ninety days plus thirteen and a half hours. - The reverse order (revoke first, write after) raises the cut requests to 128,693. Its real symptom is not the volume but the payment bridge, which is not affected at all in the correct order, losing 12 requests. - The change that shortens the overlap belongs to the code stage: refresh-on-auth-failure takes the minimum overlap from 1,440 to 240 minutes and reduces the requests that are cut at a 60-minute window by a factor of 23. #### Next Step What was measured over two lessons was two properties of the same object: the life of the secret and the cost of changing it. Both assumed that the secret **exists**. That assumption can be removed. The value an app carries to connect to a database is a way of proving who that app is; if the proof is given by the platform, there is no value left to carry. The next lesson counts this along a single axis: how many of the thirty-one items workload identity removes, how many items and which systems remain, how many apps come to carry no static secret at all — and how many lines the trust relation that takes the place of the secret is itself. ### Workload Identity Source: https://academia.sh/en/courses/runtime-security/workload-identity Workload identity is measured with a single number: how many static secrets it replaces. Twenty of the thirty-one items disappear, eleven remain, and all of the remainder gathers in six secrets — the keys that sit on the signature-verifying side and the external providers that do not support federation. Of the nine apps only two come to carry no static secret at all, and seven keep carrying at least one. The trust relation that replaces the secret does not reduce the item count; it reduces how leakable the item is — and when written at the group level it opens 36 pairs instead of 20. What was measured over two lessons was two properties of the same object: the life of the secret and the cost of changing it. Both assumed that the secret **exists**. This lesson removes that assumption. The value an app carries to connect to a database is, in fact, a way of proving who that app is. If the proof comes not from a value the app itself carries but from the platform that runs the app, there is no value left to carry. The mechanics of workload identity — how the identity is given, how it is verified, which components are involved — were measured in an earlier course and are not repeated here. The measure here is a single number: **how many static secrets it replaces, and how many it cannot.** - **SM18.** The model of the previous three lessons continues: nine apps, thirty-one items, seventeen secrets. The environment is **fictional**, what is measured is a **model** written in `node`, and it **carries no real secret value**. - **SM19.** Whether a target system can verify workload identity directly is given in the model table and written with its reason. There are two reasons: the proof has to stay on the verifying side, and the provider does not support federation. - **SM20.** Signing keys are a special case. The party that verifies a document or session signature needs the key itself; workload identity authorizes a call, it does not have a signature verified. By contrast, the signature of a **service-to-service call** is authorization and is covered by workload identity. - **SM21.** Token life is taken as fifteen minutes; this is a fictional value, and a workload identity token is assumed not to be revocable before it expires. - **SM22.** There is no randomness; every number is derived from the model tables and the output is the same from run to run. #### How Many Items Disappear ```js // identity.mjs — the static secrets that workload identity removes and the items that remain. A MODEL. // The environment is FICTIONAL; THERE ARE NO REAL SECRET VALUES, only secret paths and acceptance properties are modeled. export const READS = { "portal-frontend": ["signing/session-signing"], "portal-api": ["data/portal-primary", "data/portal-read", "signing/session-signing", "signing/internal-call-signing", "infra/cache", "infra/log-write"], "subscriber-signup": ["data/portal-primary", "external/email-gateway", "external/sms-gateway"], "billing-service": ["data/billing-primary", "data/portal-read", "data/archive", "external/payment-gateway", "signing/document-signing"], "payment-bridge": ["external/payment-gateway"], "field-api": ["data/field-primary", "queue/work-order-queue", "external/map-service", "signing/internal-call-signing"], "work-order": ["data/field-primary", "queue/work-order-queue", "queue/notification-queue", "external/map-service"], "notification": ["queue/notification-queue", "external/email-gateway", "external/sms-gateway"], "report-generator": ["data/portal-read", "data/archive", "infra/object-store", "external/email-gateway"], }; // accepts: whether the target system can verify workload identity directly (yes/no + reason). // rotation: the current rotation period (days) of a secret that cannot take an identity. const DOMAIN_DEFAULT = { data: [true, "-", 180], queue: [true, "-", 180], infra: [true, "-", 180], signing: [false, "verifier-side", 90], external: [false, "provider-unsupported", 365] }; const EXCEPTIONS = { "infra/cache": [false, "provider-unsupported", 180], "signing/internal-call-signing": [true, "-", 90], "external/map-service": [true, "-", 365], "external/payment-gateway": [false, "provider-unsupported", 90] }; export const SECRETS = Object.fromEntries([...new Set(Object.values(READS).flat())].map((s) => { const [accepts, reason, rotation] = EXCEPTIONS[s] || DOMAIN_DEFAULT[s.split("/")[0]]; return [s, { accepts, reason, rotation }]; })); export const NAMES = Object.keys(READS); export const ITEMS = NAMES.flatMap((a) => READS[a].map((s) => [a, s])); export const remaining = (a) => READS[a].filter((s) => !SECRETS[s].accepts); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([10, 9, 10, 11, 19, 26]); console.log(`model: ${ITEMS.length} (app, secret) items, ${Object.keys(SECRETS).length} secrets, ` + `${NAMES.length} apps (fictional production environment)\n`); console.log(row(["domain", "secrets", "removed", "remaining", "remaining secrets", "why it remains"])); for (const d of ["data", "queue", "infra", "signing", "external"]) { const k = ITEMS.filter(([, s]) => s.startsWith(d + "/")); const rm = k.filter(([, s]) => !SECRETS[s].accepts); const g = [...new Set(rm.map(([, s]) => SECRETS[s].reason))]; console.log(row([d, [...new Set(k.map(([, s]) => s))].length, k.length - rm.length, rm.length, [...new Set(rm.map(([, s]) => s))].length, g.length ? g.join(",") : "-"])); } const K = ITEMS.filter(([, s]) => !SECRETS[s].accepts); console.log(row(["total", Object.keys(SECRETS).length, ITEMS.length - K.length, K.length, [...new Set(K.map(([, s]) => s))].length, ""])); const row2 = gen([19, 10, 11, 36]); console.log(`\n${row2(["app", "items", "remaining", "remaining secrets"])}`); for (const a of NAMES) console.log(row2([a, READS[a].length, remaining(a).length, remaining(a).map((s) => s.split("/")[1]).join(",") || "none"])); const clean = NAMES.filter((a) => remaining(a).length === 0); console.log(`\napps carrying no static secret: ${clean.length} / ${NAMES.length} ` + `(${clean.join(", ")})`); console.log(`apps carrying at least one static secret: ${NAMES.length - clean.length} / ${NAMES.length}`); const G = K.reduce((t, [, s]) => t + SECRETS[s].rotation, 0); const T = ITEMS.reduce((t, [, s]) => t + SECRETS[s].rotation, 0); console.log(`total exposure window: ${Math.round(T / 2)} -> ${Math.round(G / 2)} item-days ` + `(${(100 - 100 * G / T).toFixed(0)}% drop); mean per item ` + `${(T / ITEMS.length / 2).toFixed(0)} -> ${(G / K.length / 2).toFixed(0)} days`); } ``` ``` model: 31 (app, secret) items, 17 secrets, 9 apps (fictional production environment) domain secrets removed remaining remaining secrets why it remains data 5 10 0 0 - queue 2 4 0 0 - infra 3 2 1 1 provider-unsupported signing 3 2 3 2 verifier-side external 4 2 7 3 provider-unsupported total 17 20 11 6 app items remaining remaining secrets portal-frontend 1 1 session-signing portal-api 6 2 session-signing,cache subscriber-signup 3 2 email-gateway,sms-gateway billing-service 5 2 payment-gateway,document-signing payment-bridge 1 1 payment-gateway field-api 4 0 none work-order 4 0 none notification 3 2 email-gateway,sms-gateway report-generator 4 1 email-gateway apps carrying no static secret: 2 / 9 (field-api, work-order) apps carrying at least one static secret: 7 / 9 total exposure window: 3123 -> 1228 item-days (61% drop); mean per item 101 -> 112 days ``` The first two rows of the upper table give the expected result: all of the database and queue items, fourteen in number, disappear. These are internal systems, and they can learn who the workload is from the platform and authorize it directly. From the standpoint of secrets management these items no longer exist — nothing is stored, nothing is rotated, nothing leaks. The signing row draws the distinction. Of the three signing secrets, one, the service-to-service call signature, is removed; because that signature is a proof of authorization and workload identity does the same job. The other two remain: the party that verifies the session signature and the party that verifies the document signature need **the key itself**. This distinction is often skipped, and it is where the claim "workload identity removes all keys" breaks. Workload identity authorizes a call; it does not have a signature verified. The external row is the largest share of what remains: of the four external gateways only the map service supports federation, three give a fixed key, and those three secrets are spread over seven items. Compared with the previous lesson's list of "eight secrets that cannot be shortened," the table is consistent — the same provider limits, in the same items. The lower table is the decision itself. The removal of twenty items leaves only **two** of the nine apps carrying no static secret: the field API and the work order service. The other seven apps keep carrying at least one static secret, and most carry only one or two. This is the most important finding of the measurement. **While the item count falls by sixty-five percent, the number of apps carrying a secret falls only from nine to seven.** The work of secrets management is proportional not to the item count but to the number of apps that carry a secret: every app must still have a secret distribution path, an owner, a rotation calendar, and a leak procedure. Saving two apps does not remove any of those seven procedures. The last line says the same thing in terms of the exposure window. The total exposure falls from three thousand one hundred twenty-three item-days to one thousand two hundred twenty-eight, down sixty-one percent. But the mean window per item **rises** from one hundred one days to one hundred twelve: the removed items were internal items with short rotation periods, and the remaining ones are external keys with three hundred sixty-five days. The quality of what remains gets worse. #### What Takes the Place of the Secret The twenty removed items do not go into nothing; in place of each, a **trust relation** is written: a binding rule that says which workload identity the target system will accept. ```js // binding.mjs — the trust relation that replaces the secret: how many rules, how many pairs, how many single points import { NAMES, READS, SECRETS, ITEMS, remaining, gen } from "./identity.mjs"; const TOKEN = 15; // life of the workload identity token (min, fictional) export const GROUPS = { portal: ["portal-frontend", "portal-api", "subscriber-signup"], billing: ["billing-service", "payment-bridge"], field: ["field-api", "work-order"], back: ["notification", "report-generator"] }; const withIdentity = (a) => READS[a].filter((s) => SECRETS[s].accepts); const row = gen([10, 11, 9, 13, 13, 17]); console.log(row(["group", "workloads", "targets", "group rules", "group pairs", "workload rules"])); let GR = 0, GP = 0, WR = 0; for (const [g, members] of Object.entries(GROUPS)) { const targets = [...new Set(members.flatMap(withIdentity))]; const wl = members.reduce((t, a) => t + withIdentity(a).length, 0); GR += targets.length; GP += members.length * targets.length; WR += wl; console.log(row([g, members.length, targets.length, targets.length, members.length * targets.length, wl])); } console.log(row(["total", NAMES.length, "", GR, GP, WR])); console.log(`\ngroup-level binding: ${GR} rules, ${GP} (workload, target) pairs open`); console.log(`workload-level binding: ${WR} rules, ${WR} pairs open; ` + `unused pairs ${GP - WR}`); console.log(`in both cases the number of rules carrying a leakable value: 0 ` + `(a rule is an authorization record, not a secret)`); const row2 = gen([19, 10, 16, 26]); console.log(`\n${row2(["app", "identity", "static secrets", "single point bound to"])}`); for (const a of NAMES) { const k = withIdentity(a).length, st = remaining(a).length; console.log(row2([a, k, st, k && st ? "store+identity" : k ? "identity" : "store"])); } const D = NAMES.filter((a) => remaining(a).length > 0), K = NAMES.filter((a) => withIdentity(a).length > 0); console.log(`\nbound to the secret store ${D.length}, bound to the identity provider ${K.length}, ` + `to both ${NAMES.filter((a) => remaining(a).length && withIdentity(a).length).length}, ` + `bound to neither ${NAMES.filter((a) => !remaining(a).length && !withIdentity(a).length).length}`); const removed = ITEMS.filter(([, s]) => SECRETS[s].accepts); console.log(`today the exposure of the ${removed.length} removed items is ` + `${removed.reduce((t, [, s]) => t + SECRETS[s].rotation / 2, 0)} item-days; the life of the token ` + `replacing them is ${TOKEN} min, that is ${(removed.length * TOKEN / 2 / 1440).toFixed(2)} item-days`); console.log(`apps that stop ${TOKEN} min into an identity provider outage: ${K.length} / ` + `${NAMES.length}; apps not affected by the outage at all: 0 / ${NAMES.length}`); ``` ``` group workloads targets group rules group pairs workload rules portal 3 4 4 12 5 billing 2 3 3 6 3 field 2 5 5 10 8 back 2 4 4 8 4 total 9 16 36 20 group-level binding: 16 rules, 36 (workload, target) pairs open workload-level binding: 20 rules, 20 pairs open; unused pairs 16 in both cases the number of rules carrying a leakable value: 0 (a rule is an authorization record, not a secret) app identity static secrets single point bound to portal-frontend 0 1 store portal-api 4 2 store+identity subscriber-signup 1 2 store+identity billing-service 3 2 store+identity payment-bridge 0 1 store field-api 4 0 identity work-order 4 0 identity notification 1 2 store+identity report-generator 3 1 store+identity bound to the secret store 7, bound to the identity provider 7, to both 5, bound to neither 0 today the exposure of the 20 removed items is 1895 item-days; the life of the token replacing them is 15 min, that is 0.10 item-days apps that stop 15 min into an identity provider outage: 7 / 9; apps not affected by the outage at all: 0 / 9 ``` The upper table repeats, as is, the finding of the first lesson of this topic; only the object has changed. When the binding rule is written for a workload group, sixteen rules are enough but thirty-six `(workload, target)` pairs are opened; when it is written per workload, twenty rules are needed and exactly twenty pairs are opened. The sixteen unused pairs are the same form as the forty-eight unused grants of the first lesson: **a rule written broadly means few lines and much access.** Moving to workload identity does not solve this problem; it moves the problem from the secret path prefix to the workload group. The third line says where the difference is. Even if the number of binding rules stays at twenty, a rule **carries no leakable value**. The exposure of the twenty removed items was one thousand eight hundred ninety-five item-days; the life of the token that replaces them is fifteen minutes, that is, zero point one item-days. The gain is not in the item count but in the quality of the item: an authorization record is not something that can be stolen, because when copied it is of no use — it takes the workload that carries the identity itself. The lower table gives the cost, and the cost is familiar. The first lesson measured that the central secret store writes a shared startup dependency for nine apps. Workload identity does not remove this dependency, it **raises it to two**: seven apps are still bound to the secret store, seven apps to the identity provider, and five to both. The number of apps bound to neither is zero. The field API and the work order service are completely free of the secret store, but when the identity provider is unreachable for fifteen minutes both stop. The name of the dependency changed, its count did not. #### Where This Decision Belongs The removal of twenty items is not something that happens at runtime. No runtime defense caught, stopped, or flagged a secret; the items vanished because **the deployment no longer produces them.** This is the cheapest class of item measured in this course: not catching an item but never giving birth to it. The work done in exchange is also at the deploy stage. The twenty binding rules are written into the configuration of the target systems and tied to the workload definition; once written, there is nothing left to watch at runtime. The only measurement that falls to runtime is the same as the one written in the first lesson: comparing the set of pairs the binding rules open with the calls actually made. If thirty-six pairs are opened and twenty are used, sixteen pairs are unused, and this can be seen only from the call log. The remaining eleven items cannot be closed at any stage. Three external providers do not support federation, two signing keys sit on the verifying side, and the cache server does not accept identity. All of these items fall back on the procedures of the previous two lessons — the rotation calendar, the overlap window — and on the leak flow that is the subject of the next lesson. #### Summary - Workload identity removes 20 of the 31 items and 11 remain. The remainder gathers in six secrets: two signing keys, three external gateway keys, and the cache server. - The distinction has a reason. The service-to-service call signature is an authorization and is removed; the session and document signature requires the key itself on the verifying side and remains. Of the four external gateways only one supports federation. - While the item count falls by 65%, the number of apps carrying a secret falls only from 9 to 7. The work of secrets management is proportional not to the item count but to the number of apps carrying a secret; the procedure of seven apps stays in place. - The total exposure window falls from 3,123 item-days to 1,228 (61%), but the mean window per item rises from 101 days to 112: the removed items were internal items, and the remaining ones are external keys. - The binding rule that replaces the secret does not reduce the count, it changes the quality: an exposure of 1,895 item-days falls to 0.1 item-days. But when the rule is written at the group level it opens 36 pairs with 16 lines; at the workload level 20 pairs with 20 lines. The 16 unused pairs are the same form as the 48 unused grants of the first lesson. - The dependency does not go away, it rises to two: 7 apps are bound to the secret store, 7 apps to the identity provider, 5 to both; the number of apps bound to neither is zero. #### Next Step This lesson removed most of the secrets, the previous two lessons shortened the life of the rest and wrote the rotation procedure. All three left the same question unanswered: **what happens when a secret leaks.** The exposure window was the time for a secret to become invalid on its own; if a leak is noticed and revoked, the window closes earlier. But noticing takes time, finding the owner takes time, producing the new one takes time, and the propagation of the new value takes as long as the overlap measured in the previous lesson. The next lesson runs this flow over the six remaining secrets: how many minutes pass from the moment of the leak to revocation, how many calls pass through the secret in those minutes, what shortening which stage gains, and how many legitimate requests are cut by revoking without waiting for propagation. ### Leak Response Source: https://academia.sh/en/courses/runtime-security/leak-response Leak response is run as a revocation flow: across the remaining six static secrets, 1,386 minutes pass on average from the leak to revocation and 4,650 minutes in the worst item, and 224,171 calls pass through the secrets in that time. An ownership record gains 340 minutes and a pre-staged replacement gains 465; continuous repository scanning gains nothing because only two of the six leaks land in a repository. Revoking without waiting for propagation saves 150,934 calls but cuts 68,503 legitimate requests — one cut request for every 2.2 calls saved. The first four lessons of the topic measured the number, the life, and the change cost of secrets, and left the same question unanswered: what happens when a secret leaks. The exposure window was the time it takes for the secret to become invalid **on its own**; if the leak is detected and revoked, the window closes earlier. This lesson measures that "earlier." What is measured is the revocation flow, and the flow is on the defense side. Nothing is written about what can be done with a leaked secret or how such a secret would be found; the only things written are the minutes from the detection of the leak to the invalidation of the old value and the call volume that passes through the secret in those minutes. The measurement runs on the **six static secrets** left over from the previous lesson. - **SM23.** The model of the previous four lessons continues. The environment is **fiction**, what is measured is a **model** written with `node`, and it **carries no real secret value**; the model holds only secret paths and flow durations. - **SM24.** The leak channel of each secret is chosen in the model and determines the detection path: a leak that lands in the code repository is detected by continuous scanning, one that lands in a log file by usage anomaly, one at a third party by provider notice, and one in a client bundle by external report. - **SM25.** The flow has five stages: detection, confirmation and impact assessment, finding the owner, generating the new value, propagation of the new value. The propagation time is the minimum overlap of the previous lesson. - **SM26.** "Calls passing through the secret" is the total call volume that the secret authorized during the exposure window; it is an upper bound that shows the **scale** of the window, not the volume that was abused. - **SM27.** There is no randomness; all numbers are derived from the model tables and the output is the same from run to run. #### How Many Minutes from Detection to Revocation ```js // revocation.mjs — minutes from the leak to revocation and calls passing through the secret in that time. A MODEL. // The environment is FICTION; THERE IS NO REAL SECRET VALUE, only secret paths and flow durations are modeled. // How a secret would be used is not written; what is measured is the revocation flow on the defense side. // Six static secrets left over from the previous lesson. channel: where the leak lands (chosen in the model). // owner: whether the owner is written in the secret record. external: whether an external provider generates the new value. // calls: daily calls the secret authorizes. spread: minimum overlap of the previous lesson (min). export const SECRETS = { "infra/cache": { channel: "repository", owner: true, external: false, calls: 240000, spread: 60 }, "signing/session-signing": { channel: "log", owner: true, external: false, calls: 180000, spread: 960 }, "signing/document-signing": { channel: "repository", owner: true, external: false, calls: 6200, spread: 240 }, "external/payment-gateway": { channel: "third-party", owner: true, external: true, calls: 3400, spread: 240 }, "external/email-gateway": { channel: "client-bundle", owner: false, external: true, calls: 13400, spread: 1440 }, "external/sms-gateway": { channel: "log", owner: false, external: true, calls: 8900, spread: 960 }, }; // the detection path depends on the channel: only a leak that lands in the code repository is caught by continuous scanning. export const DETECTION = { repository: ["continuous scanning", 20], log: ["usage anomaly", 90], "third-party": ["provider notice", 240], "client-bundle": ["external report", 2880] }; export const CONFIRM = 30; // confirmation and impact assessment (min) export const OWNED = 10, UNOWNED = 180; // finding the owner (min) export const PRODUCTION = 45, EXTERNAL_PRODUCTION = 120; // generating the new value (min) export const NAMES = Object.keys(SECRETS); export const stages = (s, o = {}) => { const k = SECRETS[s]; return { detection: o.scan && k.channel === "repository" ? 20 : DETECTION[k.channel][1], confirmation: CONFIRM, owner: k.owner || o.record ? OWNED : UNOWNED, production: o.staged ? 5 : k.external ? EXTERNAL_PRODUCTION : PRODUCTION, spread: o.noWait ? 0 : k.spread }; }; export const total = (s, o) => Object.values(stages(s, o)).reduce((t, x) => t + x, 0); export const callsUntil = (s, o) => Math.round(SECRETS[s].calls * total(s, o) / 1440); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const row = gen([18, 22, 8, 9, 8, 10, 11, 8]); console.log(`revocation flow — ${NAMES.length} static secrets (fictional production environment); durations in minutes\n`); console.log(row(["secret", "detection path", "detect", "confirm", "owner", "generate", "propagate", "total"])); for (const s of NAMES) { const a = stages(s); console.log(row([s.split("/")[1], DETECTION[SECRETS[s].channel][0], a.detection, a.confirmation, a.owner, a.production, a.spread, total(s)])); } const T = NAMES.reduce((t, s) => t + total(s), 0), C = NAMES.reduce((t, s) => t + callsUntil(s), 0); console.log(row(["total", "", "", "", "", "", "", T])); console.log(`\n${gen([18, 12, 12, 14, 16])(["secret", "total min", "= hours", "daily calls", "until revoked"])}`); for (const s of [...NAMES].sort((x, y) => callsUntil(y) - callsUntil(x))) console.log(gen([18, 12, 12, 14, 16])([s.split("/")[1], total(s), (total(s) / 60).toFixed(1), SECRETS[s].calls, callsUntil(s)])); console.log(gen([18, 12, 12, 14, 16])(["total", T, (T / 60).toFixed(1), "", C])); console.log(`\naverage per secret ${Math.round(T / NAMES.length)} min ` + `(${(T / NAMES.length / 60).toFixed(1)} hours); longest ${Math.max(...NAMES.map((s) => total(s)))} min`); console.log(`if the same leak were in a short-lived credential the revocation flow would never run, ` + `the window would stay at 60 min: ${Math.round(Math.max(...NAMES.map((s) => total(s))) / 60)} times`); } ``` ``` revocation flow — 6 static secrets (fictional production environment); durations in minutes secret detection path detect confirm owner generate propagate total cache continuous scanning 20 30 10 45 60 165 session-signing usage anomaly 90 30 10 45 960 1135 document-signing continuous scanning 20 30 10 45 240 345 payment-gateway provider notice 240 30 10 120 240 640 email-gateway external report 2880 30 180 120 1440 4650 sms-gateway usage anomaly 90 30 180 120 960 1380 total 8315 secret total min = hours daily calls until revoked session-signing 1135 18.9 180000 141875 email-gateway 4650 77.5 13400 43271 cache 165 2.8 240000 27500 sms-gateway 1380 23.0 8900 8529 payment-gateway 640 10.7 3400 1511 document-signing 345 5.8 6200 1485 total 8315 138.6 224171 average per secret 1386 min (23.1 hours); longest 4650 min if the same leak were in a short-lived credential the revocation flow would never run, the window would stay at 60 min: 78 times ``` The upper table shows where the flow is spent. The average per secret is one thousand three hundred eighty-six minutes, that is twenty-three hours; but between the rows, from one hundred sixty-five to four thousand six hundred fifty minutes, there is a twenty-eight-fold difference. Looking at the columns, the whole of the difference comes from two stages. Confirmation is thirty minutes in every row, finding the owner is ten or one hundred eighty, generating is forty-five or one hundred twenty. The detection column, though, goes from twenty to two thousand eight hundred eighty — a one-hundred-forty-four-fold range — and the propagation column from sixty to one thousand four hundred forty. **What determines the length of the flow is not the speed of the response team but how the leak is detected and how slowly the new value propagates.** Both are fixed before the leak. The lower table converts the same rows into call volume and breaks the ordering. In duration, the worst item is the email gateway key, at four thousand six hundred fifty minutes; but the most calls pass through the session signing key by the time of revocation: one hundred forty-one thousand eight hundred seventy-five. That key authorizes one hundred eighty thousand calls a day, and one thousand one hundred thirty-five minutes is a long time at that volume. At the opposite end is the document signing key: in three hundred forty-five minutes only one thousand four hundred eighty-five calls. The same flow, the same procedure — and the result differs ninety-six-fold between two items, because the traffic behind them differs. The last line ties back to the second lesson of this topic. If the same leak were in a short-lived credential, no step of this flow would run: it would not need to be detected, its owner would not need to be found, a new one would not need to be generated; the token would become invalid on its own in sixty minutes. The difference from the worst item is seventy-eight-fold. The revocation flow is the price paid for secrets whose life cannot be shortened. #### What Shortening Which Stage Gains ```js // speedup.mjs — what shortening which stage of the flow gains, and what it cuts in return import { NAMES, SECRETS, DETECTION, stages, total, callsUntil, gen } from "./revocation.mjs"; // if the old value is revoked without waiting for the switch to the new one, readers are cut for the switch duration. // [app, switch (min), daily requests] — from the switch table of the previous lesson. const READS = { "infra/cache": [["portal-api", 60, 96000]], "signing/session-signing": [["portal-frontend", 960, 84000], ["portal-api", 60, 96000]], "signing/document-signing": [["billing-service", 240, 6200]], "external/payment-gateway": [["billing-service", 240, 6200], ["payment-bridge", 0, 3400]], "external/email-gateway": [["subscriber-signup", 960, 1400], ["notification", 15, 12000], ["report-generator", 1440, 320]], "external/sms-gateway": [["subscriber-signup", 960, 1400], ["notification", 15, 12000]], }; const cut = (o) => o.noWait ? Math.round(NAMES.reduce((t, s) => t + READS[s] .reduce((x, [, g, i]) => x + i * g / 1440, 0), 0)) : 0; const SETTINGS = [["today", {}, "-"], ["+ ownership record", { record: 1 }, "deployment"], ["+ pre-staged replacement", { record: 1, staged: 1 }, "deployment"], ["+ continuous repository scanning", { record: 1, staged: 1, scan: 1 }, "build"], ["+ revoke without waiting for propagation", { record: 1, staged: 1, scan: 1, noWait: 1 }, "runtime"]]; const row = gen([43, 11, 11, 15, 16]); console.log(row(["measure", "total min", "gain", "calls passed", "cut requests"])); const T0 = NAMES.reduce((t, s) => t + total(s), 0); for (const [name, o] of SETTINGS) { const T = NAMES.reduce((t, s) => t + total(s, o), 0); console.log(row([name, T, T0 - T, NAMES.reduce((t, s) => t + callsUntil(s, o), 0), cut(o)])); } const last = SETTINGS[SETTINGS.length - 1][1], prev = SETTINGS[SETTINGS.length - 2][1]; const dC = NAMES.reduce((t, s) => t + callsUntil(s, prev) - callsUntil(s, last), 0); console.log(`\nrevoke without waiting for propagation: saves ${dC} calls, cuts ${cut(last)} legitimate requests ` + `(${(dC / cut(last)).toFixed(1)} calls saved per 1 request cut)`); console.log(`continuous repository scanning gains 0: two secrets already have the repository as their channel and detection is 20 min, ` + `the leak of the other four never lands in a repository`); const row2 = gen([18, 16, 9, 11, 34]); console.log(`\n${row2(["secret", "leak channel", "detect", "calls", "earlier closure"])}`); const EARLIER = { repository: ["build", "pipeline secret scanning"], log: ["code", "masking in the log"], "client-bundle": ["build", "bundle secret scanning"], "third-party": ["none", "-"] }; for (const s of NAMES) console.log(row2([s.split("/")[1], SECRETS[s].channel, stages(s).detection, callsUntil(s), `${EARLIER[SECRETS[s].channel][0]}: ${EARLIER[SECRETS[s].channel][1]}`])); const closable = NAMES.filter((s) => EARLIER[SECRETS[s].channel][0] !== "none"); console.log(`\nsecrets whose leak channel can be closed earlier: ${closable.length} / ${NAMES.length}; ` + `the detection latency of these channels totals ` + `${closable.reduce((t, s) => t + stages(s).detection, 0)} min`); const top = NAMES.reduce((x, s) => stages(s).detection > stages(x).detection ? s : x); console.log(`largest single share: ${top} — ${stages(top).detection} min detection latency and ` + `${Math.round(SECRETS[top].calls * stages(top).detection / 1440)} calls, all of it from the build stage`); ``` ``` measure total min gain calls passed cut requests today 8315 0 224171 0 + ownership record 7975 340 221538 0 + pre-staged replacement 7510 805 207648 0 + continuous repository scanning 7510 805 207648 0 + revoke without waiting for propagation 3610 4705 56714 68503 revoke without waiting for propagation: saves 150934 calls, cuts 68503 legitimate requests (2.2 calls saved per 1 request cut) continuous repository scanning gains 0: two secrets already have the repository as their channel and detection is 20 min, the leak of the other four never lands in a repository secret leak channel detect calls earlier closure cache repository 20 27500 build: pipeline secret scanning session-signing log 90 141875 code: masking in the log document-signing repository 20 1485 build: pipeline secret scanning payment-gateway third-party 240 1511 none: - email-gateway client-bundle 2880 43271 build: bundle secret scanning sms-gateway log 90 8529 code: masking in the log secrets whose leak channel can be closed earlier: 5 / 6; the detection latency of these channels totals 3100 min largest single share: external/email-gateway — 2880 min detection latency and 26800 calls, all of it from the build stage ``` The first three rows give the counterpart of the two items written into the response procedure. Writing the owner into every secret record gains three hundred forty minutes; preparing and keeping ready a replacement value in advance for every secret gains four hundred sixty-five minutes more. Together they bring the eight thousand three hundred fifteen minute flow down to seven thousand five hundred ten — ten percent. Both are cheap and are written before a leak; but the total gain is a tenth of the flow. The fourth row is the kind of result this course looks for: **a measure whose gain is zero.** Continuous repository scanning does not shorten the response time by even a minute. The reason is written in the lower table: of the six leaks, only two have the code repository as their channel, and in those two detection is already twenty minutes. The other four never land in a repository. Increasing the scanning power does not make visible a leak in places the scan never looks at; every improvement made without counting the set of items a defense covers produces this row. The fifth row contains the real decision. Revoking the old value without waiting for the new value to propagate brings the flow down from seven thousand five hundred ten to three thousand six hundred ten, and cuts the calls that pass from two hundred seven thousand to fifty-six thousand — one hundred fifty thousand nine hundred thirty-four calls are saved. In return, sixty-eight thousand five hundred three legitimate requests are cut: one legitimate request for every two point two calls saved. This is the real decision of leak response and it cannot be settled by a procedure item, because the right answer changes with the secret: for the payment gateway key, not waiting for propagation may be reasonable; for the session signing key, it means cutting sixty thousand requests. The last column of the lower table is the axis of the course. Five of the six leak channels can be closed at an earlier stage, and the detection latency of those five channels totals three thousand one hundred minutes. The largest single share is in the email gateway key: because it is embedded in the client bundle of the field app, the leak is noticed only by external report, two days later, and in those two days twenty-six thousand eight hundred calls pass through the key. There is nothing to be done on the response side for this item; the key entered the bundle at the build stage, and with a secret scan there it would never have entered. The same holds for the two keys that land in the log file, one stage earlier: masking in log writing is a code decision. The only item left is the payment gateway key. The leak happened at a third party; neither a build scan nor code masking would have seen it. For this item the revocation flow really is the only way — one of six items. #### Summary - The revocation flow has five stages: detection, confirmation, finding the owner, generating the new value, propagation. Across the remaining six static secrets the total is 8,315 minutes, the average per secret is 1,386 minutes (23.1 hours), and the worst item takes 4,650 minutes. - What determines the duration is not the response speed but the detection and propagation columns: confirmation is 30 minutes in every row, while detection varies between 20 and 2,880 and propagation between 60 and 1,440. - Until revocation, 224,171 calls pass through the secrets, and the ordering is not the same as the duration ordering: the longest flow is the email gateway (4,650 min) but the most calls pass through session signing (141,875). - An ownership record gains 340 minutes and a pre-staged replacement 465 — ten percent of the total. The gain of continuous repository scanning is zero: only two of the six leaks land in a repository and in those two detection is already 20 minutes. - Revoking without waiting for propagation saves 150,934 calls and cuts 68,503 legitimate requests: one cut request for every 2.2 calls saved. The right answer changes with the secret and cannot be written into a procedure. - Five of the six leak channels could have been closed at an earlier stage (3,100 minutes of detection latency in total); the largest single share is the key embedded in the client bundle — 2,880 minutes and 26,800 calls, all of it from the build stage. #### Next Step This topic measured the life of secrets and shortened it: the access matrix went from seventy-nine items to thirty-one, the exposure window shrank two-thousand-fold on sixteen items, twenty items disappeared entirely with workload identity, and the revocation flow was written minute by minute for the remaining six secrets. Secrets now have an owner, a life, a rotation schedule, and a leak procedure. Under all of this there is an assumption, and it has not been touched at all. Everything measured in this topic was about a service **proving itself**: a call that carries the right credential passes, one that does not carry it does not. But what happens when a call does carry the credential. When the field app API calls the billing service, the billing service never asks whether that call is **authorized to read billing data**; it sees that the call came from somewhere and fulfills it. Even if the secret is managed correctly, if this stays so, the capture of a single service is a door that opens onto the others: every trust relation among the nine services is a path along which one can move from one service to another. The next topic counts those paths — how many service pairs have no authentication, how many of them mutual authentication closes, and at which stage the ones that remain open could have been closed. ## Infrastructure Posture The runtime posture of the layer beneath the application is measured with numbers: the lateral-movement pairs closed by service-to-service authentication and the items certificate management adds, the processes that deviate from the baseline in container runtime detection along with false positives and detection latency, the legitimate deployments rejected by admission controller rules and the number of teams affected when moving to enforce mode, the findings returned to application teams and the unowned findings in continuous configuration auditing, and how many days rebuild-based updates cut patch lag to and the disruption they bring in return. ### Network Segmentation and Service Zero Trust Source: https://academia.sh/en/courses/runtime-security/network-segmentation-and-service-zero-trust Lateral movement is counted in service pairs: of the nine services' 72 directed pairs, 52 pass the network policy and only 17 correspond to a legitimate call. One-way transport verification and mutual verification without an allow list close zero pairs; a per-caller allow list closes 35 pairs and brings average reach down from 7.1 to 4.1, but all eight of eight are still reachable from the edge service and the secret store is at most two hops away. The price is 31 certificates, 11,315 renewals a year at a short life, and 3.8 legitimate pairs lost per service. The previous topic shortened the life of secrets and wrote the revocation flow after a leak. All of that measurement leaned on a single assumption: the party that reads the secret is the party that is supposed to read it. Nothing in the production environment tests that assumption. If services trust one another without question, the compromise of a single service is a door open to the others, and shortening the life of a secret does not close that door — it only limits how long the key in the hand of whoever walks through it will keep working. This topic measures the layer beneath the application, and the first lesson counts **lateral movement**. From here on the unit of measurement is the **service pair**: how many directed pairs exist, how many correspond to a legitimate call, which verification scheme closes how many. The rule count of network segmentation, the label width and the growth rate of the matrix were measured in the **Container Orchestration** and **Network Security** courses; they are not repeated here, they are taken as **input**. - **IP1.** The environment is the production environment of the fictional regional measurement network: nine services, one secret store, one cluster. No real cluster is run; what is measured is a model built with `node`. - **IP2.** A call is directed: `a->b` and `b->a` are separate pairs. Endpoints, port numbers and protocols are not modeled; they would enlarge the matrix and would not change the direction of the result. - **IP3.** The set of legitimate calls is seventeen and was read from the call contract. In reality this list can be extracted only with a long observation window or from the design document. - **IP4.** The network policy is written at the label level and is **taken as given**; it has five rules and four labels. - **IP5.** Compromise is a model: process control of one service falls into the attacker's hands and every pair that service is open on becomes usable. No exploitation step, call sequence or escape method is written. - **IP6.** The secret store is a node that makes no calls; four services read it legitimately, and the network policy lets all nine reach it. #### The Pair Set and Four Schemes ```js // services.mjs — nine services, the pair set and the verification schemes. IT IS A MODEL, not a real network. const splitRows = (s) => s.trim().replace(/\s{2,}/g, "\n").split("\n").map((l) => l.trim().split("|")); // Service: name|zone labels (segmentation was measured in earlier courses, here it is INPUT). export const SERVICES = Object.fromEntries(splitRows(` portal-frontend|edge subscriber-service|app billing|app work-order|app,collector-client notification|app report|app,collector-client measurement-collector|collection validator|collection field-sync|collection`).map(([a, e]) => [a, e.split(",")])); // Legitimate call pair: caller|callee. Seventeen pairs, from the call contract. export const LEGIT = splitRows(` portal-frontend|subscriber-service portal-frontend|billing portal-frontend|work-order portal-frontend|report subscriber-service|notification measurement-collector|validator validator|billing validator|measurement-collector billing|subscriber-service billing|notification work-order|field-sync work-order|notification field-sync|measurement-collector field-sync|work-order report|billing report|measurement-collector notification|subscriber-service`).map(([a, b]) => `${a}->${b}`); // The network policy is written at the label level: target label <- source label. export const NET_RULES = [["app", "edge"], ["app", "app"], ["app", "collection"], ["collection", "collection"], ["collection", "collector-client"]]; export const NAMES = Object.keys(SERVICES); export const allPairs = () => NAMES.flatMap((a) => NAMES.filter((b) => b !== a).map((b) => `${a}->${b}`)); export function netOpen(s = SERVICES) { const open = new Set(), names = Object.keys(s); for (const [t, f] of NET_RULES) for (const target of names) if (s[target].includes(t)) for (const source of names) if (source !== target && s[source].includes(f)) open.add(`${source}->${target}`); return open; } // Scheme: how much the called service verifies who the caller is. // none : the caller identity is never verified // one-way : only the called party proves itself, the caller identity is still not verified // mutual : the caller proves itself too, but any identity from the internal authority is accepted // authorized: mutual verification + a per-caller allow list export function pairs(scheme, net = netOpen()) { if (scheme === "authorized") return new Set(LEGIT.filter((c) => net.has(c))); return new Set(net); } // The set of services reached when one service is compromised, and the hop counts (breadth-first search). export function spread(start, edges) { const distance = { [start]: 0 }, queue = [start]; while (queue.length) { const n = queue.shift(); for (const c of edges) { const [f, t] = c.split("->"); if (f === n && !(t in distance)) { distance[t] = distance[n] + 1; queue.push(t); } } } return distance; } export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(""); // The secret store is a separate node and makes no calls. Four services read it legitimately; // the network policy lets all nine reach it (the store label is open to app and collection). export const SECRET_READERS = new Set(["subscriber-service", "billing", "measurement-collector", "field-sync"]); export function secretHops(start, edges, authorized) { const u = spread(start, edges); const reachable = authorized ? [...SECRET_READERS] : NAMES; const d = reachable.filter((s) => s in u).map((s) => u[s] + 1); return d.length ? Math.min(...d) : Infinity; } ``` The body of the `pairs` function is short because what is measured is short: three of the four schemes return the same set. This is not a shortcoming, it is the result of the lesson. #### The Closed Pairs ```js // run1.mjs — four verification schemes run on the same pair set import { NAMES, LEGIT, allPairs, netOpen, pairs, spread, secretHops, gen } from './services.mjs'; const net = netOpen(); console.log(`${NAMES.length} services -> ${allPairs().length} directed pairs; network policy allows ${net.size} pairs`); console.log(`legitimate pairs in the call contract ${LEGIT.length}, all pass the network policy: ` + `${LEGIT.every((c) => net.has(c)) ? 'yes' : 'no'}`); console.log(`pairs that pass the network policy but match no legitimate call: ${net.size - LEGIT.length}\n`); const fmtScheme = gen([28, 11, 10, 15, 17, 15]); console.log(fmtScheme(['scheme', 'open pairs', 'closed', 'avg reached', 'lowest reach', 'secret hops'])); for (const [name, scheme] of [['no verification', 'none'], ['one-way transport', 'one-way'], ['mutual, no allow list', 'mutual'], ['mutual + allow list', 'authorized']]) { const k = pairs(scheme, net), authorized = scheme === 'authorized'; const reached = NAMES.map((s) => Object.keys(spread(s, k)).length - 1); const hops = NAMES.map((s) => secretHops(s, k, authorized)); console.log(fmtScheme([name, k.size, net.size - k.size, (reached.reduce((a, b) => a + b, 0) / NAMES.length).toFixed(1), Math.min(...reached), `${Math.min(...hops)}-${Math.max(...hops)}`])); } const authorizedEdges = pairs('authorized', net); console.log('\nspread per service with the allow list on:'); const fmtSpread = gen([23, 11, 9, 16]); console.log(fmtSpread(['compromised service', 'reached', 'before', 'secret hops'])); for (const s of NAMES) { const after = Object.keys(spread(s, authorizedEdges)).length - 1; const before = Object.keys(spread(s, net)).length - 1; console.log(fmtSpread([s, after, before, `${secretHops(s, net, false)} -> ${secretHops(s, authorizedEdges, true)}`])); } ``` ``` 9 services -> 72 directed pairs; network policy allows 52 pairs legitimate pairs in the call contract 17, all pass the network policy: yes pairs that pass the network policy but match no legitimate call: 35 scheme open pairs closed avg reached lowest reach secret hops no verification 52 0 7.1 7 1-1 one-way transport 52 0 7.1 7 1-1 mutual, no allow list 52 0 7.1 7 1-1 mutual + allow list 17 35 4.1 1 1-2 spread per service with the allow list on: compromised service reached before secret hops portal-frontend 8 8 1 -> 2 subscriber-service 1 7 1 -> 1 billing 2 7 1 -> 1 work-order 6 7 1 -> 2 notification 1 7 1 -> 2 report 5 7 1 -> 2 measurement-collector 4 7 1 -> 1 validator 4 7 1 -> 2 field-sync 6 7 1 -> 1 ``` The first three rows carry the same numbers and this is where the table has to be read. **One-way transport verification closes zero pairs.** The called party proves itself, the caller proves nothing; since who is calling is not known, no pair can be rejected. The difference between the second and third rows is sharper: **mutual verification on its own also closes zero pairs.** Every service carries a valid workload identity, so a compromised service also presents a valid identity. If you ask the verification question as "is it a valid internal identity", the answer is always yes. The one row that closes is the fourth: a per-caller allow list. Thirty-five of fifty-two pairs close and the average number of reached services drops from 7.1 to 4.1. This is not a closing that can be thought free because thirty-five pairs **correspond to no legitimate call**; the list itself is seventeen rows, and someone has to write those seventeen rows. #### The Path That Does Not Close The lower table shows where the gain is distributed, and the distribution is not even. When the notification service is compromised, the number of reached services drops from seven to **one**; so does the subscriber service. These are the leaves of the call graph and the allow list really does isolate them. When the portal frontend is compromised, on the other hand, the number drops from eight to **eight**: it does not change at all. The edge service makes four legitimate calls, those four call the others, and the chain covers the whole cluster. The allow list closes only the edges of the call graph that **do not exist**; it does not stop a spread that walks over the edges that do. The secret hops column ties the same limit to a number. With no verification the secret store is **one** hop from every service. With the allow list on it is still one hop for four services — they read the store legitimately — and two hops for the remaining five. So the highest value goes from one to two. Lateral movement does not close, it **lengthens**: the attacker first has to use an intermediate service, and the use of that intermediate service produces a log line. That is the gain, and it has to be named correctly — thirty-five pairs closed, the path to the secret store did not. #### What Verification Brings ```js // run2.mjs — the items the allow list and certificate management bring import { NAMES, LEGIT, SERVICES, netOpen, gen } from './services.mjs'; const REPLICAS = { 'portal-frontend': 6, 'subscriber-service': 4, 'billing': 3, 'work-order': 3, 'notification': 2, 'report': 2, 'measurement-collector': 6, 'validator': 3, 'field-sync': 2 }; const replicaTotal = Object.values(REPLICAS).reduce((a, b) => a + b, 0); console.log(`${NAMES.length} services, ${replicaTotal} replicas; each replica carries a workload identity`); const fmtCert = gen([18, 13, 17, 13, 16]); console.log('\n' + fmtCert(['certificate life', 'certificates', 'annual renewals', 'manual items', 'exposure window'])); for (const [name, days, manual] of [['90 days', 90, replicaTotal], ['24 hours', 1, 0], ['1 hour', 1 / 24, 0]]) console.log(fmtCert([name, replicaTotal, Math.round((365 / days) * replicaTotal), manual, `${(days * 24).toFixed(0)} h`])); console.log(`allow list ${LEGIT.length} rows; network policy 5 rules and ` + `${new Set(Object.values(SERVICES).flat()).size} labels; root trust anchor 1 item, rotated by hand once a year`); console.log('\nlegitimate pairs lost if the certificate of a service cannot be renewed:'); const fmtLost = gen([23, 10, 11, 9]); console.log(fmtLost(['service', 'caller', 'callee', 'total'])); const lost = NAMES.map((s) => ({ s, c: LEGIT.filter((x) => x.startsWith(`${s}->`)).length, h: LEGIT.filter((x) => x.endsWith(`->${s}`)).length })).sort((a, b) => b.c + b.h - a.c - a.h); for (const d of lost.slice(0, 3)) console.log(fmtLost([d.s, d.c, d.h, d.c + d.h])); const last = lost[lost.length - 1]; console.log(`average ${(lost.reduce((a, b) => a + b.c + b.h, 0) / NAMES.length).toFixed(1)} pairs, ` + `lowest ${last.c + last.h} pairs (${last.s})`); const NEW_PAIRS = ['work-order->archive', 'archive->billing']; const before = netOpen().size, after = netOpen({ ...SERVICES, archive: ['app'] }).size; console.log('\nwhen a tenth service (archive, 2 replicas, app label) is added:'); console.log(` network rules 5 -> 5, pairs opened by the network policy ${before} -> ${after}`); console.log(` allow list ${LEGIT.length} -> ${LEGIT.length + NEW_PAIRS.length} rows, certificates ${replicaTotal} -> ${replicaTotal + 2}`); console.log(` unneeded open pairs of the new service: ${after - before - NEW_PAIRS.length} under the network scheme, 0 under the allow list`); ``` ``` 9 services, 31 replicas; each replica carries a workload identity certificate life certificates annual renewals manual items exposure window 90 days 31 126 31 2160 h 24 hours 31 11315 0 24 h 1 hour 31 271560 0 1 h allow list 17 rows; network policy 5 rules and 4 labels; root trust anchor 1 item, rotated by hand once a year legitimate pairs lost if the certificate of a service cannot be renewed: service caller callee total billing 2 3 5 portal-frontend 4 0 4 subscriber-service 1 3 4 average 3.8 pairs, lowest 3 pairs (field-sync) when a tenth service (archive, 2 replicas, app label) is added: network rules 5 -> 5, pairs opened by the network policy 52 -> 66 allow list 17 -> 19 rows, certificates 31 -> 33 unneeded open pairs of the new service: 12 under the network scheme, 0 under the allow list ``` A ninety-day certificate life produces one hundred twenty-six renewals a year, and if they are done by hand thirty-one items enter the calendar. Cutting the life to twenty-four hours zeroes the manual items because renewals at this count cannot be done by hand; in return annual renewals climb to eleven thousand and an automated distribution mechanism becomes mandatory. A one-hour life asks two hundred seventy thousand renewals on the same mechanism and cuts the exposure window from twenty-four hours to one hour. The three rows are three values of the same setting; which one is chosen is determined by looking at how long an identity can be abused. The middle table counts the failure face of this mechanism. If the certificate of the billing service cannot be renewed, five legitimate pairs fall — the two calls it makes and the three it receives. The average is three point eight pairs and even the lowest value is three. With verification on, a renewal failure does not look like a security incident; it looks like an outage, and its cause is first sought in the application layer. This is exactly the price of closing for legitimate work: in return for the thirty-five closed pairs, the closing mechanism itself is a new source of failure that can drop three to five legitimate pairs. The last block adds the tenth service. The network policy does not change at all — five rules, five rules — but the pairs it opens rise from fifty-two to sixty-six: while the need of the new service is two pairs, twelve unneeded pairs open on their own. Under the allow list scheme two rows are written and unneeded open pairs stay zero. A policy written with labels adapts to growth without effort and accumulates openness; a per-caller list asks two rows for each service and does not accumulate openness. #### Where It Would Have Closed Earlier The seventeen rows in the allow list look like a runtime configuration but their content does not belong to runtime. That list is the list of which service calls which service, and this information is born at **design time**. The moment the call contract is written the seventeen pairs are known; the thirty-five unneeded pairs come from the contract having been written nowhere. The difference between the prices of closing the same item at two stages can be read in the table. Closing at design time is a document and a review. Closing at runtime is seventeen policy rows, thirty-one certificates, eleven thousand renewals a year, a root trust anchor, a rotation calendar and a source of failure that can drop three to five legitimate pairs per service. The runtime layer must do this, because the contract having been written does not mean it is enforced — but if the reason the layer was set up is the absence of the contract, the whole of the counted cost could have been closed more cheaply. #### Summary - Nine services produce 72 directed pairs; the network policy allows 52 of them, the legitimate call pairs are 17 and 35 pairs stay open without corresponding to any call. - One-way transport verification and mutual verification without an allow list close **zero pairs**; a compromised service also presents a valid internal identity. - A per-caller allow list closes 35 pairs and brings the average number of reached services from 7.1 to 4.1; on leaf services reach drops from seven to one, on the edge service from eight to eight — it does not change at all. - The number of hops to the secret store rises from 1 to at most 2: the path does not close, it lengthens, and the use of the intermediate service produces a log line. - The price is 31 certificates, 126 renewals a year and 31 manual items at a ninety-day life, 11,315 renewals and an automated distribution mechanism at a twenty-four-hour life; the loss of one renewal cuts 3.8 legitimate pairs on average. - When the tenth service is added the label policy opens 12 unneeded pairs and the allow list opens 0 with 2 rows; the content of the list is the call contract, born at design time. #### Next Step All the measurement of this lesson stopped at the service boundary: who can call whom, what happens when a call is rejected. The **inside** of the boundary was never opened. Even if the allow list were written completely, a shell opened inside the service's own pod, an unexpected binary or an unusual file access violates no pair — because it makes no call. The next lesson measures this side: a baseline is extracted for the process behavior of the nine services, the processes that deviate from the baseline are counted over a week of records, and how many of the deviations are real anomalies and how many legitimate changes are separated. How many seconds late detection arrives is written in the same table. ### Container Runtime Detection Source: https://academia.sh/en/courses/runtime-security/container-runtime-detection Four baselines are run on the same cluster over a week of process launch records: the image inventory catches 5 of the 21 anomalous events and misses 16, seven days of learning catches 12 but turns 9 legitimate events into false positives, writing the allow list as pairs brings the false positives down to 4, and writing it as binaries drops the caught events to 6. Detection latency is 17 seconds in stream mode and 47 seconds in batch mode, and only 8 and 6 of the 12 caught events arrive while the process is still running; 90.9% of the 68,746 events the pipeline carries deviates in no baseline, and narrowing the image from 342 binaries to 41 pairs gives the same three numbers without building a learning window. The previous lesson stopped at the service boundary: the allow list per caller closed thirty-five pairs, and the path to the secret store grew without closing. What is **inside** the boundary was never asked. A shell opened inside a service's own pod violates no call pair: neither the network policy nor mutual verification sees it, because there is no call. This lesson measures that side, and the unit of measurement is **the process launch**: how many processes started in a week of records, how many of them deviate from the baseline, and how many of the deviating ones are a real anomaly and how many a legitimate change. Container run options — dropped capabilities, a read-only root, a user ID — were measured in the **Containers** course and are taken here as input. The question is a different one: when a mechanism is built that looks at the **behavior** of the running container, what happens to the three numbers. - **IP7.** The environment is the same as before: nine services, thirty-one replicas, one cluster. No real container is run; what is measured is a model built with `node`. - **IP8.** The record consists only of process launch events; file access and network connections are not modeled. The process class and its weekly count are fixed, every event in the same class carries the same binary and the same life; in reality life is a distribution. - **IP9.** The true class of an event is known in the model. In production that class is known only after review; here it is assumed known so that false positives can be counted. - **IP10.** The anomalous classes are written as **models**: which binary is unexpected in which service and how many seconds the process ran. No exploitation step or evasion method is written. - **IP11.** The learning window is seven days, and it is assumed that no anomalous behavior leaks into that window. Detection latency is the sum of the pipeline stage times; wall-clock time is not measured. #### The Record and the Four Baselines ```js // process.mjs — a week of process launch records and four baselines. IT IS A MODEL, not a real cluster. export const REPLICAS = { 'portal-frontend': 6, 'subscriber-service': 4, 'billing': 3, 'work-order': 3, 'notification': 2, 'report': 2, 'measurement-collector': 6, 'validator': 3, 'field-sync': 2 }; // Number of binaries in the service image (base image + application layer). export const IMAGE_BINARIES = { 'portal-frontend': 34, 'subscriber-service': 41, 'billing': 41, 'work-order': 38, 'notification': 29, 'report': 38, 'measurement-collector': 44, 'validator': 29, 'field-sync': 48 }; export const SERVICES = Object.keys(REPLICAS); export const scope = (s) => (s[1] === '*' ? SERVICES : [s[1]]); // [name, service ('*' = all nine), binary, weekly events, true class, in the image?, // seen in the learning window?, process life (s), earlier closing stage] export const CLASSES = [ ['application entry point', '*', 'app', 620, 'legitimate', 1, 1, 0, '-'], ['health probe', '*', 'health', 62496, 'legitimate', 1, 1, 0.4, '-'], ['log rotator', '*', 'log', 5208, 'legitimate', 1, 1, 1.2, '-'], ['certificate renewal client', '*', 'cert', 217, 'legitimate', 1, 1, 2, '-'], ['nightly backup job', 'measurement-collector', 'backup', 7, 'legitimate', 1, 1, 340, '-'], ['scheduled report generation', 'report', 'report-gen', 168, 'legitimate', 1, 1, 26, '-'], ['monthly reconciliation job', 'billing', 'reconcile', 1, 'legitimate', 1, 0, 880, '-'], ['quarterly archive job', 'report', 'archive', 1, 'legitimate', 1, 0, 410, '-'], ['operator diagnostic shell', 'work-order', 'shell', 4, 'legitimate', 1, 0, 210, '-'], ['transition step of the new release', 'subscriber-service', 'migrate', 3, 'legitimate', 1, 0, 64, '-'], ['shell opened inside the pod', 'validator', 'shell', 6, 'anomalous', 1, 0, 95, 'build'], ['network tool not in the image', 'field-sync', 'net-tool', 3, 'anomalous', 0, 0, 12, 'build'], ['binary brought in from outside', 'portal-frontend', 'temp-binary', 2, 'anomalous', 0, 0, 40, 'none'], ['short-lived timer record', 'work-order', 'timer', 1, 'anomalous', 1, 0, 0.5, 'none'], ['execution inside an interpreter', 'subscriber-service', 'app', 9, 'anomalous', 1, 1, 6, 'code'], ]; // The hand-written allow list in two forms: a (service, binary) pair or the binary name alone. export const PAIR_LIST = new Set(['billing|reconcile', 'work-order|shell']); export const BINARY_LIST = new Set(['reconcile', 'shell']); export const BASELINES = { 'image inventory': (s) => !s[5], '7-day learning': (s) => !s[6], 'learning + pair list': (s) => !s[6] && !PAIR_LIST.has(`${s[1]}|${s[2]}`), 'learning + binary list': (s) => !s[6] && !BINARY_LIST.has(s[2]), }; export function measure(deviates) { let caught = 0, missed = 0, falsePositive = 0, deviating = 0; const alerts = new Set(), real = new Set(); for (const s of CLASSES) { const d = deviates(s); if (d) { deviating += s[3]; alerts.add(`${s[1]}|${s[2]}`); if (s[4] === 'anomalous') real.add(`${s[1]}|${s[2]}`); } if (s[4] === 'anomalous') d ? (caught += s[3]) : (missed += s[3]); else if (d) falsePositive += s[3]; } return { deviating, caught, missed, falsePositive, alerts: alerts.size, real: real.size }; } export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(''); export const total = (f) => CLASSES.filter(f).reduce((a, s) => a + s[3], 0); ``` In this model the baseline is a single thing: which binary is considered legitimate in which service. The four definitions differ only in where that list comes from — what is found in the image, what is observed, and an allow list written at two different widths on top of what is observed. #### The Same Week, Four Baselines ```js // run1.mjs — the same weekly record; the only thing that changes is where the baseline comes from import { CLASSES, SERVICES, REPLICAS, IMAGE_BINARIES, BASELINES, measure, gen, total, scope } from './process.mjs'; const sum = (o) => Object.values(o).reduce((a, b) => a + b, 0); const learned = new Set(CLASSES.filter((s) => s[6]).flatMap((s) => scope(s).map((v) => `${v}|${s[2]}`))); console.log(`${SERVICES.length} services, ${sum(REPLICAS)} replicas; ${total(() => true)} process launches in a week ` + `(${total((s) => s[4] === 'legitimate')} legitimate / ${total((s) => s[4] === 'anomalous')} anomalous)`); console.log(`baseline width: image inventory ${sum(IMAGE_BINARIES)} pairs, learning window ${learned.size} pairs\n`); const hdr = gen([26, 11, 7, 16, 7, 14]); console.log(hdr(['baseline', 'caught', 'missed', 'false positive', 'alert', 'real alert'])); for (const [name, deviates] of Object.entries(BASELINES)) { const r = measure(deviates); console.log(hdr([name, r.caught, r.missed, r.falsePositive, r.alerts, r.real])); } ``` ``` 9 services, 31 replicas; 68746 process launches in a week (68725 legitimate / 21 anomalous) baseline width: image inventory 342 pairs, learning window 38 pairs baseline caught missed false positive alert real alert image inventory 5 16 0 2 2 7-day learning 12 9 9 8 4 learning + pair list 12 9 4 6 4 learning + binary list 6 15 4 5 3 ``` The first row is the place to read exactly. The image inventory baseline produces **no false positives at all** — zero — and on a day-to-day dashboard that looks like a flawless layer. The same row has sixteen missed events, three quarters of the twenty-one anomalous ones. The cause is not a narrow definition but a wide one: the baseline is three hundred forty-two pairs, and because every binary sitting in a service image is counted as legitimate, opening a shell that is already in the image never violates the baseline. Zero false positives here is not a measure of success; it is a sign of how wide the baseline is. The second row cuts the baseline to thirty-eight pairs, and the caught events rise from five to twelve. The price is written in the same row: nine legitimate events turn into false positives, and those nine do not come at random but from **four different kinds**. The monthly reconciliation and the quarterly archive job never entered the learning window; the operator diagnostic shell is something an engineer really does; the transition step is a normal part of a deployment. The alert column gives the number the operator sees: the deviating events fall into eight separate groups and four of them are real, so half of the alerts are records that will be closed after review. #### The Width of the Allow List The third and fourth rows close the same flaw at two different widths. The pair list writes `service|binary`, the binary list writes only the binary name; both close the same two false positives, and the false positives fall from nine to four. The difference is in the caught column. With the pair list the caught events stay at twelve; with the binary list they fall to six. The six lost events are the shell opened in the validator service — the moment `shell` is written into the list, that binary becomes legitimate in **all nine of nine services**. The difference between the two rows is one column, and its price is half of the caught events. Writing an exclusion without counting the set it covers is the same mistake as the endpoint-based exclusion in the first lesson of this course: the list is not a setting but an inventory whose scope has to be measured. #### Latency and the Events the Pipeline Carries Detection latency is the sum of four stages: event buffer, send window, rule evaluation and notification. The send window is zero seconds in stream mode and thirty seconds in batch mode. ```js // run2.mjs — detection latency, the events the pipeline carries and narrowing the baseline from the image import { CLASSES, SERVICES, REPLICAS, IMAGE_BINARIES, BASELINES, measure, gen, total, scope } from './process.mjs'; // Pipeline stages: event buffer 2 s + send window 0/30 s + evaluation 3 s + notification 12 s const STREAM = 2 + 0 + 3 + 12, BATCH = 2 + 30 + 3 + 12; const deviates = BASELINES['learning + pair list']; const caughtClasses = CLASSES.filter((s) => s[4] === 'anomalous' && deviates(s)); const hdr = gen([32, 7, 8, 14, 15]); console.log(hdr(['caught anomalous class', 'events', 'life s', `stream ${STREAM} s`, `batch ${BATCH} s`])); for (const s of caughtClasses) console.log(hdr([s[0], s[3], s[7], s[7] > STREAM ? 'yes' : 'no', s[7] > BATCH ? 'yes' : 'no'])); const count = (b) => caughtClasses.filter((s) => s[7] > b).reduce((a, s) => a + s[3], 0); const caughtTotal = caughtClasses.reduce((a, s) => a + s[3], 0); console.log(`detections arriving while the process is still running: stream ${count(STREAM)}/${caughtTotal} events, batch ${count(BATCH)}/${caughtTotal} events`); const events = total(() => true), replicas = Object.values(REPLICAS).reduce((a, b) => a + b, 0); const health = total((s) => s[2] === 'health'), r = measure(deviates); const rate = events / replicas / (7 * 24 * 3600 / 30), releases = 2 * SERVICES.length; console.log(`\nevents carried by the pipeline ${events}/week; share of the health probe ${health} events ` + `(${(100 * health / events).toFixed(1)}%) and it deviates in no baseline`); console.log(`deviating events in this baseline ${r.deviating} (${(100 * r.deviating / events).toFixed(3)}%); expected events per replica ` + `in a 30 s window ${rate.toFixed(2)}, compression factor ${Math.max(1, rate).toFixed(2)} -> latency ${BATCH} s instead of ${STREAM} s`); console.log(`${releases} releases a week, deployment interval per service ${(7 * SERVICES.length / releases).toFixed(1)} days: if the baseline ` + `is tied to the image digest the 7-day learning window is cut in half, if it is tied to the service every ` + `new binary produces ${total((s) => s[2] === 'migrate')} false positives a week`); const MINIMAL = new Set(CLASSES.filter((s) => s[4] === 'legitimate').flatMap((s) => scope(s).map((v) => `${v}|${s[2]}`))); MINIMAL.delete('work-order|shell'); const m = measure((s) => scope(s).some((v) => !MINIMAL.has(`${v}|${s[2]}`))); const imageTotal = Object.values(IMAGE_BINARIES).reduce((a, b) => a + b, 0); console.log(`\nif the image is narrowed from ${imageTotal} binaries to ${MINIMAL.size} pairs (the diagnostic shell is removed):`); const hdr2 = gen([26, 11, 7, 16, 7]); console.log(hdr2(['baseline', 'caught', 'missed', 'false positive', 'alert'])); for (const [name, o] of [['learning + pair list', r], ['minimal image inventory', m]]) console.log(hdr2([name, o.caught, o.missed, o.falsePositive, o.alerts])); const stage = {}; for (const s of caughtClasses) stage[s[8]] = (stage[s[8]] || 0) + s[3]; console.log(`early closing stage of the ${caughtTotal} caught events: ${Object.entries(stage).map(([a, n]) => `${a} ${n}`).join(', ')}; ` + `the ${r.missed} missed events are a single class: ${CLASSES.find((s) => s[4] === 'anomalous' && !deviates(s))[0]}, early stage code`); ``` ``` caught anomalous class events life s stream 17 s batch 47 s shell opened inside the pod 6 95 yes yes network tool not in the image 3 12 no no binary brought in from outside 2 40 yes no short-lived timer record 1 0.5 no no detections arriving while the process is still running: stream 8/12 events, batch 6/12 events events carried by the pipeline 68746/week; share of the health probe 62496 events (90.9%) and it deviates in no baseline deviating events in this baseline 16 (0.023%); expected events per replica in a 30 s window 0.11, compression factor 1.00 -> latency 47 s instead of 17 s 18 releases a week, deployment interval per service 3.5 days: if the baseline is tied to the image digest the 7-day learning window is cut in half, if it is tied to the service every new binary produces 3 false positives a week if the image is narrowed from 342 binaries to 41 pairs (the diagnostic shell is removed): baseline caught missed false positive alert learning + pair list 12 9 4 6 minimal image inventory 12 9 4 5 early closing stage of the 12 caught events: build 9, none 3; the 9 missed events are a single class: execution inside an interpreter, early stage code ``` The upper table separates what detection is good for. Of the twelve events caught in stream mode, the alert arrives while the process is still running for eight; in batch mode that number falls to six. For the rest there is no point of intervention, not because the alert did not come but because it came **late**: the network tool ran for twelve seconds and exited, the timer record for half a second. For these two classes detection is not a prevention but a record. The middle block writes the price of the pipeline. Sixty-eight thousand seven hundred forty-six events are collected in a week, and ninety point nine percent of them are the health probe: a process that runs every five minutes on every replica and deviates in no baseline. The deviating events are sixteen, that is zero point zero two three percent of the volume collected. The thirty-second batch send does not shrink this volume: because the expected number of events falling into a window per replica is less than one, the compression factor stays at one and the thirty-second latency is bought for nothing. The third line says how long the baseline holds: with eighteen releases a week, if the baseline is tied to the image digest no seven-day window ever closes, and if it is tied to the service the window closes but every new binary produces a false positive. A baseline is not something set up once and left alone; it is a maintenance item tied to the speed of deployment. #### Where It Would Have Closed Earlier The two rows of the last table are the same in all three numbers: twelve caught, nine missed, four false positives. The upper row requires a learning window, an allow list and a maintenance schedule; the lower row requires only that the image be narrowed — to the forty-one pairs the application really runs instead of three hundred forty-two binaries. The same three numbers are obtained without a runtime mechanism, at the **build stage** and permanently, and with one alert group fewer besides. The early-stage distribution says the same result a second time. Nine of the twelve caught events can be closed at the build stage: had the shell not been in the validator, six events, and had the network tool not been in field sync, three events would never have been born. For the remaining three events there is no early stage — the binary brought in from outside and the timer record are items only a running mechanism can see. **The measure of the layer is not twelve, it is three.** The nine missed events say the opposite. Execution inside an interpreter does not launch a new process; the application's own binary is already running. A mechanism that looks at process behavior by definition misses a behavior that spawns no process; this item belongs to the code stage. #### Summary - A week of records has 68,746 process launches; 21 are anomalous. The image inventory baseline, 342 pairs wide, catches 5 events, misses 16 and produces zero false positives — this zero is the sign of a **wide** definition, not a narrow one. - The seven-day learning baseline drops to 38 pairs, raises the caught events to 12 and turns 9 legitimate events into false positives: rare jobs, the operator diagnostic and the transition step of the new release. - When the allow list is written as a `service|binary` pair the false positives fall to 4 and the 12 caught events stay; when written with the binary name alone the false positives are again 4 but the caught events fall to 6. - Detection latency is 17 seconds in stream mode and 47 seconds in batch mode; of the 12 caught events 8 and 6 respectively arrive while the process is still running. Batch send provides no compression at this event density. - 90.9% of the events the pipeline carries are the health probe; the deviating share is 0.023%. With 18 releases a week, a baseline tied to the image digest is never completed. - Narrowing the image to 41 pairs gives the same three numbers at the build stage. 9 of the 12 caught events can be closed at the build stage; the share that can be closed only at runtime is 3 events. #### Next Step All the measurement of this lesson was made **after the container started running**: the process started, the event was collected, the rule was evaluated, and seventeen or forty-seven seconds later an alert was produced. The presence of the shell in the validator and the privileges with which the thirty-one replicas run were known before the run began; all of them are fields that can be read at the moment an object is sent to the cluster. The next lesson measures that moment: admission controller rules are run over twenty-two cluster objects, and when enforce mode is switched on it counts how many deployments would be rejected and how many of those are **legitimate**. Whom and when a rejection touches is written in the same table: how many teams are affected and at which stage of the pipeline the same rule could have run without making anyone wait. ### Admission Controller Source: https://academia.sh/en/courses/runtime-security/admission-controller Ten admission rules are run over 22 cluster objects: of the 35 violations, 22 are defects, 8 are justified needs and 5 are object types the rule does not fit. If enforce mode is switched on all at once, 21 objects are rejected, 5 of them legitimate deployments, and all five teams are affected; when the rules are ordered by defect ratio, the first six rules close 19 of the 22 defects with zero legitimate rejections, and the last three rules reject four legitimate deployments without closing a single defect. An exclusion is 13 rows per object, or 7 rows per namespace covering 38 cells and silently exempting one defect; all ten rules can also run at the pipeline gate, and the defects that can be closed only at the admission gate number 3. The previous lesson measured the behavior of the running container and wrote that detection arrives seventeen or forty-seven seconds late. For nine of the twelve caught events there was an earlier stage: a binary that should not have been in the image could have been removed before the run began. The place where that decision is made is a gate, and that gate has not been measured yet. This lesson measures the **admission controller**: the gate that, when an object is sent to the cluster, looks at the declared fields of the object and gives accept or reject. The mechanics of cluster objects — object types, namespaces, authorization — were measured in the **Container Orchestration** course and are taken here as input. The measure here has three columns: how many defects close, how many **legitimate deployments** are rejected, how many teams are affected. When the third is not written, the first two measure nothing, because moving to enforce mode is not a configuration change but a decision that touches the deployment flow of five teams. - **IP12.** The environment is the same fictional regional measurement network: one cluster, the objects of nine services, scheduled job objects and platform workloads; five teams. The object set is a model built with `node`. - **IP13.** The only thing modeled is which rules an object violates. The field structure of the objects, the relation between a release object and a pod, and the mechanics of namespaces are not modeled. - **IP14.** Violations are divided into three classes: **defect** (an item the owner must fix), **justified** (a documented legitimate need), **mismatch** (the rule does not fit that object type). In production this separation is the result of a review and it is the most expensive step; in the model it is taken as given. - **IP15.** An object is rejected if it violates **one** of the enforced rules; there is no partial acceptance. - **IP16.** Weekly deployment counts are fixed per object. A decimal value represents a rarer frequency: 0.2 means a deployment every five weeks. - **IP17.** The gate's own failure — whether deployment should open or stop when the gate is down — is not modeled; that is a separate decision. #### The Object Set and the Rule Set ```js // objects.mjs — twenty-two cluster objects, ten admission rules and the class of the violations. IT IS A MODEL. export const RULES = ['no resource limit', 'root filesystem not read-only', 'privileged mode', 'host network namespace', 'root user', 'mutable image tag', 'unsigned image', 'no health probe', 'no owner label', 'secret in environment variable']; export const R = Object.fromEntries(RULES.map((a, i) => [`R${i + 1}`, a])); // [name, team, goes through the pipeline?, weekly deployments, {rule: violation class}] // violation class: defect = an item the owner must fix, justified = a documented legitimate // need, mismatch = the rule does not fit this object type (one-off job object). export const OBJECTS = [ ['portal-frontend', 'subscriber', 1, 2, { R6: 'defect' }], ['subscriber-service', 'subscriber', 1, 2, { R1: 'defect', R10: 'defect' }], ['notification', 'subscriber', 1, 1, { R2: 'defect' }], ['portal-batch-job', 'subscriber', 1, 0.5, { R1: 'defect', R8: 'mismatch' }], ['billing', 'billing', 1, 2, { R5: 'defect', R10: 'defect' }], ['report', 'billing', 1, 1, { R6: 'defect' }], ['monthly-reconciliation', 'billing', 1, 0.2, { R8: 'mismatch' }], ['quarterly-archive', 'billing', 1, 0.1, { R8: 'mismatch', R9: 'defect' }], ['work-order', 'field', 1, 2, { R2: 'defect' }], ['field-sync', 'field', 1, 2, { R1: 'defect', R2: 'defect', R7: 'defect' }], ['field-upload-job', 'field', 1, 0.5, { R8: 'mismatch' }], ['measurement-collector', 'measurement', 1, 2, {}], ['validator', 'measurement', 1, 2, { R5: 'defect' }], ['measurement-backup-job', 'measurement', 1, 0.3, { R2: 'defect', R8: 'mismatch' }], ['ingress-controller', 'platform', 1, 0.5, { R4: 'justified' }], ['log-collector', 'platform', 1, 0.5, { R2: 'defect', R3: 'justified' }], ['monitoring-collector', 'platform', 1, 0.5, { R3: 'justified' }], ['certificate-distributor', 'platform', 1, 0.3, { R10: 'defect' }], ['registry-mirror', 'platform', 1, 0.3, { R6: 'defect', R7: 'defect' }], ['backup-driver', 'platform', 0, 0.2, { R1: 'defect', R5: 'defect' }], ['network-plugin', 'platform', 0, 0.2, { R3: 'justified', R4: 'justified', R5: 'justified' }], ['node-agent', 'platform', 0, 0.1, { R3: 'justified', R5: 'justified', R9: 'defect' }], ]; export const TEAMS = [...new Set(OBJECTS.map((n) => n[1]))]; export const violations = (n, E) => Object.entries(n[4]).filter(([k]) => E.includes(k)); // In enforce mode an object is rejected if it violates one of the enforced rules. // If a rejected object has no defect among the enforced rules, this is a LEGITIMATE rejection. export function enforce(E) { const rejected = OBJECTS.filter((n) => violations(n, E).length); const legit = rejected.filter((n) => violations(n, E).every(([, s]) => s !== 'defect')); return { rejected, legit, defects: OBJECTS.reduce((a, n) => a + violations(n, E).filter(([, s]) => s === 'defect').length, 0), teams: new Set(rejected.map((n) => n[1])).size, legitTeams: new Set(legit.map((n) => n[1])).size, deployments: rejected.reduce((a, n) => a + n[3], 0), legitDeployments: legit.reduce((a, n) => a + n[3], 0) }; } export const ALL = Object.keys(R); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(''); ``` The rule set looks only at the **declared fields** of an object; not at something that is running. This is also the basis of the comparison at the end of the lesson. #### Three Modes and the Balance Per Rule ```js // run1.mjs — three modes, the balance per rule and enforcing the rules in order import { OBJECTS, RULES, R, ALL, TEAMS, violations, enforce, gen } from './objects.mjs'; const count = (s) => OBJECTS.reduce((a, n) => a + Object.values(n[4]).filter((x) => x === s).length, 0); const z = enforce(ALL); console.log(`${OBJECTS.length} objects, ${TEAMS.length} teams, ${RULES.length} rules; violations ${count('defect') + count('justified') + count('mismatch')} ` + `(defect ${count('defect')} / justified ${count('justified')} / mismatch ${count('mismatch')})\n`); const hm = gen([10, 15, 12, 18, 16, 17]); console.log(hm(['mode', 'audit records', 'rejected', 'legit. rejection', 'affected teams', 'rejections/week'])); console.log(hm(['off', 0, 0, 0, 0, 0])); console.log(hm(['audit', z.rejected.reduce((a, n) => a + violations(n, ALL).length, 0), 0, 0, 0, 0])); console.log(hm(['enforce', 0, z.rejected.length, z.legit.length, `${z.teams}/${TEAMS.length}`, z.deployments.toFixed(1)])); const balance = ALL.map((k) => ({ k, defects: enforce([k]).defects, legit: enforce([k]).legit.length })); console.log('\n' + gen([6, 32, 9, 18])(['rule', 'name', 'defects', 'legit. rejection'])); for (const b of balance) console.log(gen([6, 32, 9, 18])([b.k, R[b.k], b.defects, b.legit])); const order = [...balance].sort((a, b) => b.defects / (1 + b.legit) - a.defects / (1 + a.legit)).map((b) => b.k); console.log(`\nrules are ordered by the defect/(1+legitimate rejection) ratio and enforced one at a time:`); const hs = gen([8, 32, 16, 12, 18, 14]); console.log(hs(['step', 'rule added', 'defects closed', 'rejected', 'legit. rejection', 'legit. teams'])); for (let i = 1; i <= order.length; i++) { const r = enforce(order.slice(0, i)); console.log(hs([i, R[order[i - 1]], r.defects, r.rejected.length, r.legit.length, r.legitTeams])); } ``` ``` 22 objects, 5 teams, 10 rules; violations 35 (defect 22 / justified 8 / mismatch 5) mode audit records rejected legit. rejection affected teams rejections/week off 0 0 0 0 0 audit 35 0 0 0 0 enforce 0 21 5 5/5 18.2 rule name defects legit. rejection R1 no resource limit 4 0 R2 root filesystem not read-only 5 0 R3 privileged mode 0 4 R4 host network namespace 0 2 R5 root user 3 2 R6 mutable image tag 3 0 R7 unsigned image 2 0 R8 no health probe 0 5 R9 no owner label 2 0 R10 secret in environment variable 3 0 rules are ordered by the defect/(1+legitimate rejection) ratio and enforced one at a time: step rule added defects closed rejected legit. rejection legit. teams 1 root filesystem not read-only 5 5 0 0 2 no resource limit 9 8 0 0 3 mutable image tag 12 11 0 0 4 secret in environment variable 15 13 0 0 5 unsigned image 17 13 0 0 6 no owner label 19 15 0 0 7 root user 22 17 1 1 8 privileged mode 22 18 2 1 9 host network namespace 22 19 3 1 10 no health probe 22 21 5 3 ``` The upper table gives the price of moving to enforce mode all at once. Audit mode produces thirty-five records and stops no deployment; this is the stage where the gate is installed but changes nothing. When enforcement is turned on, twenty-one objects are rejected, eighteen point two deployments a week drop out, and **five of the five teams** are affected. Five of the rejected objects have not a single defect the gate complains about: every rule they violate is either a documented justification or a rule that does not fit that object type. The middle table shows where these five legitimate rejections come from, one by one. Three rules — privileged mode, host network namespace and health probe — **close no defect at all** and produce the entire legitimate rejection. The health probe rule alone gives five legitimate rejections, because all five of its objects are one-off job objects and the rule should never have been applied to them. The root user rule stands between the two: it closes three defects and gives two legitimate rejections. The decision to enforce a rule is the ratio of these two columns, not the name of the rule. #### How Far Staged Enforcement Is Free The lower table orders the rules by that ratio and enforces them one at a time. In the first six steps **legitimate rejection is zero** and the defects closed rise to nineteen: eighty-six percent of the twenty-two defects close without stopping a single legitimate deployment. The seventh step adds the root user rule, closes three more defects to complete the twenty-two, and its price is a single legitimate rejection. The last three steps close zero defects. In the eighth, ninth and tenth rows the defects closed column stays fixed at twenty-two while legitimate rejection rises from one to five and the teams with a legitimate rejection from one to three. These three rules are the source of the gate stopping the deployments of three teams within a week without justification, and the gain of all three is zero. The wrongness of moving to enforce mode all at once is tied to a number here: the same twenty-two defects could be closed by an ordered rollout with a single legitimate rejection. #### The Width of the Exclusion and the Pipeline Gate ```js // run2.mjs — the width of the exclusion and the same rule running at the pipeline gate import { OBJECTS, RULES, TEAMS, gen } from './objects.mjs'; // The exclusion has two granularities: per object (name|rule) or per namespace (team|rule). const keys = (i) => OBJECTS.flatMap((n) => Object.entries(n[4]) .filter(([, s]) => s !== 'defect').map(([k]) => `${n[i]}|${k}`)); const OBJECT_EXCL = new Set(keys(0)), TEAM_EXCL = new Set(keys(1)); const cells = (M, i) => OBJECTS.reduce((a, n) => a + RULES.filter((_, j) => M.has(`${n[i]}|R${j + 1}`)).length, 0); const measure = (excluded) => { const open = (n) => Object.entries(n[4]).filter(([k]) => !excluded(n, k)); const rejected = OBJECTS.filter((n) => open(n).length); return { rejected: rejected.length, legit: rejected.filter((n) => open(n).every(([, s]) => s !== 'defect')).length, defects: OBJECTS.reduce((a, n) => a + open(n).filter(([, s]) => s === 'defect').length, 0), deployments: rejected.reduce((a, n) => a + n[3], 0) }; }; const SETUPS = [['no exclusion', () => false, 0, 0], ['per object', (n, k) => OBJECT_EXCL.has(`${n[0]}|${k}`), OBJECT_EXCL.size, cells(OBJECT_EXCL, 0)], ['per namespace', (n, k) => TEAM_EXCL.has(`${n[1]}|${k}`), TEAM_EXCL.size, cells(TEAM_EXCL, 1)]]; const hx = gen([18, 8, 12, 13, 13, 18, 17]); console.log(hx(['exclusion form', 'rows', 'covered', 'open defects', 'rejected', 'legit. rejection', 'rejections/week'])); for (const [name, f, rows, covered] of SETUPS) { const r = measure(f); console.log(hx([name, rows, covered, r.defects, r.rejected, r.legit, r.deployments.toFixed(1)])); } const pipeline = OBJECTS.filter((n) => n[2]), outside = OBJECTS.filter((n) => !n[2]); const defects = (l) => l.reduce((a, n) => a + Object.values(n[4]).filter((s) => s === 'defect').length, 0); console.log(`\nten of ten rules look at the declared fields of the object: the same rule can run at the pipeline gate`); console.log(` objects through the pipeline ${pipeline.length}/${OBJECTS.length}, defects they carry ${defects(pipeline)}/${defects(OBJECTS)}`); console.log(` objects created outside the pipeline ${outside.length} (${outside.map((n) => n[0]).join(', ')}), defects ${defects(outside)}`); const legitObjects = OBJECTS.filter((n) => Object.values(n[4]).length && Object.values(n[4]).every((s) => s !== 'defect')); const total = OBJECTS.reduce((a, n) => a + n[3], 0), permanent = legitObjects.reduce((a, n) => a + n[3], 0); console.log(`\nweekly deployments ${total.toFixed(1)}; rejected in the week enforcement is turned on ${measure(() => false).deployments.toFixed(1)}`); console.log(` after ${defects(OBJECTS)} defects are fixed, permanent legitimate rejection ${permanent.toFixed(1)}/week, ` + `in ${new Set(legitObjects.map((n) => n[1])).size} of ${TEAMS.length} teams; the cause of its permanence is not a defect but rule scope`); ``` ``` exclusion form rows covered open defects rejected legit. rejection rejections/week no exclusion 0 0 22 21 5 18.2 per object 13 13 22 16 0 16.3 per namespace 7 38 21 16 0 16.3 ten of ten rules look at the declared fields of the object: the same rule can run at the pipeline gate objects through the pipeline 19/22, defects they carry 19/22 objects created outside the pipeline 3 (backup-driver, network-plugin, node-agent), defects 3 weekly deployments 20.2; rejected in the week enforcement is turned on 18.2 after 22 defects are fixed, permanent legitimate rejection 1.9/week, in 3 of 5 teams; the cause of its permanence is not a defect but rule scope ``` The second and third rows of the upper table both zero out legitimate rejection, and both bring the rejected objects down to sixteen. The difference is in two columns. The per-object exclusion is thirteen rows and covers exactly thirteen cells; the per-namespace exclusion is seven rows and covers thirty-eight cells — roughly half the rows, close to three times the coverage. The price is in the third column: the open defects go from twenty-two to twenty-one. The backup driver running as the root user quietly becomes legitimate under the platform exclusion written for the network plugin and the node agent. The number of covered cells includes future objects too: every object created later in that namespace finds the exclusion ready. For this reason the exclusion list is of the same kind as the lists measured in the previous two lessons — not a setting but an inventory that, when its scope is not counted, is more decisive than the rule set. #### Where It Would Have Closed Earlier Ten of the ten rules look at the declared fields of the object; none reads something that is running. This means that all of the rules can run at the **pipeline gate**, before the object is sent to the cluster. The difference is not in the rule itself but in whom the rejection touches and when: at the pipeline gate the rejection reaches the person who wrote the change at merge time; at the admission gate it comes during deployment, often to someone else who happens to be on call at that moment, and it blocks a release window. Nineteen of the twenty-two objects go through the pipeline, and nineteen of the twenty-two defects are in those objects. That is, eighty-six percent of the defects could have been closed at the owner's desk without stopping anyone's deployment. The remaining three defects are in the three platform objects created by hand outside the pipeline, and the pipeline gate by definition never sees them. **The share that belongs to the admission gate alone is these three**; for the other nineteen the gate is the second line of defense, not the first. The last line writes a permanent item. Even after the twenty-two defects are fixed, one point nine deployments a week continue to be rejected, and this concerns three teams. The cause is not a defect but a rule scope that does not match the object type. This item can be closed with an exclusion, and when it is closed an inventory of thirteen or seven rows is born — or the rule is rewritten to read the object type and the inventory is never born. #### Summary - There are 35 violations in 22 objects: 22 defects, 8 documented justifications and 5 objects the rule does not fit. Audit mode produces 35 records and stops no deployment. - Moving to enforce mode all at once rejects 21 objects, drops 18.2 deployments a week and affects 5 of 5 teams; 5 of the rejected objects are **legitimate deployments** that carry no defect. - Three rules (privileged mode, host network namespace, health probe) produce the entire legitimate rejection without closing any defect; the health probe alone gives 5 legitimate rejections. - When the rules are ordered by defect ratio, the first 6 steps close 19 of the 22 defects with **zero legitimate rejections**, and the 7th step closes the remaining 3 defects with a single legitimate rejection; the last 3 steps close zero defects and raise legitimate rejection to 5 and the affected teams to 3. - An exclusion covers 13 cells with 13 rows per object and 38 cells with 7 rows per namespace; the second quietly exempts one real defect and also covers future objects in the namespace. - All ten rules can run at the pipeline gate; 19 of the 22 defects are in objects that go through the pipeline. The defects that can be closed only at the admission gate number 3, and all three are in platform objects created by hand. #### Next Step The gate of this lesson saw only the object that was **sent**. Objects already running in the cluster never pass through the gate again; even if the rule set is tightened today, the twenty-two objects accepted yesterday stay where they are. Infrastructure resources outside the cluster — storage, network, registry, identity definitions — do not pass through any admission gate to begin with. Only a continuously running audit scans them, and that audit produces a list of findings. The next lesson measures that list, but not the number of findings: **whose job each finding falls to**, how many have no owner at all, and at what rate the unowned findings accumulate. ### Continuous Configuration Audit Source: https://academia.sh/en/courses/runtime-security/continuous-configuration-audit The 147 open findings of continuous audit are run through the ownership resolution chain: the owner label resolves 65 of them, namespace ownership 24 and repository path mapping 18, and 40 (27.2%) remain unowned. 81 of the findings fall to application teams and 26 stay with the platform; because the capacity of the 40 unowned findings is zero, the queue grows to 112 in twelve weeks while the backlog of the fixing teams shrinks. 31 of the unowned findings are on resources outside the cluster, and no cluster mechanism can resolve them; 122 of the 147 findings could have been closed at an earlier stage, but for 25 of the 40 unowned findings there is no earlier stage. The gate of the previous lesson saw only the object that was sent, and all ten rules read the declared fields of the object. Even if the gate is tightened today, objects accepted yesterday stay where they are; storage, network and identity definitions outside the cluster pass through no admission gate at all. Only a continuously running audit scans that area. This lesson measures that audit, but not the number of findings. Which misconfiguration closes at which gate was measured in the **Cloud and Virtualization Security** course and is not repeated here. The measure here is a single question: **whose job does a finding fall to.** How many findings return to an application team, how many stay with the platform team and how many have no owner at all. An unowned finding is different from an unmeasured finding: it is visible on the dashboard, it enters the count, it is reported, and no one closes it. - **IP18.** The audit scans the whole of the fictional regional measurement network: cluster objects and the storage, network, identity and registry resources outside the cluster. The finding inventory is a model built with `node`. - **IP19.** The content of the rule set is not measured; the numbers are read from the open finding inventory. As long as the same resource violates the same rule it is counted as a single finding. - **IP20.** The ownership resolution chain has three steps and they are tried in order: the owner label of the resource, the namespace ownership table, the repository path mapping. The first matching step wins; if none matches the finding is unowned. - **IP21.** At which step a finding is resolved and to which team it falls is given in the model. In production this is read from the actual state of the inventory that day and changes over time. - **IP22.** Capacity is fixed per week and counts only the item set aside for these findings. The queue runs by `max(0, queue + new - capacity)`; the difference in how hard the findings are to close is not modeled. #### The Finding Inventory and the Ownership Chain ```js // finding.mjs — the open finding inventory of continuous audit and the ownership chain. IT IS A MODEL. // [name, resource type, open findings, severity, step where ownership resolves, team, earlier closing stage] export const FINDINGS = [ ['no pod resource limit', 'cluster', 18, 'medium', 'label', 'subscriber', 'deploy'], ['root filesystem not read-only', 'cluster', 14, 'medium', 'label', 'field', 'deploy'], ['secret in environment variable', 'cluster', 6, 'high', 'label', 'billing', 'code'], ['mutable image tag', 'cluster', 9, 'low', 'namespace', 'subscriber', 'build'], ['unsigned image', 'registry', 7, 'high', 'namespace', 'field', 'build'], ['broadly accessible storage bucket', 'storage', 11, 'high', 'label', 'measurement', 'design'], ['unencrypted backup volume', 'storage', 5, 'high', 'repo', 'platform', 'design'], ['open management port', 'network', 9, 'high', 'none', '-', 'design'], ['unused identity definition', 'identity', 16, 'low', 'none', '-', 'none'], ['overly permissive role binding', 'identity', 8, 'high', 'namespace', 'platform', 'design'], ['resource with logging disabled', 'network', 13, 'low', 'repo', 'platform', 'deploy'], ['outdated runtime version', 'cluster', 16, 'medium', 'label', 'measurement', 'build'], ['old namespace left without an owner', 'cluster', 9, 'medium', 'none', '-', 'none'], ['hand-created network rule', 'network', 6, 'medium', 'none', '-', 'deploy'], ]; export const CHAIN = ['label', 'namespace', 'repo']; export const STEP_NAME = { label: 'owner label', namespace: 'namespace ownership', repo: 'repo path mapping' }; // The item a team sets aside for security findings per week; the unowned finding has no capacity. export const CAPACITY = { subscriber: 4, billing: 2, field: 3, measurement: 5, platform: 3, unowned: 0 }; export const NEW_PER_WEEK = 22; // new findings per week; tied to the deployment rate, given in the model export const side = (b) => (b[4] === 'none' ? 'unowned' : b[5] === 'platform' ? 'platform' : 'application'); export const bucket = (b) => (b[4] === 'none' ? 'unowned' : b[5]); export const sum = (f) => FINDINGS.filter(f).reduce((a, b) => a + b[2], 0); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(''); ``` The three steps of the chain read from three different places and all three are an **inventory** decision: has a label been put on the resource, is the namespace tied to a team, does the file that defines the resource sit under a repository path. None of them is a decision of the audit itself. #### Whose Job the Finding Falls To ```js // run1.mjs — the ownership chain is run: whose job does the finding fall to import { FINDINGS, CHAIN, STEP_NAME, side, sum, gen } from './finding.mjs'; const SEV = ['high', 'medium', 'low']; const TOTAL = sum(() => true); console.log(`${FINDINGS.length} finding classes; open findings ${TOTAL} ` + `(${SEV.map((s) => `${s} ${sum((b) => b[3] === s)}`).join(' / ')})\n`); const hs = gen([22, 8, 12, 9]); console.log(hs(['resolution step', 'findings', 'cumulative', 'share'])); let k = 0; for (const a of CHAIN) { k += sum((b) => b[4] === a); console.log(hs([STEP_NAME[a], sum((b) => b[4] === a), k, `${(100 * k / TOTAL).toFixed(1)}%`])); } console.log(hs(['unresolved', TOTAL - k, TOTAL, `${(100 * (TOTAL - k) / TOTAL).toFixed(1)}%`])); const ht = gen([22, 10, 9, 8, 9]); console.log('\n' + ht(['who gets the finding', 'findings', ...SEV])); for (const [name, t] of [['application teams', 'application'], ['platform team', 'platform'], ['unowned', 'unowned']]) console.log(ht([name, sum((b) => side(b) === t), ...SEV.map((s) => sum((b) => side(b) === t && b[3] === s))])); const un = FINDINGS.filter((b) => side(b) === 'unowned'); console.log(`unowned findings in ${un.length} classes: ` + un.map((b) => `${b[0]} (${b[2]})`).join(', ')); ``` ``` 14 finding classes; open findings 147 (high 46 / medium 63 / low 38) resolution step findings cumulative share owner label 65 65 44.2% namespace ownership 24 89 60.5% repo path mapping 18 107 72.8% unresolved 40 147 27.2% who gets the finding findings high medium low application teams 81 24 48 9 platform team 26 13 0 13 unowned 40 9 15 16 unowned findings in 4 classes: open management port (9), unused identity definition (16), old namespace left without an owner (9), hand-created network rule (6) ``` The first step of the chain is the strongest: the owner label alone finds the owner of sixty-five findings, forty-four percent of the inventory. The second and third steps resolve twenty-four and eighteen findings more. The three steps together reach seventy-three percent and stop there. The remaining forty findings are items the audit sees but cannot send to anyone. The lower table gives the real distribution. Eighty-one findings return to application teams, twenty-six stay with the platform team and forty are unowned. The severity columns separate these three buckets from each other: half of the platform team's findings are high severity, and the majority of the application teams' findings are medium. The unowned bucket holds nine high severity findings, and all nine are open management ports. This shows that being unowned is not limited to low severity items — the reason a finding is unowned is not that it is unimportant but that the resource has not been tied to anyone in the inventory. #### If the Queue Runs for Twelve Weeks ```js // run2.mjs — the twelve-week course of the queue, the source of unownedness and the early stage import { FINDINGS, CAPACITY, NEW_PER_WEEK, bucket, sum, gen } from './finding.mjs'; const TOTAL = sum(() => true), WEEKS = 12; const hq = gen([12, 7, 15, 10, 14, 11]); console.log(hq(['bucket', 'open', 'new per week', 'capacity', 'net per week', `week ${WEEKS}`])); const last = {}; for (const bk of Object.keys(CAPACITY)) { const open = sum((b) => bucket(b) === bk), added = NEW_PER_WEEK * open / TOTAL, net = added - CAPACITY[bk]; let q = open; for (let w = 0; w < WEEKS; w++) q = Math.max(0, q + net); last[bk] = Math.round(q); console.log(hq([bk, open, added.toFixed(1), CAPACITY[bk], net.toFixed(1), last[bk]])); } const owned = Object.keys(CAPACITY).filter((k) => k !== 'unowned'); const cap = owned.reduce((a, k) => a + CAPACITY[k], 0); const unOpen = sum((b) => bucket(b) === 'unowned'), need = NEW_PER_WEEK * unOpen / TOTAL + unOpen / WEEKS; console.log(`total backlog ${TOTAL} -> ${Object.values(last).reduce((a, b) => a + b, 0)}; owned buckets ` + `${TOTAL - unOpen} -> ${owned.reduce((a, k) => a + last[k], 0)}, unowned bucket ${unOpen} -> ${last.unowned}`); console.log(`clearing the unowned bucket in ${WEEKS} weeks takes ${need.toFixed(1)} items a week: ` + `${(need / Math.max(...owned.map((k) => CAPACITY[k]))).toFixed(1)} times the capacity of the largest single team, ` + `${(100 * need / cap).toFixed(0)}% of the total capacity of the five teams (${cap})`); const un = (f) => sum((b) => bucket(b) === 'unowned' && f(b)); const inside = un((b) => b[1] === 'cluster'), allUn = un(() => true), high = un((b) => b[3] === 'high'); console.log(`\nof the ${allUn} unowned findings ${inside} are in cluster objects, ${allUn - inside} in resources outside the cluster ` + `(${[...new Set(FINDINGS.filter((b) => bucket(b) === 'unowned' && b[1] !== 'cluster').map((b) => b[1]))].join(', ')})`); const LEVERS = [['namespace ownership row', inside, high - un((b) => b[1] === 'cluster' && b[3] === 'high'), 'only cluster objects'], ['owner field in resource inventory', allUn, 0, 'all resources'], ['muting the low severity class', un((b) => b[3] === 'low'), high, 'finding disappears, resource stays'], ['owner label at the admission gate', 0, high, 'only newly created resources']]; const hl = gen([34, 10, 11, 16, 35]); console.log('\n' + hl(['lever', 'resolved', 'remaining', 'remaining high', 'scope'])); for (const [name, c, h, scope] of LEVERS) console.log(hl([name, c, allUn - c, h, scope])); console.log(`the admission gate does not touch the existing ${allUn} findings; it lowers the cluster share of weekly new unowned ` + `from ${(NEW_PER_WEEK * inside / TOTAL).toFixed(1)} to 0`); const STAGES = ['design', 'build', 'code', 'deploy', 'none']; const hg = gen([14, 10, 9]); console.log('\n' + hg(['earlier stage', 'findings', 'unowned'])); for (const s of STAGES) console.log(hg([s, sum((b) => b[6] === s), un((b) => b[6] === s)])); const early = sum((b) => b[6] !== 'none'); console.log(`${early} findings (${(100 * early / TOTAL).toFixed(0)}%) could have been closed at an earlier stage; ` + `for ${un((b) => b[6] === 'none')} of the ${allUn} unowned findings there is no earlier stage`); ``` ``` bucket open new per week capacity net per week week 12 subscriber 27 4.0 4 0.0 27 billing 6 0.9 2 -1.1 0 field 21 3.1 3 0.1 23 measurement 27 4.0 5 -1.0 15 platform 26 3.9 3 0.9 37 unowned 40 6.0 0 6.0 112 total backlog 147 -> 214; owned buckets 107 -> 102, unowned bucket 40 -> 112 clearing the unowned bucket in 12 weeks takes 9.3 items a week: 1.9 times the capacity of the largest single team, 55% of the total capacity of the five teams (17) of the 40 unowned findings 9 are in cluster objects, 31 in resources outside the cluster (network, identity) lever resolved remaining remaining high scope namespace ownership row 9 31 9 only cluster objects owner field in resource inventory 40 0 0 all resources muting the low severity class 16 24 9 finding disappears, resource stays owner label at the admission gate 0 40 9 only newly created resources the admission gate does not touch the existing 40 findings; it lowers the cluster share of weekly new unowned from 1.3 to 0 earlier stage findings unowned design 33 9 build 32 0 code 6 0 deploy 51 6 none 25 25 122 findings (83%) could have been closed at an earlier stage; for 25 of the 40 unowned findings there is no earlier stage ``` The upper table sends six buckets in different directions. The capacity of the billing and measurement teams is above the incoming load, so their backlog shrinks: billing goes to zero in twelve weeks, measurement goes from twenty-seven to fifteen. The subscriber team stands in exact balance. The field and platform teams fall a little behind every week, and the platform's backlog rises from twenty-six to thirty-seven. All of these are manageable numbers, and what they have in common is that each has an owner. The sixth row is something else. In the unowned bucket capacity is zero, because capacity is a concept that belongs to a team and this bucket has no team. The forty findings receive six new findings every week, none ever closes, and in twelve weeks they rise to one hundred twelve. While the total backlog of the five buckets that have an owner falls from one hundred seven to one hundred two in the same period, the total inventory rises from one hundred forty-seven to two hundred fourteen: the whole of the increase comes from a single bucket, and that bucket responds to no fixing effort. The last line writes the price of this. Clearing the unowned bucket in twelve weeks takes a capacity of nine point three items a week. This number is one point nine times the capacity of the largest single team and fifty-five percent of the **total** capacity of the five teams. That is, if the unowned findings were handed over to one team in bulk, that team's whole capacity would not be enough; not even the capacity of the two largest teams would be enough, the share of a third team would be needed, and those teams would have to give up their own findings. The cost of being unowned is not that no one looks, but that the number to be paid if someone were to look is this large. #### Who Will Give the Ownership The middle block separates where the unownedness comes from: of the forty findings only nine are in a cluster object, and thirty-one are in network and identity resources. This separation determines the whole of the lever table. Writing a row into the namespace ownership table resolves nine findings and leaves thirty-one in place — because those resources are not in a namespace. The owner label rule added to the admission gate of the previous lesson touches **none** of the existing forty findings; it affects only newly created cluster objects and lowers the cluster share of weekly new unowned from one point three to zero. Muting the low severity class lowers the inventory from forty to twenty-four but never touches the remaining nine high severity findings; the resource stays where it is, it only becomes invisible. The only row that resolves all forty of the forty is the second: an owner field in the resource inventory. That field is not a mechanism of the cluster and not an output of the audit either — it is metadata filled in when the resource is created. Continuous audit cannot **produce** ownership; it can only report its absence. #### Where It Would Have Closed Earlier Looked at with two columns, the last table gives the result of the lesson. One hundred twenty-two of the hundred forty-seven findings — eighty-three percent — could have been closed at an earlier stage: fifty-one with a manifest check at the deploy stage, thirty-three with an access decision at design, thirty-two with a rebuild at the build stage, six in code. Continuous audit finds these one hundred twenty-two again in every scan and reports them again in every scan. The second column reads in the opposite direction. For twenty-five of the forty unowned findings an earlier stage **does not exist**, and this is not a coincidence: unused identity definitions and the old namespace left without an owner are resources that were born at some stage and forgotten. The decision that produced them belongs not to a stage but to a record that was never kept. The measure of this course — how many times cheaper it is to close an item earlier — takes another form here: ownership is not an item that can be closed at a stage but a field the resource has to carry from the first moment it exists. A finding that appears when the field is left empty becomes cheaper at no stage; it merely accumulates in a list that no one looks at. #### Summary - The ownership of 147 open findings is resolved by a three-step chain: owner label 65, namespace ownership 24, repository path mapping 18. The chain stops at 72.8% and 40 findings (27.2%) remain unowned. - 81 of the findings fall to application teams and 26 stay with the platform team. The unowned bucket has 9 high severity findings; being unowned is not limited to low severity. - The capacity of the unowned bucket is zero: while the 40 findings rise to 112 in twelve weeks, the total backlog of the five owned buckets falls from 107 to 102. Clearing this bucket in the same period takes 9.3 items a week, that is a share of 55% of the total capacity of the five teams. - 31 of the 40 unowned findings are in network and identity resources outside the cluster; namespace ownership resolves only 9 of them, the admission gate touches none of the existing ones, and muting makes 16 of them invisible and leaves the resource in place. The only thing that resolves all forty is an owner field in the resource inventory. - 122 of the 147 findings (83%) could have been closed at an earlier stage; for 25 of the 40 unowned findings there is no earlier stage, because ownership is a decision of resource creation and not of a stage. #### Next Step In the earlier stage column thirty-two findings read **build**: the mutable image tag, the unsigned image and the outdated runtime version. What these three classes have in common is that none of them can be fixed in place — to close them a new image has to be built and thirty-one replicas redeployed. The audit finds them again in every scan, because the distance between scan and fix is not a configuration change but a release cycle. The next lesson measures that cycle: after a fix is published, how many days does it take to reach the replicas, what is the difference in days between in-place patching and rebuilding, and how many replicas does the disruption that rebuilding brings in return touch. ### Rebuild-Based Update Source: https://academia.sh/en/courses/runtime-security/rebuild-based-update The same base image fix is run on thirty-one replicas by three paths and patch lag is measured in days: in-place patching closes the whole fleet in 5.25 days, but protection disappears after 2.61 days on a replica-weighted average and 72.8% of thirty days is spent open; a rebuild tied to a code change closes the last replica in 14.25 days and the open share is 19.4%; a scheduled rebuild closes all nine services in 1.75 days and brings the open share down to 5.8%. The price is a 31.6% rise in weekly replica restarts, from 73.5 to 96.75; services with two replicas lose half their capacity for ninety seconds on every round. The previous lesson split the findings of continuous auditing by owner, and in the early-stage column thirty-two findings read **build**: a mutable image tag, an unsigned image and an outdated runtime version. None of the three can be fixed in place; to close them, a new image has to be built and the thirty-one replicas redeployed. This lesson measures that distance, and the unit of measurement is the **day**: after a fix is published, how many days does it take to start running on the last replica of the fleet, and how many replicas stay open in the meantime. The case for immutable infrastructure — machines drifting apart, the rollback round, local state that cannot be moved — was measured in the **Infrastructure as Code** course and is taken here as an input. The question is a different one: how many days does patch lag drop to, and how many restarts are demanded in return. - **IP23.** The environment is the same: nine services, thirty-one replicas, one cluster. No real build or deployment is run; what is measured is a model built with `node`. All stage durations are given in days in the model and the numbers are derived from them; wall-clock time is not measured. - **IP24.** Deployments per service are evenly spaced and the weekly release total is the eighteen used in the second lesson of the topic. In reality the interval scatters; the distribution lengthens the tail and does not change the ranking. - **IP25.** An in-place patch is applied to the file system of the running pod and is not written to the image; the next deployment brings back the unpatched image. This is the deciding assumption of the model. - **IP26.** In a rolling deployment at most one replica is down at a time and the readiness time is fixed per replica; the restart's own failure probability is not modeled. #### Three Paths and Patch Lag ```js // update.mjs — thirty-one replicas, three update paths and a thirty-day window. IT IS A MODEL. // [service, replicas, weekly releases]; the release total is 18, a distribution of the value in the second lesson of the topic. export const SERVICES = [ ['portal-frontend', 6, 5], ['subscriber-service', 4, 4], ['billing', 3, 3], ['work-order', 3, 2], ['notification', 2, 1], ['report', 2, 1], ['measurement-collector', 6, 1], ['validator', 3, 0.5], ['field-sync', 2, 0.5], ]; export const WINDOW = 30, READY = 45; // days; readiness time per replica (s) export const interval = (s) => 7 / s[2]; export const replicas = SERVICES.reduce((a, s) => a + s[1], 0); // Stage durations (days): in-place patch = package repository 0.5 + approval 1 + maintenance window 3.5 + // application 0.25; rebuild = propagation to the registry 0.5 + build and pipeline 0.25; scheduled // adds the nightly trigger (1) to these. const PATCH = 0.5 + 1 + 3.5 + 0.25, BUILD = 0.25, REGISTRY = 0.5, TRIGGER = 1; const nextDeploy = (s, t) => Math.ceil((t + 1e-9) / interval(s)) * interval(s); // An in-place patch is not written to the image: the next deployment brings back the unpatched image. export function protectedSpans(path, s) { const spans = []; if (path === 'in-place patch') for (let w = PATCH; w < WINDOW; w += 7) spans.push([w, Math.min(nextDeploy(s, w), WINDOW)]); else if (path === 'code-change rebuild') spans.push([Math.min(nextDeploy(s, REGISTRY) + BUILD, WINDOW), WINDOW]); else spans.push([Math.min(REGISTRY + TRIGGER + BUILD, WINDOW), WINDOW]); return spans.filter(([b, e]) => e > b); } export const PATHS = ['in-place patch', 'code-change rebuild', 'scheduled rebuild']; export const protectedDays = (path, s) => protectedSpans(path, s).reduce((a, [b, e]) => a + e - b, 0); export const firstSpan = (path, s) => protectedSpans(path, s)[0] ?? [WINDOW, WINDOW]; export const openDays = (path) => SERVICES.reduce((a, s) => a + s[1] * (WINDOW - protectedDays(path, s)), 0); export const gen = (B) => (h) => h.map((v, j) => j ? String(v).padStart(B[j]) : String(v).padEnd(B[0])).join(''); ``` ```js // run1.mjs — same patch, three paths: the day the last replica closes and open replica-days import { SERVICES, PATHS, WINDOW, replicas, interval, protectedDays, firstSpan, openDays, gen } from './update.mjs'; console.log(`${SERVICES.length} services, ${replicas} replicas, ${SERVICES.reduce((a, s) => a + s[2], 0)} releases per week; ` + `deployment interval between ${Math.min(...SERVICES.map(interval)).toFixed(2)} and ${Math.max(...SERVICES.map(interval))} days\n`); const fmtMain = gen([32, 14, 14, 19, 11]); console.log(fmtMain(['path', 'first replica', 'last replica', 'open replica-days', 'open share'])); for (const p of PATHS) { const g = SERVICES.map((s) => firstSpan(p, s)[0]); console.log(fmtMain([p, Math.min(...g).toFixed(2), Math.max(...g).toFixed(2), openDays(p).toFixed(1), `${(100 * openDays(p) / (replicas * WINDOW)).toFixed(1)}%`])); } const fmtSvc = gen([23, 10, 10, 14, 17]); console.log('\n' + fmtSvc(['service', 'replicas', 'interval', 'patch holds', 'rebuild 1st day'])); for (const s of SERVICES) { const [b, e] = firstSpan('in-place patch', s); console.log(fmtSvc([s[0], s[1], interval(s).toFixed(2), (e - b).toFixed(2), firstSpan('code-change rebuild', s)[0].toFixed(2)])); } const holds = SERVICES.reduce((a, s) => { const [b, e] = firstSpan('in-place patch', s); return a + s[1] * (e - b); }, 0) / replicas; console.log(`in-place patch holds for a replica-weighted mean of ${holds.toFixed(2)} days; over ${WINDOW} days ` + `the share of time spent protected is ${(100 * SERVICES.reduce((a, s) => a + s[1] * protectedDays('in-place patch', s), 0) / (replicas * WINDOW)).toFixed(1)}%`); ``` ``` 9 services, 31 replicas, 18 releases per week; deployment interval between 1.40 and 14 days path first replica last replica open replica-days open share in-place patch 5.25 5.25 676.6 72.8% code-change rebuild 1.65 14.25 180.6 19.4% scheduled rebuild 1.75 1.75 54.3 5.8% service replicas interval patch holds rebuild 1st day portal-frontend 6 1.40 0.35 1.65 subscriber-service 4 1.75 1.75 2.00 billing 3 2.33 1.75 2.58 work-order 3 3.50 1.75 3.75 notification 2 7.00 1.75 7.25 report 2 7.00 1.75 7.25 measurement-collector 6 7.00 1.75 7.25 validator 3 14.00 8.75 14.25 field-sync 2 14.00 8.75 14.25 in-place patch holds for a replica-weighted mean of 2.61 days; over 30 days the share of time spent protected is 27.2% ``` The first two columns of the upper table make in-place patching look like the best option: the whole fleet closes in five point two five days, because the patch is applied to all replicas in the same maintenance window. The last replica of the rebuild tied to a code change, on the other hand, closes in fourteen point two five days. The third column reverses the decision. Over the thirty-day window, in-place patching leaves six hundred seventy-six point six replica-days open (seventy-two point eight percent of the fleet), the rebuild one hundred eighty point six replica-days (nineteen point four percent). Same patch, same fleet, same thirty days: **the path that does the first closing three times faster makes the open time four times larger.** #### Where the Patch Does Not Hold The lower table gives the reason service by service. Because the in-place patch is not written to the image, the next deployment brings back the unpatched image; how long the patch holds therefore depends not on the patch itself but on the **deployment frequency of the service**. The ranking is the reverse of what one would expect. The portal frontend is deployed more often than once in two days and its patch holds for zero point three five days, less than nine hours: a six-replica service sitting at the edge is the service in the fleet that loses the patch fastest. The validator and field sync are deployed once in two weeks, so they carry the patch for eight point seven five days. In-place patching works best on the service that is deployed least and usually faces the least exposed traffic. The replica-weighted mean is two point six one days; only twenty-seven point two percent of the thirty days passes protected, and even if the weekly maintenance window is repeated the table does not change, because the window comes once a week while deployment is more frequent than once a day for some services. The scheduled rebuild column, on the other hand, carries the same number for all nine services: one point seven five days. The trigger is the base image changing, not a code change in the service; deployment frequency drops out of the equation. #### The Price of Disruption ```js // run2.mjs — the disruption rebuilding brings and the window the runtime layer holds import { SERVICES, PATHS, WINDOW, READY, replicas, openDays, gen } from './update.mjs'; const fmtRestart = gen([10, 9, 15, 17, 18]); console.log(fmtRestart(['replicas', 'services', 'window (s)', 'capacity loss', 'weekly restarts'])); for (const k of [...new Set(SERVICES.map((s) => s[1]))].sort((a, b) => b - a)) { const g = SERVICES.filter((s) => s[1] === k); console.log(fmtRestart([k, g.length, k * READY, `${(100 / k).toFixed(1)}%`, g.reduce((a, s) => a + s[1] * s[2], 0).toFixed(1)])); } const current = SERVICES.reduce((a, s) => a + s[1] * s[2], 0), added = (3 / 4) * replicas; const sorted = SERVICES.map((s) => [s[0], s[1] * s[2]]).sort((a, b) => b[1] - a[1]); console.log(`fleet window: nine services in parallel ${(Math.max(...SERVICES.map((s) => s[1])) * READY / 60).toFixed(2)} min, ` + `sequentially ${(replicas * READY / 60).toFixed(2)} min; most restarts ${sorted[0][0]} (${sorted[0][1]}/week), ` + `fewest ${sorted[8][0]} (${sorted[8][1]}/week)`); console.log(`weekly replica restarts ${current.toFixed(1)}; scheduled rebuild (3 base images per month) ` + `adds ${added.toFixed(2)}: ${(100 * added / current).toFixed(1)}% increase. In-place patch adds zero restarts, ` + `but in return requires 4 maintenance windows per month and ${4 * replicas} replica touches`); // The class that can only be stopped at runtime: a component vulnerability with no patch yet, 40 requests // per day (from the first lesson of this course). Patch lag is the width of the net that holds that class. const DAILY = 40, fmtNet = gen([32, 16, 24, 18]); console.log('\n' + fmtNet(['path', 'open fleet-days', 'requests the net holds', 'multiple of best'])); const best = openDays('scheduled rebuild') / replicas; for (const p of PATHS) { const g = openDays(p) / replicas; console.log(fmtNet([p, g.toFixed(2), Math.round(g * DAILY), `${(g / best).toFixed(1)}x`])); } console.log(`a total of ${WINDOW * DAILY} requests arrive in the thirty-day window`); ``` ``` replicas services window (s) capacity loss weekly restarts 6 2 270 16.7% 36.0 4 1 180 25.0% 16.0 3 3 135 33.3% 16.5 2 3 90 50.0% 5.0 fleet window: nine services in parallel 4.50 min, sequentially 23.25 min; most restarts portal-frontend (30/week), fewest field-sync (1/week) weekly replica restarts 73.5; scheduled rebuild (3 base images per month) adds 23.25: 31.6% increase. In-place patch adds zero restarts, but in return requires 4 maintenance windows per month and 124 replica touches path open fleet-days requests the net holds multiple of best in-place patch 21.83 873 12.5x code-change rebuild 5.83 233 3.3x scheduled rebuild 1.75 70 1.0x a total of 1200 requests arrive in the thirty-day window ``` The cost of the scheduled rebuild is that weekly replica restarts rise from seventy-three point five to ninety-six point seven five: a thirty-one point six percent increase. This increase is not evenly distributed. The portal frontend already restarts thirty times a week and the extra load makes no difference; field sync restarts once a week and one and a half times that is added on top. The load falls on **the least-deployed services**, and those were the services that carried the patch the longest. The capacity loss column gives the second cost. If one replica is down at a time in a rolling deployment, the loss is sixteen point seven percent for a six-replica service and fifty percent for a two-replica service. Three services run at half capacity for ninety seconds on every round; if the nine services are deployed in parallel, the fleet turns over in four and a half minutes. The real decision is whether those minutes are acceptable, and that question cannot be asked separately from patch lag. #### The Width of the Net The lower table ties the whole measurement of the course to a single number. In the first lesson, of the six hundred sixty-six requests the web application firewall stopped, only forty — six percent — belonged to the class that could not be closed at an earlier stage: a vulnerability in a component with no patch yet, forty requests a day, waiting for the patch to arrive. Patch lag is exactly the length of time the net has to stay stretched. Under the scheduled rebuild the runtime layer holds seventy requests on its own; under the code-change rebuild, two hundred thirty-three; under in-place patching, eight hundred seventy-three — twelve and a half times. Same rule set, same threshold; the only thing that changes is the update path behind it. The value of runtime defense is measured not by the quality of the rule set but by **how long the item behind it stays open**. #### Summary - On the three paths the day the last replica closes is 5.25 / 14.25 / 1.75; the thirty-day open replica-days are 676.6 / 180.6 / 54.3, that is 72.8% / 19.4% / 5.8%. - In-place patching does the first closing fastest but, because it is not written to the image, it falls at the next deployment; protection holds for 2.61 days on a replica-weighted average. The most frequently deployed service loses the patch in 0.35 days, the least frequently deployed carries it for 8.75. - The scheduled rebuild closes all nine services in 1.75 days: the trigger is the base image changing, and deployment frequency drops out of the equation. - Its price is weekly replica restarts rising from 73.5 to 96.75 (31.6%), and the load falls on the least-deployed services; the three services with two replicas run at half capacity for 90 seconds on every round. - The class that can only be stopped at runtime is 40 requests a day; the requests the layer holds on its own are 70 / 233 / 873. Shortening patch lag cuts the layer's load by a factor of 12.5 without writing a rule. #### Course Wrap-Up | Lesson | What the layer catches | What it misses / falsely blocks | Stage that would close it earlier | |---|---|---|---| | Web Application Firewall | 666 bad requests/day | 1,460 missed, 8,000 cut at threshold 3 | 626 code, only 40 here | | Rate Limiting Windows | 5,760 requests (sliding window) | 6,900 missed, 880 legitimate peaks | 3,910 here, the rest code and design | | Credential Stuffing Defense | 5,512 attempts, 13 takeovers | 0 blocked in stuffing, 91 account lockouts | code from 70 to 5, design 0 | | Security Headers Coverage | 55 cells, 63.4% of requests | 3 endpoints outside, 24 cells ineffective | 3 headers code, 3 headers here | | Application-Level Anomaly Detection | 3 bad sessions (threshold 20) | 2/4 legitimate signals, 7,106 records | 1 code, 1 design, none for 1 | | Secret Store Architecture | 31 items in use | 48 unused grants (61%) | deployment manifest, 9 times cheaper | | Short-Lived Credentials | 2,033 times on 16 items | 15 items fixed, 99.96% of exposure | design, 8 secrets | | Key Rotation | 0 cuts at minimum overlap | 87,427 requests cut in 60 minutes | code, from 1,440 to 240 minutes | | Workload Identity | 20 static secrets removed | 11 items remain, 7 applications with secrets | design, 3 of 4 gates | | Leak Response | early revocation 150,934 calls | 224,171 calls, 68,503 requests cut | build, 2,880 minutes latency | | Network Segmentation and Service Zero Trust | 35 pairs closed, 7.1 down to 4.1 | 1 to 2 hops to the store, 3.8 pairs of disruption | design, call contract | | Container Runtime Detection | 12 of 21 events | 9 missed, 4 false positives | 9 events build, only 3 here | | Admission Controller | 22 defects, first 6 rules 19 of them | 5 legitimate rejections, 5/5 teams | 19 pipeline gate, only 3 here | | Continuous Configuration Audit | 107 findings find an owner | 40 unowned, 112 in 12 weeks | 122 earlier, none for 25 | | Rebuild-Based Update | last replica in 1.75 days | 5.8% open, +31.6% restarts | build, base image | The third column is the rule of this course. In none of the fifteen lessons could the runtime layer claim the whole of the volume it caught: of every hundred requests the firewall stopped, ninety-four could have been closed in code; of the twenty-two defects the admission controller shut, nineteen sat in the pipeline gate; of the one hundred forty-seven findings continuous auditing found, one hundred twenty-two could have been closed at an earlier stage. The layer's own share is small in every lesson and real in every lesson — forty requests, three events, three defects, twenty-five unowned findings. **Runtime defense is not a patch but a net stretched for the item that cannot be closed earlier.** When the question "where would this have closed more cheaply" is not asked for each item, the layer stops being a defense and turns into an excuse: the big number on the dashboard is nothing but an unclosed code defect being caught again every day. The second column must be read with the same insistence. Eight thousand falsely blocked requests, ninety-one locked accounts, five rejected legitimate deployments, eighty-seven thousand four hundred twenty-seven requests cut during rotation — these are not side effects of the layer but its price. A runtime defense that does not write down its false positives counts as unmeasured. The next course — **M25/K05 Detection, Response and Compliance** — takes up the question this table does not answer. The layers were built and what each one misses was written down one by one; but **which record** would show that what was missed was missed was never asked. Who will do what when a breach occurs, and how all of this practice will be proven in an audit, were not asked either. A measured defense is true only in its own table unless it leaves a record that can be read at the moment of the incident and evidence that can be verified from the outside.