--- title: 'Detection, Response and Compliance' source: 'https://academia.sh/en/courses/application-detection-and-compliance' language: en updated: '2026-10-07T04:31:05+00:00' license: 'CC BY-SA 4.0' lessons: 9 outcomes: - 'Produces security detections from application telemetry' - 'Runs the response process for application breaches' - 'Maps engineering practices to compliance controls' --- # Detection, Response and Compliance Measuring how many of the questions to be asked after an incident can be answered: matching the question set against the log schema, whether an unanswered question stems from a missing field, a missing link or the retention window, the cost of every additional field in volume and retention period, what threshold selection misses and what it falsely triggers in abuse alerts based on business logic, which step of application breach response depends on which record and how many minutes a step takes longer when the record is missing, the window of items that can be collected in short-lived workloads, the time from report to verification and from verification to fix in the external report intake and the bug bounty program, the engineering equivalent of an audit control, moving evidence production into the pipeline, mean time to remediate and open finding age metrics, and paying down security debt on a plan. ## Detection and Response Measuring how well application telemetry can answer the questions asked after an incident: matching the question set against the log schema and counting the unanswered questions, the volume and retention cost of each extra field, what a threshold misses and falsely wakes up in detection scenarios based on business logic, the step–record dependency in application breach response and the minute cost of a missing record, the window of items that can be written out in short-lived workloads, and the verification and fix times in the external report intake and bug bounty program. ### Audit Log Design Source: https://academia.sh/en/courses/application-detection-and-compliance/audit-log-design Matched against three log sources, a set of sixteen questions leaves six answered and ten unanswered: four because the field is not written, five because no join key links the sources, one because of the retention window. Three added fields raise the answered count to 13, but once all five are added the daily volume grows 1.45 times, retention under the fixed budget falls from 65 days to 45, and the five questions that reach into the 60-day audit period go unanswered again, so the answered count drops to 10. The Runtime Security and Secrets Management course built the layers and wrote down what each layer misses. The existence of a missed item was read there off the model's ground truth. A real organization has no such ground truth: what was missed becomes known only from a record. This course measures that record and asks a single question — how many of the questions asked after an incident can be answered. The unit of measurement is the **unanswered question**. A log design is not called "comprehensive" or "inadequate"; a question set is put in front of it, the matching is run, and the questions left without an answer are counted. The second half of the same measure is cost: how many added fields, how many times the volume, and how many personal data items it takes to make them answerable. This lesson runs the two together. How a log is written, kept immutable, and protected for integrity was measured in the Server Security and Going to Production course; it is not repeated here, it becomes an input. - **DR1.** The example network is **fictional**: an unauthorized access breach in the subscriber portal of a regional measurement network, nine services, three log sources, and a 60-day audit period. Every number is the output of a model. - **DR2.** The **question set** is the questions asked after an incident, modeled as a data structure. Each question has two attributes: the **fields** needed to answer it and how many days back it reaches. - **DR3.** A question is answered only if all of its required fields lie within a **single joinable field set**. Fields in two separate sources cannot be joined unless they share a join key; the two records sit side by side, but it cannot be shown that they belong to the same event. - **DR4.** Being unanswerable splits into three causes: **no field** (no source writes it), **no join key** (the fields exist but do not match), **window** (the fields and the key exist but the log is not retained that far back). - **DR5.** Retention is determined not by a decision but by arithmetic: a fixed retention budget divided by the daily volume. In the model the budget is 600 GB and the daily event count is 42 million; 1 GB counts as 10⁹ bytes. - **DR6.** Personal data is counted **by type**: the client address, the session identifier, and the field value from before a change are one item each. Each item places a separate upper limit on retention. #### Matching the Question Set to the Schema ```js // record-matching.mjs — matches the question set to the log schema; the unanswered question // is the unit of measurement of this course. A MODEL: the network, logs, and questions are fictional. export const BUDGET_B = 600e9, DAILY_EVENTS = 42e6; // fixed retention budget and daily events // Source = one log stream. Sources that have a `trace` field can be joined to each other; // a source without it stands alone and its fields cannot be combined with other fields. export const SOURCES = [ { name: "application audit log", share: 0.18, bytes: 240, fields: ["time", "trace", "actor", "action", "outcome"] }, { name: "network gateway access log", share: 0.62, bytes: 180, fields: ["time", "trace", "path", "method", "status", "client_ip"] }, { name: "database change log", share: 0.20, bytes: 320, fields: ["time", "table", "object", "operation"] }, ]; // Question = [text, required fields, how many days back it reaches]. The audit period is 60 days. export const QUESTIONS = [ ["which account read which subscriber record", ["actor", "object", "action"], 30], ["was the read accepted or denied", ["actor", "action", "outcome"], 30], ["which client address did the request come from", ["client_ip", "trace"], 7], ["which account changed which table", ["actor", "table", "operation"], 30], ["what was the value before the change", ["table", "old_value"], 30], ["which application version was running", ["actor", "action", "version"], 30], ["what else was done in the same session", ["session", "action"], 7], ["what was the authorization decision and its basis", ["actor", "action", "authz_decision"], 30], ["when was the first denial at the start of the period", ["actor", "action", "outcome"], 60], ["how many subscriber records were read in the period", ["object", "action"], 60], ["which endpoints were called", ["path", "method", "status"], 30], ["which request changed which row", ["trace", "table", "object"], 60], ["how many accounts were tried from the same address", ["client_ip", "actor"], 30], ["what did the same account do in the quarter before the incident", ["actor", "action"], 90], ["what record set was exported", ["actor", "action", "object", "outcome"], 60], ["how many rows changed in the period", ["table", "operation"], 60], ]; // A question is answered only if ALL of its required fields are inside ONE joinable // cluster; fields in separate clusters cannot be combined. export function clusters(src) { const linked = src.filter((s) => s.fields.includes("trace")); const alone = src.filter((s) => !s.fields.includes("trace")); const merged = [...new Set(linked.flatMap((s) => s.fields))]; return [merged, ...alone.map((s) => s.fields)].filter((f) => f.length); } export function match(src, retention) { const cluster = clusters(src), all = new Set(src.flatMap((s) => s.fields)); return QUESTIONS.map(([text, fields, days]) => { const missing = fields.filter((f) => !all.has(f)); if (missing.length) return { text, days, status: "no field", missing: missing[0] }; if (!cluster.some((c) => fields.every((f) => c.includes(f)))) return { text, days, status: "no join key", missing: "-" }; if (days > retention) return { text, days, status: "window", missing: "-" }; return { text, days, status: "answered", missing: "-" }; }); } export const dailyBytes = (src) => src.reduce((t, s) => t + DAILY_EVENTS * s.share * s.bytes, 0); export const retentionDays = (src) => Math.floor(BUDGET_B / dailyBytes(src)); export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const gb = dailyBytes(SOURCES) / 1e9, retention = retentionDays(SOURCES); console.log(`model: ${SOURCES.length} sources, ${QUESTIONS.length} questions, ` + `${gb.toFixed(2)} GB daily, ${BUDGET_B / 1e9} GB budget -> ${retention} days retention`); console.log(`joinable field sets: ` + clusters(SOURCES).map((c) => c.length).join(" and ") + " fields"); const y = gen([64, 6, 13, 15]); console.log("\n" + y(["question", "day", "status", "missing field"])); for (const s of match(SOURCES, retention)) console.log(y([s.text, s.days, s.status, s.missing])); const d = match(SOURCES, retention); const count = (t) => d.filter((s) => s.status === t).length; console.log(`\nanswered ${count("answered")}, unanswered ` + `${QUESTIONS.length - count("answered")}: no field ${count("no field")}, ` + `no join key ${count("no join key")}, window ${count("window")}`); } ``` ``` model: 3 sources, 16 questions, 9.19 GB daily, 600 GB budget -> 65 days retention joinable field sets: 9 and 4 fields question day status missing field which account read which subscriber record 30 no join key - was the read accepted or denied 30 answered - which client address did the request come from 7 answered - which account changed which table 30 no join key - what was the value before the change 30 no field old_value which application version was running 30 no field version what else was done in the same session 7 no field session what was the authorization decision and its basis 30 no field authz_decision when was the first denial at the start of the period 60 answered - how many subscriber records were read in the period 60 no join key - which endpoints were called 30 answered - which request changed which row 60 no join key - how many accounts were tried from the same address 30 answered - what did the same account do in the quarter before the incident 90 window - what record set was exported 60 no join key - how many rows changed in the period 60 answered - answered 6, unanswered 10: no field 4, no join key 5, window 1 ``` All three sources write regularly, none is switched off, and 9.19 GB of records are produced per day. Even so, ten of the sixteen questions have no answer. The real information in this table is not the number of unanswered questions but the **distribution of causes** in the third column, because the three causes are fixed by three different jobs and their costs differ. The largest cluster is in the **no join key** rows: five questions. In these questions, all of the required fields are written. The `actor` that the question "which account changed which table" needs sits in the application audit log, and `table` and `operation` sit in the database change log. The two records may have been written in the same second, but because the database log has no `trace` field, it cannot be shown which request caused which row change. In a busy period there are hundreds of requests and hundreds of row changes in the same second; a join based on timestamps produces a claim, not evidence. **Two log sources with no join key answer only as many questions as each collects on its own, not as many as their sum.** The second cluster is **no field**: four questions. These are design gaps and are closed by writing new fields. The third is a single row but the most insidious: `window`. The question "what did the same account do in the quarter before the incident" has all of its fields and its join key; it goes unanswered because it reaches back 90 days. This question can never be found by looking at the schema. The schema is right, the fields are complete, the join key seems established — unanswerability comes from the retention arithmetic and surfaces only when the question set is modeled together with the window. #### What Making Questions Answerable Costs Four missing fields and one join key can be written. Each added field enters every record and grows the daily volume; because the budget is fixed, volume turns directly into retention days. ```js // additions.mjs — each added field raises the answered count but grows the daily volume; // under a fixed budget volume becomes retention days, and retention days feed back into answers. A MODEL. import { SOURCES, QUESTIONS, match, dailyBytes, retentionDays, gen } from "./record-matching.mjs"; // Each addition: which field goes to which sources, how many bytes per record, personal data or not. const ADDITIONS = [ { field: "trace", targets: ["database change log"], bytes: 18, personal: false }, { field: "authz_decision", targets: ["application audit log"], bytes: 16, personal: false }, { field: "version", targets: ["application audit log"], bytes: 12, personal: false }, { field: "session", targets: ["application audit log", "network gateway access log"], bytes: 26, personal: true }, { field: "old_value", targets: ["database change log"], bytes: 340, personal: true }, ]; const apply = (src, a) => src.map((s) => a.targets.includes(s.name) ? { ...s, bytes: s.bytes + a.bytes, fields: [...s.fields, a.field] } : { ...s }); const BASE_GB = dailyBytes(SOURCES) / 1e9; const y = gen([16, 10, 7, 11, 12, 14, 9]); console.log(y(["added field", "daily GB", "ratio", "retention", "answered", "unanswered", "personal"])); let src = SOURCES.map((s) => ({ ...s })), personal = 1; // client_ip is already personal const row = (name) => { const gb = dailyBytes(src) / 1e9, ret = retentionDays(src); const c = match(src, ret).filter((s) => s.status === "answered").length; console.log(y([name, gb.toFixed(2), (gb / BASE_GB).toFixed(2), ret, c, QUESTIONS.length - c, personal])); return { ret, c }; }; const trail = [row("(base schema)")]; for (const a of ADDITIONS) { src = apply(src, a); if (a.personal) personal++; trail.push(row(a.field)); } const best = trail.reduce((a, b) => (b.c > a.c ? b : a)), last = trail.at(-1); console.log(`\nbest schema retains ${best.ret} days: ${best.c} questions`); console.log(`last schema answers ${best.c - last.c} fewer questions, because retention fell from ` + `${best.ret} days to ${last.ret} and the audit period is 60 days`); console.log(`base schema to last schema: answered ${trail[0].c} -> ${last.c}, ` + `volume ${(dailyBytes(src) / 1e9 / BASE_GB).toFixed(2)}x, personal data 1 -> ${personal} items`); ``` ``` added field daily GB ratio retention answered unanswered personal (base schema) 9.19 1.00 65 6 10 1 trace 9.34 1.02 64 11 5 1 authz_decision 9.46 1.03 63 12 4 1 version 9.55 1.04 62 13 3 1 session 10.43 1.13 57 9 7 2 old_value 13.28 1.45 45 10 6 3 best schema retains 62 days: 13 questions last schema answers 3 fewer questions, because retention fell from 62 days to 45 and the audit period is 60 days base schema to last schema: answered 6 -> 10, volume 1.45x, personal data 1 -> 3 items ``` The first addition is the cheapest in the table and gains the most: an 18-byte `trace` field in the database log grows the daily volume by 2% and raises the answered count from 6 to 11. On its own it records no new fact; it only joins two records that were already being written. **In a log design, the highest-yield field is often not a new fact but the field that says two existing facts belong to the same job.** The next two additions progress linearly: `authz_decision` and `version` grow the volume by 4% and raise the answered count to 13. This row is the best point in the table and cannot be found without looking at the table. The **order** of the added fields is not arbitrary either: each field in the list brings in volume equal to its bytes per record times the daily share of the source it enters. The `version` field is 12 bytes but enters only the source that carries 18% of the events; `session` is 26 bytes and enters 80% of them. The second seems to take a little more than twice the space of the first; in reality it takes more than nine times as much. **A field's cost is measured not by its bytes but by the number of records its bytes are written to.** The fourth row shows the break. The `session` field enters two high-volume sources and the daily volume rises to 10.43 GB; under the fixed budget retention falls from 62 days to 57. This addition makes one question answerable and **leaves the five questions that reach 60 days unanswered all at once**: the answered count drops from 13 to 9. In the fifth row, writing the value from before the change raises the volume 1.45 times, lowers retention to 45 days, and the table closes at 10 — only four questions better than the base schema, three worse than the best schema. This is the name of what stands as a single curve in the table: **more records, fewer answers under a fixed budget.** The break happens not because the log volume grew but because the retention window fell below how far the question set reaches. The audit period is 60 days; a 57-day retention leaves the first three days of the period unanswered even if the log itself is flawless. The third column of the cost looks single-digit in the table but is the longest-lived. The personal data items go from 1 to 3: the session identifier and the field value from before the change are items that can be tied directly to the subscriber's identity. These two fields do not only bring volume; they restrict access to the record, put a separate upper limit on retention, and require them to be scrubbed from every evidence copy handed out. The cost of writing a field "just in case" is read across these three columns at once. #### Summary - A log design is measured not by labeling but by the question set put in front of it; the unit of measurement is the unanswered question. In the model, 10 of 16 questions are unanswered with the base schema. - Being unanswerable splits into three distinct causes, and the three are fixed by three distinct jobs: no field (4 questions), no join key (5 questions), retention window too short (1 question). - The highest-yield field is not a new fact but the join key that says two records belong to the same job: for 18 bytes of volume, the answered count rises from 6 to 11. - Under a fixed retention budget, each added field turns into retention days: five fields raise the volume 1.45 times and lower retention from 65 days to 45. - More records can give fewer answers: the best schema answers 13 questions with 62 days of retention, while the schema with two more fields stays at 10 questions with 45 days, because the audit period is 60 days. - The lasting column of the cost is the personal data item: this number, rising from 1 to 3, brings an access restriction, a separate retention limit, and a scrubbing burden on every evidence copy. #### Next Step All of this lesson's measurement was made **after** the incident: the question had already been asked, and the log was checked for whether it held an answer. The same records have a second job — announcing the incident that gave rise to the question while it is still going on. But an abuse at the application level does not resemble an attack at the protocol level: the requests are valid, the session is open, the authorization decision is mostly positive, and the only anomaly is in a number that a business rule measures. The next lesson runs detection scenarios built on these numbers over an event stream and measures the two costs of choosing a threshold together: how many abuse sessions are missed and how many times someone is woken up for nothing. ### Application-Specific Detection Scenarios Source: https://academia.sh/en/courses/application-detection-and-compliance/application-specific-detection-scenarios In an event stream of 900 sessions, 18 abuse sessions are searched for with four detection scenarios. Scanning thresholds one at a time gives the same trade-off on every metric: a denied-authorization threshold of 3 yields 364 false positives and 211 off-hours pages, while 25 yields zero false positives but zero catches. When the four scenarios are chosen together, an alert budget of 25 catches 8 of the 18 sessions and a budget of 180 catches all 18 — the latter means 101 night pages in 30 days. When the out-of-region read field is not written, the number caught at the same budget falls from 15 to 13. The previous lesson measured the side that queries the log after the incident: how many of the sixteen questions had an answer in the schema. The same records have a second job — announcing the incident that gave rise to the question while it is still going on. At the application level this job does not resemble the one at the network level. The abusing party makes requests with a valid session, to valid endpoints, and often with a positive authorization decision; what is abnormal is not the protocol but a number that a business rule measures. This lesson runs the **detection scenarios** built on those numbers over a fictional event stream. The architecture of the detection pipeline, the normalization of events, and the correlation rule were measured in the Security Operations and Incident Response course; they are not repeated here, they become inputs. The only thing measured is the two costs of choosing a threshold: how many abuse sessions are missed and how many times someone is woken up for nothing. - **DR7.** The event stream is **fictional**: 900 sessions in 30 days in the subscriber portal, four roles, and four abuse classes. The numbers come from a generator whose seed is visible. - **DR8.** A **detection scenario** is a threshold rule built on a single application metric. The metrics are read from the business logic: distinct subscriber records read in a session, the number of denied authorization decisions, distinct accounts tried from the same client address, and records read outside the account's authorized region. - **DR9.** An alert counts as **real** only if the session is an abuse session; otherwise it is a **false positive**. Business hours are taken as 08–18; a false positive outside this range is an **off-hours page** and is counted separately. - **DR10.** 8% of normal sessions produce a heavy tail, such as an "end-of-day batch job". This tail is the source of the overlap that no rule can remove. - **DR11.** The out-of-region read metric depends on a field that is **not** in the previous lesson's schema. If the field is not written, the scenario cannot be built; this is the detection-side counterpart of log design. #### Threshold Scan on a Single Metric ```js // event-stream.mjs — generates an application event stream and runs the business-logic // detection scenarios threshold by threshold. A MODEL: the sessions and the abuse are fictional. export const SEED = 20260418, SESSION_COUNT = 900, DAYS = 30; const rng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = rng(SEED); // A session with v < 0.08 produces a normal but heavy tail, like an "end-of-day batch job". const count = (mean, u, v = 1) => Math.round(mean * (0.35 + 1.3 * u) * (v < 0.08 ? 3.5 : 1)); // Role = the account's normal workload. An integration account is high-volume even when normal. const ROLES = [ { name: "subscriber representative", share: 0.42, reads: 12, denials: 2.4, address: 1.4, region: 0.3 }, { name: "field technician", share: 0.28, reads: 6, denials: 3.0, address: 1.2, region: 0.5 }, { name: "integration account", share: 0.12, reads: 140, denials: 1.0, address: 2.2, region: 2.5 }, { name: "subscriber", share: 0.18, reads: 1, denials: 1.6, address: 1.6, region: 0.0 }, ]; const pickRole = (u) => { let t = 0; for (const x of ROLES) { t += x.share; if (u < t) return x; } return ROLES.at(-1); }; // Abuse class = which metric of the session it pushes outside normal. export const ABUSE = [ { name: "bulk read", n: 6, apply: (o, u) => { o.reads = count(300, u); } }, { name: "authorization probing", n: 4, apply: (o, u) => { o.denials = count(18, u); } }, { name: "out-of-region leak", n: 5, apply: (o, u) => { o.region = count(12, u); } }, { name: "account takeover", n: 3, apply: (o, u) => { o.address = count(6, u); } }, ]; export const SESSIONS = Array.from({ length: SESSION_COUNT }, (_, i) => { const role = pickRole(r()), hour = Math.floor(r() * 24), tail = r(); return { i, role: role.name, hour, abuse: null, reads: count(role.reads, r(), tail), denials: count(role.denials, r(), tail), address: count(role.address, r(), tail), region: count(role.region, r(), tail) }; }); for (const k of ABUSE) for (let n = 0; n < k.n; n++) { const o = SESSIONS[Math.floor(r() * SESSION_COUNT)]; o.abuse = k.name; k.apply(o, r()); } export const ABUSE_TOTAL = SESSIONS.filter((o) => o.abuse).length; // Scenario = a threshold rule built on a single application metric. export const SCENARIOS = [ { name: "distinct record reads", metric: "reads", target: "bulk read", thresholds: [40, 80, 150, 260] }, { name: "denied authorization", metric: "denials", target: "authorization probing", thresholds: [3, 6, 12, 25] }, { name: "accounts per address", metric: "address", target: "account takeover", thresholds: [2, 3, 5, 7] }, { name: "out-of-region reads", metric: "region", target: "out-of-region leak", thresholds: [2, 4, 8, 12] }, ]; const offHours = (o) => o.hour < 8 || o.hour >= 18; export function run(s, t) { const alerts = SESSIONS.filter((o) => o[s.metric] >= t); const falseAlerts = alerts.filter((o) => !o.abuse); return { threshold: t, alerts: alerts.length, real: alerts.filter((o) => o.abuse).length, targetCaught: alerts.filter((o) => o.abuse === s.target).length, targetTotal: ABUSE.find((k) => k.name === s.target).n, falseAlerts: falseAlerts.length, pages: falseAlerts.filter(offHours).length }; } export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model (seed ${SEED}): ${SESSION_COUNT} sessions, ${DAYS} days, ` + `${ABUSE_TOTAL} abuse sessions`); console.log("role distribution: " + ROLES.map((x) => `${x.name} ${SESSIONS.filter((o) => o.role === x.name).length}`).join(", ")); const y = gen([23, 11, 8, 10, 8, 11, 12]); console.log("\n" + y(["scenario", "threshold", "alerts", "missed", "real", "false+", "pages"])); for (const s of SCENARIOS) for (const t of s.thresholds) { const o = run(s, t); console.log(y([s.name, t, o.alerts, o.targetTotal - o.targetCaught, o.real, o.falseAlerts, o.pages])); } } ``` ``` model (seed 20260418): 900 sessions, 30 days, 18 abuse sessions role distribution: subscriber representative 388, field technician 229, integration account 120, subscriber 163 scenario threshold alerts missed real false+ pages distinct record reads 40 149 0 7 142 83 distinct record reads 80 110 0 7 103 60 distinct record reads 150 58 1 6 52 29 distinct record reads 260 10 2 4 6 5 denied authorization 3 377 0 13 364 211 denied authorization 6 49 0 5 44 28 denied authorization 12 17 0 4 13 6 denied authorization 25 0 4 0 0 0 accounts per address 2 467 0 12 455 261 accounts per address 3 115 0 6 109 65 accounts per address 5 43 1 4 39 25 accounts per address 7 20 2 2 18 14 out-of-region reads 2 127 0 5 122 68 out-of-region reads 4 41 0 5 36 21 out-of-region reads 8 7 2 3 4 3 out-of-region reads 12 3 4 1 2 1 ``` All four scenarios draw the same shape: as the threshold rises, the missed count grows and the false positives fall. What is interesting is **where** this curve stops. The denied-authorization scenario's threshold of 3 produces 377 alerts, and 364 of them are false; 211 arrive outside business hours, which is 211 night pages in 30 days. The same scenario's threshold of 25 gives zero false positives and misses all four of the four abuse sessions at once. **At one end there is an alert stream nobody reads, at the other a rule that never alerts; both are two settings of the same rule.** The distinct record reads scenario shows why the overlap does not come from a tuning error. Integration accounts read 140 distinct records on average in their normal work; an abuse session doing bulk reads reads 300 on average. The two distributions overlap. When the threshold is pulled up to 260, the false positives fall to 6 but two of the six bulk-read sessions are missed. What produces the overlap is not the quality of the tool but **normal itself**: there is a role that legitimately runs high on the same metric, and taking that role out of the rule means leaving the easiest place for abuse to hide outside the rule. The out-of-region read scenario stands apart in this table. Its metric is not a volume but a **rule violation**: the account touching a record outside its authorized region. For this reason, at a threshold of 8 it catches three leak sessions with only 4 false positives; in the same stream the accounts-per-address metric pays 39 false positives for four real alerts. The difference comes not from the tightness of the rule but from the choice of metric: a metric based on a business rule separates normal from abnormal by definition, not in the tail of a distribution. It still depends on the tail — at a threshold of 12, four of the five sessions are missed — but it gives the same separating power far more cheaply. The heavy tail spreads the same fact across all roles. 8% of sessions do an end-of-day job that is normal but three and a half times as intense. At the accounts-per-address scenario's threshold of 2, 467 sessions raise an alert — more than half of the stream. This threshold is not "sensitive"; it stands at a point where the metric has no separating power and turns the rule's output into a random sample of the population. #### Choosing the Thresholds Together Scenarios are not tuned one by one. The output of the four rules goes to the person on call together; the cost also arises together. ```js // thresholds.mjs — the thresholds of the four scenarios are chosen together: the threshold // combination that catches the most abuse sessions under an alert budget is searched for. A MODEL. import { SESSIONS, SCENARIOS, ABUSE, ABUSE_TOTAL, gen } from "./event-stream.mjs"; const offHours = (o) => o.hour < 8 || o.hour >= 18; function combo(thresholds, scenarios = SCENARIOS) { const alerts = SESSIONS.filter((o) => scenarios.some((s, j) => o[s.metric] >= thresholds[j])); const falseAlerts = alerts.filter((o) => !o.abuse); return { alerts: alerts.length, caught: alerts.filter((o) => o.abuse).length, falseAlerts: falseAlerts.length, pages: falseAlerts.filter(offHours).length, byClass: ABUSE.map((k) => alerts.filter((o) => o.abuse === k.name).length) }; } function best(budget, scenarios = SCENARIOS) { const lists = scenarios.map((s) => s.thresholds); let top = null; const walk = (j, choice) => { if (j === lists.length) { const o = combo(choice, scenarios); if (o.falseAlerts > budget) return; if (!top || o.caught > top.o.caught || (o.caught === top.o.caught && o.falseAlerts < top.o.falseAlerts)) top = { choice: [...choice], o }; return; } for (const t of lists[j]) walk(j + 1, [...choice, t]); }; walk(0, []); return top; } const y = gen([9, 19, 8, 12, 12, 10, 12, 15]); console.log(y(["budget", "chosen thresholds", "alerts", "caught", "missed", "false+", "pages", "false/real"])); for (const b of [25, 50, 100, 180, 400]) { const e = best(b); console.log(y([b, e.choice.join(" "), e.o.alerts, `${e.o.caught}/${ABUSE_TOTAL}`, ABUSE_TOTAL - e.o.caught, e.o.falseAlerts, e.o.pages, (e.o.falseAlerts / e.o.caught).toFixed(1)])); } const full = best(100), byClass = ABUSE.map((k, i) => `${k.name} ${full.o.byClass[i]}/${k.n}`); console.log(`\nclass breakdown of the combination at budget 100: ${byClass.join(", ")}`); // if the region field is not written, the fourth scenario cannot be built at all. const three = SCENARIOS.slice(0, 3), withoutRegion = best(100, three); console.log(`without the region field at the same budget: caught ${withoutRegion.o.caught}/${ABUSE_TOTAL}, ` + `from the out-of-region leak ${withoutRegion.o.byClass[2]}/${ABUSE[2].n}`); // The field's cost from the previous lesson's model: 42e6 events/day, application audit log share 0.18, // 9.55 GB daily and a 600 GB budget (the schema with the version field added). const DAILY_EVENTS = 42e6, SHARE = 0.18, BASE = 9.55248e9, BUDGET = 600e9, REGION_BYTES = 14; const next = BASE + DAILY_EVENTS * SHARE * REGION_BYTES; console.log(`region field ${REGION_BYTES} bytes: daily ${(BASE / 1e9).toFixed(2)} GB -> ` + `${(next / 1e9).toFixed(2)} GB (${((next / BASE - 1) * 100).toFixed(1)}%), ` + `retention ${Math.floor(BUDGET / BASE)} -> ${Math.floor(BUDGET / next)} days`); ``` ``` budget chosen thresholds alerts caught missed false+ pages false/real 25 260 25 7 8 29 8/18 10 21 17 2.6 50 260 12 5 8 56 13/18 5 43 27 3.3 100 260 12 5 4 87 15/18 3 72 43 4.8 180 80 12 3 4 191 18/18 0 173 101 9.6 400 80 12 3 4 191 18/18 0 173 101 9.6 class breakdown of the combination at budget 100: bulk read 4/6, authorization probing 4/4, out-of-region leak 5/5, account takeover 2/3 without the region field at the same budget: caught 13/18, from the out-of-region leak 2/5 region field 14 bytes: daily 9.55 GB -> 9.66 GB (1.1%), retention 62 -> 62 days ``` The table writes that choosing a threshold is not a setting but a **purchase**. There is a combination that catches all eighteen sessions, and its price is known: 173 false positives and 101 night pages in 30 days, which is 3.4 times each night on average. There are 9.6 false positives per caught session. In the same stream, the combination with a budget of 100 catches 15 of the 18 and brings the pages down to 43; three sessions are missed. Which one is right cannot be read from this table — what can be read is that **both decisions are made in exchange for a number**. When the budget is raised from 180 to 400 the row does not change; the separating power of the metrics has run out, and accepting more false positives no longer gets anything caught. The class breakdown brings a second distinction. The combination at a budget of 100 catches all four authorization probing sessions and all five out-of-region leaks; the three it misses come from the bulk read and account takeover classes — that is, **from the metrics where normal is widest**. The missed session count is a total; which class it escaped from is separate information, and the decision is made only with that detail. The last two lines tie this lesson to the previous one. The out-of-region read scenario depends on a field that is not in the previous lesson's schema. When the field is not written, the scenario cannot be built at any threshold: the number caught at the same budget falls from 15 to 13, and of the five leak sessions only two are seen, because they happen to come out high on other metrics. **A field absent from the log removes not a scenario's threshold but its existence.** The field's logging cost is small — a 1.1% increase in daily volume, a loss of less than one day in retention. The real cost lies elsewhere: the rule requires the account's list of authorized regions; this list is staff assignment data and enters the log set as a fourth personal data item. #### Summary - Application-specific detection looks not at a protocol anomaly but at a number measured by a business rule: distinct records read, denied authorization, accounts per address, out-of-region reads. - Every threshold gives the same trade-off, and both ends are useless: a denied-authorization threshold of 3 gives 364 false positives and 211 pages, while 25 gives zero false positives and misses all four of the four sessions at once. - The source of the overlap is not the quality of the tool but normal itself: integration accounts are legitimately high-volume, and 8% of sessions produce a heavy tail. - Thresholds are chosen together and the price arises together: all 18 sessions are caught with 173 false positives and 101 pages in 30 days, 15 of them with 72 false positives and 43 pages. - The class of a missed session is information as much as its count: the three sessions that escape come from the two metrics where normal is widest. - A field absent from the log removes not the scenario's threshold but its existence: when the region field is not written the number caught falls from 15 to 13, and the cost is a 1.1% volume increase and a fourth personal data item. #### Next Step At the end of this lesson there is an alert in hand, and the alert is right: a session has read subscriber records without authorization. The questions to be asked in the first minute of the alert are different from the earlier ones, because the answer now feeds not a report but a decision — will the account be shut down, whose data has been seen, where did the intrusion begin. The next lesson runs the response steps in this application breach scenario and measures a single thing: which step needs which record, how many minutes the step is prolonged if that record does not exist, and which step cannot be completed at all. ### Application Breach Response Source: https://academia.sh/en/courses/application-detection-and-compliance/application-breach-response In an unauthorized-access breach, twelve response steps are run against four log schemas. In the base schema seven steps are left without a record, the total takes 630 minutes and two steps cannot be completed at all; in the widest schema the same response takes 151 minutes, 4.17 times shorter. The absence of a record prolongs steps by between 4 and 8.3 times. The containment decision is made at minute 10 in every schema — the difference is not in containment but in scoping: when the affected subscriber set cannot be derived, the notification widens from 380 subscribers to an upper bound of 41,200 subscribers. At the end of the previous lesson there is an alert in hand, and the alert is right: a session has read subscriber records it was not authorized to read. Once the alert is verified, the work turns from counting to doing, and the questions change — which account, from where, through which endpoints, how many subscriber records, did data change, is there another account with the same pattern. The phases of the incident response life cycle and the on-call arrangement were measured in the Security Operations and Incident Response course; they are not repeated here. This lesson measures a single thing: **which step needs which record, and how many minutes the step is prolonged if that record does not exist.** The breach scenario enters as a log; no exploitation method or payload is described, because what is measured is not how the attack was carried out but how it is understood afterward. - **DR12.** The incident is **fictional**: unauthorized reads through one account in the subscriber portal; the tenant holds 41,200 subscriber records, and the set actually touched is 380 records. - **DR13.** The **incident response playbook** is the steps modeled as a data structure. Each step has three attributes: the required log fields, the minutes it takes with the record, and the minutes it takes without the record. - **DR14.** The required fields of a step must be present in **a single joinable set**; the join rule from the previous lesson applies here as well. - **DR15.** A step whose no-record time is undefined **cannot be completed at all**: four times the time is spent on a best-effort basis, and an upper bound is reported instead of a definite result. - **DR16.** The times are **data**; they are not a measurement made on this page. The four schemas are the schemas of the earlier lessons: the base, the one with a join key added, the one with the authorization decision and version added, and the one with the session and the pre-change value added as well. #### Step–Record Dependency ```js // response.mjs — response steps are run in the application breach scenario; each step // depends on a set of records. IT IS A MODEL: both the incident and the times are fictional. export const TENANT_SUBSCRIBERS = 41200; // upper bound to notify if the definite set cannot be derived // Schema = the log designs of the previous lesson. Fields are kept in joinable sets: // fields within the same set can be joined, fields in separate sets cannot. export const SCHEMAS = [ { name: "base", sets: [["time", "trace", "actor", "action", "outcome", "path", "method", "status", "client_ip"], ["time", "table", "object", "operation"]] }, { name: "+trace", sets: [["time", "trace", "actor", "action", "outcome", "path", "method", "status", "client_ip", "table", "object", "operation"]] }, { name: "+decision+version", sets: [["time", "trace", "actor", "action", "outcome", "path", "method", "status", "client_ip", "table", "object", "operation", "authz_decision", "version"]] }, { name: "+session+value", sets: [["time", "trace", "actor", "action", "outcome", "path", "method", "status", "client_ip", "table", "object", "operation", "authz_decision", "version", "session", "old_value"]] }, ]; // Step = [name, required fields, minutes with record, minutes without record]. If the minutes // without record are null, the step cannot be completed; four times the time goes in on a best-effort // basis, and the result is an upper bound. export const STEPS = [ ["verify the alert", ["actor", "action", "outcome"], 6, 40], ["cut off account and session", ["actor"], 4, 25], ["find the entry path", ["client_ip", "trace"], 12, 70], ["list the endpoints called", ["path", "method", "trace"], 10, 60], ["why authorization was granted", ["actor", "action", "authz_decision"], 14, 100], ["affected subscriber set", ["actor", "action", "object"], 25, null], ["did data change", ["trace", "table", "operation"], 15, 120], ["roll back changed values", ["table", "old_value"], 20, null], ["other accounts, same pattern", ["client_ip", "actor"], 18, 90], ["version and patch status", ["actor", "version"], 8, 45], ["other work in the same session", ["session", "action"], 9, 75], ["write the notification scope", ["actor", "object", "time"], 10, 60], ]; export const SCOPE_STEPS = 2; // the first two steps feed the containment decision export const hasFields = (schema, fields) => schema.sets.some((c) => fields.every((f) => c.includes(f))); export function runResponse(schema) { let total = 0, containment = 0, missing = 0, uncompletable = 0, definiteSet = true; const rows = STEPS.map(([name, fields, withRec, without], i) => { const present = hasFields(schema, fields); const min = present ? withRec : (without ?? withRec * 4); if (!present) { missing++; if (without === null) { uncompletable++; if (name === "affected subscriber set") definiteSet = false; } } total += min; if (i < SCOPE_STEPS) containment += min; return { name, min, status: present ? "from log" : (without === null ? "uncompletable" : "manual") }; }); return { rows, total, containment, missing, uncompletable, notification: definiteSet ? 380 : TENANT_SUBSCRIBERS }; } export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const s = SCHEMAS[0], o = runResponse(s); console.log(`schema "${s.name}": ${STEPS.length} response steps, ` + `${s.sets.length} joinable field sets`); const y = gen([32, 34, 10, 15]); console.log("\n" + y(["step", "required fields", "minutes", "source"])); STEPS.forEach(([name, fields], i) => console.log(y([name, fields.join(","), o.rows[i].min, o.rows[i].status]))); console.log(`\ntotal ${o.total} minutes, steps without record ${o.missing}/${STEPS.length}, ` + `uncompletable ${o.uncompletable}, containment decision at minute ${o.containment}`); console.log(`affected subscriber set could not be derived: notification scope ${o.notification} subscribers`); } ``` ``` schema "base": 12 response steps, 2 joinable field sets step required fields minutes source verify the alert actor,action,outcome 6 from log cut off account and session actor 4 from log find the entry path client_ip,trace 12 from log list the endpoints called path,method,trace 10 from log why authorization was granted actor,action,authz_decision 100 manual affected subscriber set actor,action,object 100 uncompletable did data change trace,table,operation 120 manual roll back changed values table,old_value 80 uncompletable other accounts, same pattern client_ip,actor 18 from log version and patch status actor,version 45 manual other work in the same session session,action 75 manual write the notification scope actor,object,time 60 manual total 630 minutes, steps without record 7/12, uncompletable 2, containment decision at minute 10 affected subscriber set could not be derived: notification scope 41200 subscribers ``` The first four rows of the table are fast, and all of them are read from a single joinable set: who, from where, through which endpoints. These four steps take 32 minutes and are the most visible part of the response. Of the remaining eight rows, seven are without a record, and they produce 598 minutes of the total time. **What determines the duration of a response is not the speed of the first steps but which record the middle steps cannot find.** The two most expensive rows are expensive for the same reason. The "did data change" step requires the fields `trace`, `table` and `operation` together; all three fields are written, but because the database change log has no join key, it cannot be shown which request changed which row. The same happens in the "affected subscriber set" step: `actor` is in one set, `object` in the other. Neither is a missing field; both are prolonged by a join that was never built. **The cost of a missing join is one line in the measurement ledger and two hundred and twenty minutes in the response ledger.** The table also has a row that is easy to overlook, and that row prevents a misreading: the "other accounts, same pattern" step is read from the log even in the base schema, because `client_ip` and `actor` are in the same set. This step asks the response's broadest question — is the incident limited to a single account — and it is answered in eighteen minutes. **The value of a log design is not distributed by the number of steps but by which steps can be read from the log; the cheapest field can be the one that answers the broadest question.** The last two rows write down the cost of the log design that spills beyond the response. When the affected subscriber set cannot be derived definitely, an upper bound is reported in its place: all 41,200 subscribers in the tenant. The set actually touched is 380 records. The gap between them is not a reporting detail; it determines the notification obligation, the load on the call center and the sentence the organization says to the outside world. Similarly, the "roll back changed values" step cannot be completed at all because the pre-change value was not written — the data cannot be rolled back, it can only be said which rows changed. Neither step gives the spent time back: the model writes 100 and 80 minutes for these steps respectively, and at the end of that time what is left in hand is not a definite set but a range. **An uncompletable step is not a skipped step; its cost is paid twice, once in minutes and once in the certainty of the answer.** #### The Same Response, Four Log Schemas ```js // compare.mjs — the same response is run against four log schemas; the difference in log design // is translated into minutes and notification scope. IT IS A MODEL. import { SCHEMAS, STEPS, runResponse, hasFields, gen } from "./response.mjs"; const y = gen([19, 11, 17, 12, 15, 14]); console.log(y(["schema", "total min", "containment min", "no record", "uncompletable", "notification"])); const results = SCHEMAS.map((s) => { const o = runResponse(s); console.log(y([s.name, o.total, o.containment, `${o.missing}/${STEPS.length}`, o.uncompletable, o.notification])); return o; }); // Prolongation per step: how many minutes the same step takes in the base schema and in the widest schema. const base = SCHEMAS[0]; const prolonged = STEPS.map(([name, fields, withRec, without]) => ({ name, withRec, without: hasFields(base, fields) ? withRec : (without ?? withRec * 4), definite: without !== null || hasFields(base, fields) })) .filter((u) => u.without !== u.withRec).sort((a, b) => (b.without - b.withRec) - (a.without - a.withRec)); const z = gen([32, 13, 16, 8, 13]); console.log("\nsteps prolonged in the base schema (compared with the widest schema)"); console.log(z(["step", "with record", "without record", "factor", "result"])); for (const u of prolonged) console.log(z([u.name, u.withRec, u.without, (u.without / u.withRec).toFixed(1), u.definite ? "definite" : "upper bound"])); const t = results[0], g = results.at(-1); console.log(`\nbase schema ${t.total} minutes, widest schema ${g.total} minutes: ` + `${t.total - g.total} minutes difference, ${(t.total / g.total).toFixed(2)} times`); console.log(`the containment decision can be made at minute ${g.containment} in every schema; ` + `the difference is not in containment but in scoping`); console.log(`notification scope narrows from an upper bound of ${t.notification} subscribers to ` + `a definite set of ${g.notification} subscribers: ${(t.notification / g.notification).toFixed(0)} times narrower`); console.log(`the second schema (+trace) saves ${t.total - results[1].total} minutes with a single field ` + `and lowers the uncompletable steps from ${t.uncompletable} to ${results[1].uncompletable}`); ``` ``` schema total min containment min no record uncompletable notification base 630 10 7/12 2 41200 +trace 400 10 4/12 1 380 +decision+version 277 10 2/12 1 380 +session+value 151 10 0/12 0 380 steps prolonged in the base schema (compared with the widest schema) step with record without record factor result did data change 15 120 8.0 definite why authorization was granted 14 100 7.1 definite affected subscriber set 25 100 4.0 upper bound other work in the same session 9 75 8.3 definite roll back changed values 20 80 4.0 upper bound write the notification scope 10 60 6.0 definite version and patch status 8 45 5.6 definite base schema 630 minutes, widest schema 151 minutes: 479 minutes difference, 4.17 times the containment decision can be made at minute 10 in every schema; the difference is not in containment but in scoping notification scope narrows from an upper bound of 41200 subscribers to a definite set of 380 subscribers: 108 times narrower the second schema (+trace) saves 230 minutes with a single field and lowers the uncompletable steps from 2 to 1 ``` The first column is the log design translated into response time: the same incident, the same steps, the same people; a 4.17 times difference between 630 minutes and 151 minutes. The second row shows where this difference comes from. A single join key — the 18-byte `trace` added to the database record — shortens the response by 230 minutes and lowers the uncompletable steps from two to one. In the previous lesson the same field had raised the number of answered questions from six to eleven; here the second invoice for the same field is paid. The second column cuts off a misreading. The containment decision — suspending the account and cutting off the session — can be made at minute 10 in all four schemas, because that step requires only the `actor` field, and that field is present in every schema. **Log design does not change how fast the attack is stopped; what it changes is how fast you can say what happened.** The reason breach responses take long is more often not containment but scoping: whose data was touched, what changed, and whether another account shows the same pattern. The order of the schemas carries information too. The move from the first schema to the second saves 230 minutes for one field; from the second to the third, 123 minutes for two fields; from the third to the fourth, 126 minutes for two fields. The gain per field is 230, 62 and 63 minutes. The join key saves more than three times as much as the others, and it does so without recording any new fact. On the volume side the same order reverses: the join key grows the daily volume by 2%, the last two fields by 39%. **Log fields are not priced equally; there are multiples of difference both between their gains and between their costs, and the two rankings are the reverse of each other.** The prolongation table writes the cost step by step. The factor column ranges between 4.0 and 8.3, and the highest factors fall on the smallest steps: the nine-minute "other work in the same session" step takes 75 minutes without a record. The result column brings a sharper distinction. Five steps give a **definite** result even without a record — they only take longer. When two steps are left without a record, they cannot give a definite result; what the 100 and 80 spent minutes buy is an upper bound. **A step's record dependency is of two kinds: one determines the time, the other the certainty of the answer; the second cannot be closed with any amount of human hours.** The limit of the model must also be written down: the total minutes are a one-person calculation. Most steps do not wait for one another, and when three people work, the elapsed time is smaller than the total minutes. But splitting divides only the **time**; it does not divide the certainty: when all three run the "affected subscriber set" step without a record, the result is still an upper bound. In the same way, the prolongation of the steps without a record can be closed with headcount, while the two uncompletable steps cannot be closed with any staffing. **What can be shortened by adding people and what cannot are separate columns; when a response plan confuses the two, it hands a job that cannot be solved by increasing staff over to staffing.** The other face of the cost was counted in the previous lesson and must be read again here. The widest schema makes a 151-minute response possible, but raises the daily volume to 1.45 times, lowers retention from 65 days to 45, and raises the personal data items to three. With 45-day retention, an incident at the start of the 60-day audit period never gets to use those 151 minutes — the record is not there. **A field taken for response speed buys nothing once the incident's window has slipped away.** #### Summary - A response playbook is measured not as a list of steps but as a step–record dependency: the required fields, the time with the record and the time without the record of each step are written separately. - In the base schema 7 of the 12 steps are left without a record and the response takes 630 minutes; in the widest schema the same response takes 151 minutes, a 4.17 times difference. - Most of the prolongation comes not from a missing field but from a join that was never built; a single join key saves 230 minutes and lowers the uncompletable steps from two to one. - Log design does not change the speed of the containment decision — minute 10 in every schema — only the speed of scoping. - Record dependency is of two kinds: five steps give a definite result even without a record but take 5.6 to 8.3 times longer; two steps cannot give a definite result, and the notification widens from 380 subscribers to an upper bound of 41,200 subscribers. - The cost of a fast response is written in the previous lesson's ledger: 1.45 times the volume, retention falling from 65 days to 45, and three personal data items. #### Next Step The whole calculation of this lesson rested on one assumption: if a log field was written, it is in hand at the time of the response. In a persistent service this is true. In short-lived workloads it is not — when a task container lives for forty seconds and is terminated, only the log items it managed to write out can be queried; the rest goes away with the process. The next lesson measures this window: which application items can get out before the workload dies, what happens to the share of collectable items as the level of detail rises, and how many of the earlier lessons' questions the items left in hand can answer. ### Trace in Short-Lived Workloads Source: https://academia.sh/en/courses/application-detection-and-compliance/trace-in-short-lived-workloads 2,000 short-lived workloads produce 671,501 log items and 642,504 of them are written out: the loss is only 4.3%. In the same data, the last item of the workload can be collected in only 115 of 2,000 workloads, and not a single item comes out of the 92 workloads of the failed exit class. When the level of detail is raised eight times, the items written out rise to 4,272,000, but the number of workloads that answer the authorization decision question falls from 1,218 to zero. The cost of the setting that collects all items is 91,084 extra flush requests and 33 core-minutes of extra lifetime. The whole calculation of the previous lesson rested on one assumption: if a log field was written, it is in hand at the time of the response. In a persistent service this is true. In short-lived workloads it is not. When a task container lives for forty seconds and is terminated, only the items it produced that were **written out** can be queried; the rest goes away with the process. The difference between collecting evidence in a cloud environment and in persistent systems was measured in the Cloud and Virtualization Security course; it is not repeated here, it becomes an input. This lesson measures only the **application trace**: which application items have been written out by the time the workload dies. The measure is called the **collectable item window**, and it cannot be reduced to a single number, because the lost items are not randomly distributed within the stream — they are all at the end. - **DR17.** The workload set is **fictional**: 2,000 runs, four life bands. The lives come from a generator whose seed is visible. - **DR18.** Items are produced at a constant rate within the workload and written to a local **buffer**. The shipper writes out at most `batch` items every `interval` seconds. - **DR19.** When the workload terminates, everything left in the buffer is **lost**. A final flush can be made only if the grace period covers the flush duration; in the model the flush takes 1.2 seconds and the default grace period is 0.5 seconds. - **DR20.** A **question** depends on the item produced at a relative position in the workload's life. If the item was not written out, that question is unanswered for that workload; the unit of measure is again the unanswered question. - **DR21.** The durations and rates are **data**; they are not a measurement made on this page. #### The Collectable Item Window ```js // trace.mjs — the window of items that can be written out in short-lived workloads is measured. // IT IS A MODEL: the workloads, the lives and the item rates are fictional. export const SEED = 20260502, WORKLOAD_COUNT = 2000; export const RATE = 6, INTERVAL = 5, BATCH = 200, FLUSH_TIME = 1.2, GRACE = 0.5; const rng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = rng(SEED); // Class = the life band of the workload. The failed exit lives shortest and loses most. export const CLASSES = [ { name: "short task", share: 0.55, lo: 3, hi: 20 }, { name: "batch job", share: 0.25, lo: 20, hi: 120 }, { name: "request handler", share: 0.15, lo: 60, hi: 400 }, { name: "failed exit", share: 0.05, lo: 1, hi: 4 }, ]; const pickClass = (u) => { let t = 0; for (const s of CLASSES) { t += s.share; if (u < t) return s; } return CLASSES.at(-1); }; export const WORKLOADS = Array.from({ length: WORKLOAD_COUNT }, () => { const s = pickClass(r()); return { cls: s.name, life: s.lo + r() * (s.hi - s.lo) }; }); // The buffer is FIFO: each flush carries at most BATCH items. Whatever is left in the buffer // when the workload dies is gone; the final flush happens only if the grace period covers the flush duration. export function run(life, rate = RATE, interval = INTERVAL, grace = GRACE) { const items = Math.max(1, Math.floor(life * rate)); let sent = 0, requests = 0; const flush = (t) => { const produced = Math.min(items, Math.floor(t * rate) + 1); const g = Math.min(BATCH, produced - sent); if (g > 0) { sent += g; requests++; } }; for (let t = interval; t <= life; t += interval) flush(t); if (grace >= FLUSH_TIME) flush(life); return { items, sent, lost: items - sent, requests }; } // Question = [text, relative position of the item in the workload's life]. If the item was not // written out, the question is unanswered for that workload. export const QUESTIONS = [ ["which account sent the request", 0.00], ["which endpoint was called", 0.10], ["which dependency was called", 0.45], ["what was the authorization decision", 0.85], ["which error ended the workload", 1.00], ]; export function aggregate(rate = RATE, interval = INTERVAL, grace = GRACE) { let items = 0, sent = 0, requests = 0; const answerCounts = QUESTIONS.map(() => 0); for (const w of WORKLOADS) { const o = run(w.life, rate, interval, grace); items += o.items; sent += o.sent; requests += o.requests; QUESTIONS.forEach(([, p], j) => { if (Math.floor(p * (o.items - 1)) < o.sent) answerCounts[j]++; }); } return { items, sent, lost: items - sent, requests, answerCounts }; } export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model (seed ${SEED}): ${WORKLOAD_COUNT} workloads, ${RATE} items/s, flush ` + `interval ${INTERVAL} s, batch ${BATCH} items, grace period ${GRACE} s, ` + `flush duration ${FLUSH_TIME} s`); const y = gen([18, 8, 11, 12, 12, 11, 9]); console.log("\n" + y(["workload class", "count", "mean life", "produced", "written out", "lost", "ratio"])); for (const s of CLASSES) { const l = WORKLOADS.filter((w) => w.cls === s.name).map((w) => run(w.life)); const pr = l.reduce((t, o) => t + o.items, 0), g = l.reduce((t, o) => t + o.sent, 0); const lf = WORKLOADS.filter((w) => w.cls === s.name).reduce((t, w) => t + w.life, 0); console.log(y([s.name, l.length, (lf / l.length).toFixed(1), pr, g, pr - g, (g / pr).toFixed(2)])); } const t = aggregate(); console.log(`\ntotal ${t.items} items produced, ${t.sent} written out, ` + `${t.lost} lost (${((t.lost / t.items) * 100).toFixed(1)}%)`); const z = gen([38, 10, 14, 8]); console.log("\n" + z(["question", "position", "answered", "ratio"])); QUESTIONS.forEach(([m, p], j) => console.log(z([m, p.toFixed(2), `${t.answerCounts[j]}/${WORKLOAD_COUNT}`, (t.answerCounts[j] / WORKLOAD_COUNT).toFixed(2)]))); } ``` ``` model (seed 20260502): 2000 workloads, 6 items/s, flush interval 5 s, batch 200 items, grace period 0.5 s, flush duration 1.2 s workload class count mean life produced written out lost ratio short task 1091 11.5 74404 57738 16666 0.78 batch job 536 71.2 228687 221405 7282 0.97 request handler 281 217.8 367056 363361 3695 0.99 failed exit 92 2.5 1354 0 1354 0.00 total 671501 items produced, 642504 written out, 28997 lost (4.3%) question position answered ratio which account sent the request 0.00 1757/2000 0.88 which endpoint was called 0.10 1757/2000 0.88 which dependency was called 0.45 1757/2000 0.88 what was the authorization decision 0.85 1218/2000 0.61 which error ended the workload 1.00 115/2000 0.06 ``` The two tables are two different readings of the same run, and they look nothing alike. In the top table the total loss is 4.3%. On a dashboard this number reads as "log collection is healthy"; and indeed, of the 671,501 items produced, 642,504 were written out. There is no fault anywhere in the log pipeline; the shipper is running, the network is up, the destination is writable. The bottom table reads the same data per question and says something else. The items produced in the first half of the workload's life can be collected in 1,757 of the 2,000 workloads. The item where the authorization decision is written, in 1,218 of them, and the last item, which says how the workload ended, in only **115**. That is, the question "which error ended the workload" is unanswered in 94% of the workloads — even though the log was written. **The lost-item rate and the answerability rate are not the same thing, because the lost items are not randomly distributed within the stream; they are all at the end, and the items most wanted in a response are also the ones at the end.** The class table writes down where the loss concentrates. The request handler class lives 218 seconds on average and writes out 99% of its items. The failed exit class lives 2.5 seconds on average and **not a single one of its items comes out**: 92 workloads, 1,354 items, zero. The flush interval is five seconds; a workload that lives two and a half seconds never reaches the first periodic flush, and since the grace period does not cover the flush duration, the final flush cannot be made. This class is only two in a thousand of the total items, but it is the class most looked at in a response: a workload ending unexpectedly and quickly is already the starting reason for the investigation. The reason the total loss rate misleads is in this table too. The total is an average weighted by the item count, and most items are produced by long-lived workloads: the 281 request handlers alone write 367,056 items, while the 1,091 short tasks write 74,404. The long-lived class's 99% rate pulls the total up, while the short-lived class's 78% and the failed exit's zero are lost in the total. **A collection rate that is not measured per workload reports the rate of the class that produces the most items as the rate of the whole environment.** When the same data is read per workload, it turns out that 92 workloads are completely silent, and that number is written nowhere in the total rate. #### Level of Detail and Collection Settings ```js // settings.mjs — two setting axes are run: the level of detail (items/s) and the flush layout // (interval, grace period). What is measured is not the item count but the answered question. IT IS A MODEL. import { RATE, INTERVAL, GRACE, WORKLOAD_COUNT, QUESTIONS, aggregate, gen } from "./trace.mjs"; const y = gen([10, 11, 12, 8, 14, 12, 17]); console.log(y(["detail", "produced", "written out", "lost", "authz (0.85)", "last (1.00)", "answered pairs"])); for (const mult of [1, 2, 4, 8]) { const t = aggregate(RATE * mult); const pairs = t.answerCounts.reduce((a, b) => a + b, 0); console.log(y([`${mult}x`, t.items, t.sent, ((t.lost / t.items) * 100).toFixed(1) + "%", (t.answerCounts[3] / WORKLOAD_COUNT).toFixed(2), (t.answerCounts[4] / WORKLOAD_COUNT).toFixed(2), `${pairs}/${QUESTIONS.length * WORKLOAD_COUNT}`])); } const z = gen([9, 10, 12, 14, 12, 12, 16]); console.log("\n" + z(["interval", "grace", "written out", "authz (0.85)", "last (1.00)", "requests", "extra core-sec"])); for (const grace of [GRACE, 1.5]) for (const interval of [5, 2, 1]) { const t = aggregate(RATE, interval, grace); console.log(z([interval, grace, t.sent, (t.answerCounts[3] / WORKLOAD_COUNT).toFixed(2), (t.answerCounts[4] / WORKLOAD_COUNT).toFixed(2), t.requests, (WORKLOAD_COUNT * (grace - GRACE)).toFixed(0)])); } const base = aggregate(), dense = aggregate(RATE * 8), wide = aggregate(RATE, 1, 1.5); console.log(`\nwhen detail is raised 8 times, items written out rise from ${base.sent} to ` + `${dense.sent} but the authorization decision question falls from ` + `${base.answerCounts[3]} workloads to ${dense.answerCounts[3]} workloads`); console.log(`the widest setting (interval 1 s, grace period 1.5 s) writes out the last item in ` + `${wide.answerCounts[4]} workloads instead of ${base.answerCounts[4]} at the same detail; ` + `its cost is ${wide.requests - base.requests} extra flush requests and ` + `${(WORKLOAD_COUNT * (1.5 - GRACE) / 60).toFixed(0)} core-minutes of extra lifetime`); ``` ``` detail produced written out lost authz (0.85) last (1.00) answered pairs 1x 671501 642504 4.3% 0.61 0.06 6604/10000 2x 1344000 1283327 4.5% 0.59 0.03 6500/10000 4x 2688990 2564937 4.6% 0.58 0.01 6454/10000 8x 5378988 4272000 20.6% 0.00 0.00 5221/10000 interval grace written out authz (0.85) last (1.00) requests extra core-sec 5 0.5 642504 0.61 0.06 21360 0 2 0.5 662111 0.89 0.15 55024 0 1 0.5 668090 1.00 0.31 111065 0 5 1.5 671501 1.00 1.00 23245 2000 2 1.5 671501 1.00 1.00 56715 2000 1 1.5 671501 1.00 1.00 112444 2000 when detail is raised 8 times, items written out rise from 642504 to 4272000 but the authorization decision question falls from 1218 workloads to 0 workloads the widest setting (interval 1 s, grace period 1.5 s) writes out the last item in 2000 workloads instead of 115 at the same detail; its cost is 91084 extra flush requests and 33 core-minutes of extra lifetime ``` The first table gives the fact this course shows for the second time. When the level of detail is raised eight times, the items written out rise from 642,504 to 4,272,000 — six and a half times more records. The same move lowers the number of workloads answering the authorization decision question from 1,218 to zero. The mechanism appears not in the last column but in the lost column: at eight times the detail the production rate rises to 48 items per second, while the shipper carries at most 200 items in five seconds, that is, 40 per second. When production exceeds carrying capacity, the buffer **starts to accumulate**, and the end of the stream never gets its turn. **Writing more records, with the carrying capacity fixed, pushes the last-written item further back in the queue; the total items rise, the answered questions fall.** The three rows above this break are the soft form of the same fact. When the detail is raised two and four times, the capacity is not yet exceeded; even so, the answered question–workload pairs fall from 6,604 to 6,454. There is no gain, because the added items fall inside the already collectable window; what is lost is again the item at the end. The second table writes what the two settings buy. Lowering the flush interval from five seconds to one raises the authorization decision question from 0.61 to 1.00, but the last item is still collected in only 31% of the workloads — because the problem is not the interval, it is that the flush at the moment of closing cannot be made at all. When the grace period is extended enough to cover the flush duration, all questions are answered in every workload even with the five-second interval. **What determines the collectable item window is not how often the flush happens, but that the workload can complete a flush before it dies.** The cost is written in two columns. Lowering the interval to one raises the flush requests from 21,360 to 111,065: more than five times the network requests, fifty-five requests per workload. Extending the grace period by one second brings 2,000 core-seconds, that is, 33 core-minutes of extra lifetime, and delays the termination of every workload. Together they cost 91,084 extra requests. A third cost is not visible in the table but it is in the ledger of the earlier lessons: every item written out falls inside the fixed retention budget of the previous lesson and shortens the retention days. #### Summary - In short-lived workloads the measure is not the items written but the items that can be **written out**; whatever is left in the buffer when the workload dies is lost. - The loss rate and the answerability rate are separate numbers: in the model, while the loss is 4.3%, the last item of the workload can be collected in only 115 of 2,000 workloads. - The loss is not equally distributed across classes: not a single item comes out of the 92 workloads of the failed exit class, which lives 2.5 seconds on average, yet the starting reason for an investigation is most often this class. - Raising the level of detail can lower the answer: at eight times the detail the items written out grow six and a half times while the number of workloads answering the authorization decision question falls from 1,218 to zero, because the production rate exceeds the carrying capacity and the end of the stream never gets its turn. - What determines the window is not the flush frequency but the workload being able to complete a flush before it dies: when the grace period is extended, even the five-second interval is enough. - The cost is measured: 91,084 extra flush requests, 33 core-minutes of extra lifetime, and the share every item written out takes from the fixed retention budget. #### Next Step Up to this point all the logs, alerts and items have come from within the organization itself. A vulnerability in an application is often heard of from somewhere else: someone looking from the outside finds it and reports it. This flow is not a detection pipeline, it is a work queue; its input is reports written by people, its output is items that are fixed or rejected, and its measure is not the item count but **time**. The next lesson measures the external report intake and the bug bounty program on this axis: how many days pass from report to verification and from verification to fix, how many of the incoming reports are duplicates, and at what point opening the channel clogs the queue. ### External Report Intake Source: https://academia.sh/en/courses/application-detection-and-compliance/external-notification-channel Of the 340 reports arriving in 180 days, 91 are unique; 190 are duplicates and 59 are out of scope, so 73% of the queue carries no new vulnerability. At a triage capacity of two items per day the queue rises to 104 items and the mean wait is 31.2 days. Run against four log schemas, verification falls from 28.6 days to 25.8 days but the time from report to fix rises from 85.7 days to 89.9 days. The definite answer to the exploitation question rises from 15 to 20 and then falls to 13 in the widest schema; what raises the same number to 24 is not an extra field but raising triage capacity from 2 to 5. Up to this point all the logs and alerts have come from within the organization itself. A vulnerability in an application is often heard of from somewhere else: someone looking from the outside finds it and reports it. The **external report intake and bug bounty program** is the structure these reports enter. The reporter's process was measured in the Security Tooling and Penetration Testing course; it is not repeated here, it becomes an input. This lesson measures the side that **operates** the flow, and its unit is the day: how many days from report to verification, how many days from verification to fix, how many of the incoming reports are duplicates. - **DR22.** The report flow is **fictional**: 340 reports in 180 days, 120 vulnerabilities, five finding classes. The numbers come from a generator whose seed is visible. - **DR23.** The same vulnerability is reported more than once. The first report of a vulnerability is **unique**, the later ones are **duplicates**; a report that falls under no vulnerability is **out of scope**. - **DR24.** The **triage queue** works in arrival order; the capacity is consumed by the **number** of reports, not by their kind. - **DR25.** Verification is done in two ways: if the required fields are in a single joinable set, **from the log** in one day; if not, by **manual reproduction** in three to nine days depending on the class. - **DR26.** Every report raises a second question: **has this vulnerability been exploited before.** The log reaches back `retention` days from the verification day, and the time spent in the queue is subtracted from this window. There are three outcomes: **definite**, **partial** (it was possible to look back only as far as the window) and **unanswered** (the field does not exist). - **DR27.** The schemas and the retention days come as **data** from the model of the earlier lessons. The reward is a **fictional unit**; the durations are not a measurement made on this page. #### Incoming Reports, Duplicates and the Triage Queue ```js // report.mjs — the external report intake and bug bounty program are measured from the side that // operates the flow: how many reports arrive, how many are duplicates, how many days pass until // triage. IT IS A MODEL. export const SEED = 20260611, VULN_COUNT = 120, REPORT_COUNT = 340, PERIOD = 180, TRIAGE = 2; const rng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = rng(SEED); // visibility: how many times the vulnerability will be reported. fields: the log fields the // exploitation question needs. export const CLASSES = [ { name: "configuration", share: 0.26, visibility: 3.4, manual: 3, fix: 2, reward: 8, fields: ["path", "status"] }, { name: "input validation", share: 0.21, visibility: 2.2, manual: 5, fix: 3, reward: 15, fields: ["trace", "path", "method"] }, { name: "authorization bypass", share: 0.20, visibility: 1.0, manual: 8, fix: 6, reward: 40, fields: ["actor", "action", "authz_decision"] }, { name: "data leak", share: 0.18, visibility: 1.4, manual: 9, fix: 8, reward: 55, fields: ["actor", "action", "object"] }, { name: "session management", share: 0.15, visibility: 0.8, manual: 7, fix: 5, reward: 30, fields: ["session", "action"] }, ]; const pickClass = (u) => { let t = 0; for (const s of CLASSES) { t += s.share; if (u < t) return s; } return CLASSES.at(-1); }; // Vulnerability = the real item behind a report; the same vulnerability is reported in proportion // to its visibility. export const VULNS = Array.from({ length: VULN_COUNT }, () => { const s = pickClass(r()); return { cls: s.name, weight: s.visibility * (0.3 + 1.4 * r()), age: Math.round(8 + 340 * r() ** 1.6) }; }); const TOTAL = VULNS.reduce((t, a) => t + a.weight, 0); const pickVuln = (u) => { let t = 0; for (let i = 0; i < VULN_COUNT; i++) { t += VULNS[i].weight / TOTAL; if (u < t) return i; } return VULN_COUNT - 1; }; // When the channel opens, the volume piles up at the front: arrival day PERIOD * u^2.2. 18% is out of scope. export const REPORTS = Array.from({ length: REPORT_COUNT }, (_, i) => ({ i, day: Math.floor(PERIOD * r() ** 2.2), vuln: r() < 0.18 ? null : pickVuln(r()), })).sort((a, b) => a.day - b.day); const seen = new Set(); for (const b of REPORTS) { b.kind = b.vuln === null ? "out-of-scope" : seen.has(b.vuln) ? "duplicate" : "unique"; seen.add(b.vuln); } export const count = (x) => REPORTS.filter((b) => b.kind === x).length; // Triage queue: `rate` items per day, in arrival order; the capacity is consumed by the count, not the kind. export function triage(rate = TRIAGE, write = false) { const queue = [], reviewed = new Array(REPORT_COUNT); let peak = 0, j = 0; for (let g = 0; j < REPORT_COUNT || queue.length; g++) { while (j < REPORT_COUNT && REPORTS[j].day === g) queue.push(REPORTS[j++]); peak = Math.max(peak, queue.length); for (let k = 0; k < rate && queue.length; k++) { const b = queue.shift(); reviewed[b.i] = g; if (write) b.reviewed = g; } } return { peak, last: Math.max(...reviewed), wait: REPORTS.reduce((t, b) => t + (reviewed[b.i] - b.day), 0) / REPORT_COUNT }; } export const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); if (import.meta.url === `file://${process.argv[1]}`) { const noise = count("duplicate") + count("out-of-scope"); console.log(`model (seed ${SEED}): ${REPORT_COUNT} reports in ${PERIOD} days, ${VULN_COUNT} vulnerabilities, ` + `${REPORTS.filter((b) => b.day < 30).length} reports in the first 30 days`); console.log(`unique ${count("unique")}, duplicate ${count("duplicate")}, out-of-scope ` + `${count("out-of-scope")}: ${((noise / REPORT_COUNT) * 100).toFixed(0)}% are not new vulnerabilities`); const y = gen([22, 10, 11, 11, 10]); console.log("\n" + y(["class", "reports", "unique", "duplicate", "mean age"])); for (const s of CLASSES) { const l = REPORTS.filter((b) => b.vuln !== null && VULNS[b.vuln].cls === s.name); const be = l.filter((b) => b.kind === "unique"); console.log(y([s.name, l.length, be.length, l.length - be.length, (be.reduce((a, b) => a + VULNS[b.vuln].age, 0) / be.length).toFixed(0)])); } const z = gen([10, 9, 15, 12]); console.log("\n" + z(["capacity", "peak", "mean wait", "last review"])); for (const rate of [1, 2, 3, 5]) { const t = triage(rate); console.log(z([`${rate}/day`, t.peak, t.wait.toFixed(1), t.last])); } } ``` ``` model (seed 20260611): 340 reports in 180 days, 120 vulnerabilities, 149 reports in the first 30 days unique 91, duplicate 190, out-of-scope 59: 73% are not new vulnerabilities class reports unique duplicate mean age configuration 163 35 128 151 input validation 44 20 24 120 authorization bypass 35 17 18 153 data leak 24 11 13 136 session management 15 8 7 108 capacity peak mean wait last review 1/day 174 116.1 339 2/day 104 31.2 177 3/day 70 10.4 177 5/day 49 2.3 177 ``` Of the three hundred and forty reports, ninety-one carry a new vulnerability. The remaining 249 items — 190 duplicates and 59 out of scope — are 73% of the queue. This rate comes not from the channel's poor quality but from its **structure**: the channel is open to everyone, everyone who sees the same vulnerability reports it, and who reported earlier is not visible from the outside. The class table writes down where the pile-up is. The configuration class produces 163 reports and only 35 of them are a new vulnerability; in the session management class there are 15 reports and 8 unique vulnerabilities. **The share of a finding class in the channel measures not how many vulnerabilities that class leaves in the organization, but how readily it is seen from the outside.** The composition of the queue is a photograph not of the organization's vulnerability distribution but of the external visibility distribution. The last column prepares the second half of this lesson. The mean age of the reported vulnerabilities is between 108 and 153 days: when a vulnerability is reported, it has already been there for months. The question "has this vulnerability been exploited before" requires looking back over a history longer than a hundred days; the retention windows of the earlier lessons were between 45 and 65 days. The queue table gives the third number. At a triage capacity of two items per day the queue rises to 104 items and a report waits 31.2 days on average. When capacity drops to one, the last report is reviewed on day 339: the period is 180 days, so the channel cannot catch up with its own period. The cause of the pile-up is not the mean volume; 149 reports arrive in the first 30 days, and 44% of the period total falls in the first sixth. **Opening the channel creates a load curve; when capacity is chosen according to the mean arrival, it clogs at the peak, and the clog is not one-time but permanent.** #### Verification, the Exploitation Question and the Fix Queue ```js // verification.mjs — unique reports are run against four log schemas: is verification done from // the log or manually, is the exploitation question answered, when is the fix. IT IS A MODEL. import { REPORTS, VULNS, CLASSES, triage, count, gen, REPORT_COUNT } from "./report.mjs"; const K1 = ["time", "trace", "actor", "action", "outcome", "path", "method", "status", "client_ip"]; const K2 = ["time", "table", "object", "operation"]; const EXTRA = ["authz_decision", "version", "session", "old_value"]; export const SCHEMAS = [ { name: "base", retention: 65, factor: 1.00, extra: 0, sets: [K1, K2] }, { name: "+trace", retention: 64, factor: 1.02, extra: 1, sets: [[...K1, ...K2]] }, { name: "+decision+version", retention: 62, factor: 1.04, extra: 3, sets: [[...K1, ...K2, ...EXTRA.slice(0, 2)]] }, { name: "+session+value", retention: 45, factor: 1.45, extra: 5, sets: [[...K1, ...K2, ...EXTRA]] }, ]; export const CAPACITY = 2; // work-days set aside per day for fixes const hasFields = (schema, a) => schema.sets.some((c) => a.every((x) => c.includes(x))); // Verification is 1 day from the log; without a record it is the class's manual reproduction days. In // the exploitation question the log reaches back `retention` days; the `duration` spent in the queue // is subtracted from the window. export function run(schema, rate = 2) { triage(rate, true); const items = REPORTS.filter((b) => b.kind === "unique").map((b) => { const a = VULNS[b.vuln], s = CLASSES.find((x) => x.name === a.cls); const fromLog = hasFields(schema, s.fields); const duration = b.reviewed - b.day + (fromLog ? 1 : s.manual); return { b, s, fromLog, duration, verified: b.day + duration, exploit: !fromLog ? "unanswered" : a.age <= schema.retention - duration ? "definite" : "partial" }; }).sort((x, y) => x.verified - y.verified); // Fix queue: CAPACITY work-days are added to the pool each day, the one with the higher reward takes its turn first. const waiting = []; let j = 0, pool = 0; for (let g = items[0].verified; j < items.length || waiting.length; g++) { while (j < items.length && items[j].verified <= g) waiting.push(items[j++]); pool = waiting.length ? pool + CAPACITY : CAPACITY; waiting.sort((x, y) => y.s.reward - x.s.reward); while (waiting.length && pool >= waiting[0].s.fix) { const k = waiting.shift(); pool -= k.s.fix; k.fixed = g; } } const n = items.length, sum = (f) => items.reduce((t, k) => t + f(k), 0); const tally = (x) => items.filter((k) => k.exploit === x).length; return { n, items, fromLog: items.filter((k) => k.fromLog).length, verifyDays: sum((k) => k.duration) / n, totalDays: sum((k) => k.fixed - k.b.day) / n, fixDays: sum((k) => k.fixed - k.verified) / n, reward: sum((k) => k.s.reward), definite: tally("definite"), partial: tally("partial"), unanswered: tally("unanswered") }; } if (import.meta.url === `file://${process.argv[1]}`) { const y = gen([18, 11, 10, 14, 12, 10, 9, 12]); console.log(y(["schema", "retention", "from log", "verification", "fix", "definite", "partial", "unanswered"])); const results = SCHEMAS.map((s) => { const o = run(s); console.log(y([s.name, `${s.retention} days`, `${o.fromLog}/${o.n}`, `${o.verifyDays.toFixed(1)} days`, `${o.fixDays.toFixed(1)} days`, o.definite, o.partial, o.unanswered])); return o; }); const [a, , c, d] = results, h5 = SCHEMAS.map((s) => run(s, 5)); const el = a.items.filter((k) => !k.fromLog), o = (l) => l.reduce((t, k) => t + k.s.reward, 0); console.log(`\nfrom report to fix rises from ${a.totalDays.toFixed(1)} days to ` + `${d.totalDays.toFixed(1)} days; the definite answer peaks at ${c.definite} in the ${SCHEMAS[2].name} schema ` + `and is ${d.definite} in the ${SCHEMAS[3].name} schema`); console.log(`when capacity rises from 2 to 5, definite answers are ${h5.map((x) => x.definite).join("/")} ` + `instead of ${results.map((x) => x.definite).join("/")}: without adding a field, ` + `${h5[3].definite - d.definite} more reports get a definite answer`); console.log(`the reward of the ${el.length} manually verified reports is ${o(el)} fictional units, ` + `of the ${a.n - el.length} verified from the log ${a.reward - o(el)}`); } ``` ``` schema retention from log verification fix definite partial unanswered base 65 days 55/91 28.6 days 57.1 days 15 40 36 +trace 64 days 66/91 27.6 days 60.8 days 18 48 25 +decision+version 62 days 83/91 26.3 days 63.6 days 20 63 8 +session+value 45 days 91/91 25.8 days 64.2 days 13 78 0 from report to fix rises from 85.7 days to 89.9 days; the definite answer peaks at 20 in the +decision+version schema and is 13 in the +session+value schema when capacity rises from 2 to 5, definite answers are 18/21/24/24 instead of 15/18/20/13: without adding a field, 11 more reports get a definite answer the reward of the 36 manually verified reports is 1525 fictional units, of the 55 verified from the log 580 ``` The verification column looks like a triumph of log design: the reports verified from the log rise from 55 to 91, and the mean verification falls from 28.6 days to 25.8 days. The gain is 2.8 days. The same schemas had shortened the response 4.17 times in the previous lesson; here the ratio is 1.11, because the dominant term of the duration is not the verification work but **the wait in the queue**. **A log field that shortens a job shortens the total time, when the job is behind a queue, only as much as the queue leaves it.** The fix column goes the opposite way: from 57.1 days to 64.2 days, and the total time from report to fix from 85.7 to 89.9 days. As verification speeds up, items enter the fix queue earlier, while the queue capacity is fixed; **the day that was gained is carried to the next queue and waited out there.** The bottleneck is not the log design but the fix capacity of two work-days a day. The last three columns write the measure of this course. The unanswered questions fall from 36 to zero as the schema widens: in the widest schema the exploitation question can **be asked** for all 91 reports. But the definite answer rises from 15 to 20 and then falls to 13, below even the base schema. The widest schema leaves no question unanswered and gives the fewest definite answers. The mechanism is in DR26: the condition of a definite answer is that the age of the vulnerability fits in the window of **retention minus the elapsed time**. In the widest schema the retention is 45 and the mean duration is 25.8 days; a window of about 19 days remains, while the mean age of the reported vulnerabilities is between 108 and 153 days. **Every day spent in the queue is subtracted from the retention window.** This is the bill, in the external report flow, of the design that trades volume against retention: in return for 5 extra fields, 1.45 times the daily volume and 20 days shorter retention, the definite answer **falls** by two units. The third output line plays the same number with a second lever. When the triage capacity is raised from two to five, the definite-answer series becomes 18/21/24/24 instead of 15/18/20/13; this increase in the widest schema is gained **without adding any log field and without paying any volume**. **Triage capacity looks like a service-level decision; when measured, it turns out to be a logging decision.** The reward side writes the last column of the cost. In the base schema the reward total of the 36 manually verified reports is 1,525 fictional units, and that of the 55 verified from the log is 580 units. The high-reward classes are the ones that cannot be verified from the log; the cheap and often-reported configuration class is closed in a day. **The verification load and the reward budget go in the same direction: the most expensive findings are the slowest to verify.** The 249 items that receive no reward consume 73% of the triage capacity; the real cost of the bounty program is not in the unit paid but in the line held by the items that are not paid. #### Summary - The external report intake is not a detection pipeline but a queue: in the model, of 340 reports, 91 carry a new vulnerability, 190 are duplicates and 59 are out of scope. - The composition of the queue reflects not the organization's vulnerability distribution but the external visibility distribution: the configuration class fills half of the channel with 163 reports, behind which there are only 35 vulnerabilities. - Capacity cannot be chosen according to the mean arrival: 149 reports arrive in the first 30 days, at a capacity of two items per day the queue rises to 104 items and the mean wait becomes 31.2 days. - Adding log fields lowers verification from 28.6 days to 25.8 days, but the time from report to fix rises from 85.7 days to 89.9 days: the day that was gained is carried to the fix queue, whose capacity is fixed. - The days spent in the queue are subtracted from the retention window; while the widest schema lowers the unanswered questions to zero, it lowers the definite answer from 20 to 13, below the 15 of the base schema. - The cost is measured on two levers: while 5 extra fields, 1.45 times the volume and 20 days of shorter retention reduce the definite answer, raising triage capacity from two to five carries the same number to 24 without adding a field. #### Next Step Throughout this topic a log was designed, an alert threshold was chosen, a breach was responded to, the trace of short-lived workloads was measured and an external report flow was set up. Every lesson asked the same question: how many of the questions to be asked after the incident can be answered, and what does it cost to make them answerable. One question remains, and the organization itself does not ask it: what will show someone looking from the outside that all this practice is **being done**. The next topic measures this — how many practices an audit control item corresponds to in engineering terms, for how many of them the evidence is generated automatically in the pipeline, and how many are collected by hand after the incident. ## Application Compliance Counting the engineering equivalent of an audit item: how many of twenty-six controls fall on a practice the team actually runs and how many fall on none, whether the answer to a control comes from a query, a document, or an individual's declaration, how far generating evidence in the pipeline reduces the items collected by hand and what cost automatic evidence creates in volume, retention, and personal data, how mean time to remediate and open finding age read in opposite directions from the same data, and how many items lack the timestamp the metric needs in order to be computed. ### Engineering Equivalent of Controls Source: https://academia.sh/en/courses/application-detection-and-compliance/engineering-equivalent-of-controls Twenty-six audit items are matched with the eighteen practices of an application team: after 25 links, 17 controls have an equivalent, 9 controls have no engineering equivalent at all, and five of those are gathered in a single compliance source. A control with an equivalent falls on 1.47 practices on average, 10 controls rest on a single practice, 2 practices fall on no control, and only 11 of the 26 questions are answered by querying the record. The previous topic measured the incident itself: the log was designed, application-specific detection scenarios were run, the records each response step needs were counted, and the external report flow was measured by time. All of that practice left one shared gap — the evidence that would show someone looking at the organization from outside that the work was done is collected by hand at the end of the period. The four lessons of this topic measure that gap. The first step is the mapping itself. In the Governance, Risk and Compliance course, the matrix between controls and requirements was run, and the requirements left without evidence and the controls satisfying more than one requirement were counted; that measurement is the input here and is not repeated. The ends of the matrix here are different: at one end is an audit item, at the other **the work an application team actually runs**. The question is different too — which of the team's work does an audit item fall on, how many of its work items does it fall on, and **how many items fall on none of its work.** #### The Control Set and the Practice Set **AC1.** The control set has 26 items and is gathered from four compliance sources: a function-based framework, a certifiable management system standard, an audit control set, and a sector compliance requirement. The set is **fictional**; no source, regulation, or organization is named, and the items are written **only in terms of what they ask**. **AC2.** The practice set has 18 items and consists of the work run by the subscriber portal team of the fictional regional measurement network. **AC3.** Every practice has a **site** — pipeline, repo, runtime, or manual — and a **trace**: a machine-readable record or a document. **AC4.** The mapping is a **claim**: a control falling on a practice says that the practice carries out the control, not that it produces evidence; evidence is separated out in the third section. **AC5.** The matrix sees only the work the team **itself** carries out; work done by the platform or by another team does not appear here, and this makes the number of controls with no equivalent look higher than it is. ```js // compliance/set.mjs — audit controls and team practices are a MODEL: fictional regional measurement network. // The controls come from four compliance sources, the practices are the work of an application team. export const split = (s) => s.trim().split("\n").map((l) => l.trim().split("|")); export const gen = (W) => (...h) => h.map((v, j) => (j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))).join(""); // code | compliance source | what the control asks export const CONTROLS = split(` FF-01|function-framework|component inventory is kept current FF-02|function-framework|access is granted with least privilege FF-03|function-framework|data in transit and at rest is encrypted FF-04|function-framework|security events are collected centrally FF-05|function-framework|anomalous behavior raises an alert FF-06|function-framework|incident response roles are defined FF-07|function-framework|recovery objective is tested FF-08|function-framework|supplier component is assessed MS-01|management-system|scope statement is updated MS-02|management-system|applicability statement is justified MS-03|management-system|internal audit runs periodically MS-04|management-system|corrective action is closed MS-05|management-system|management review is recorded MS-06|management-system|staff competence is recorded CS-01|audit-set|change goes through approval CS-02|audit-set|production access is granted with a second pair of eyes CS-03|audit-set|backup restore is tested CS-04|audit-set|privileged access is reviewed CS-05|audit-set|code change gets a second reviewer CS-06|audit-set|closed within a defined period CS-07|audit-set|system clocks are synchronized SR-01|sector|payment data is processed in a separate segment SR-02|sector|personal data item is classified SR-03|sector|data retention period is enforced SR-04|sector|data subject request is traceable SR-05|sector|transfer to a third party is recorded `).map(([code, source, text]) => ({ code, source, text })); // code | site where the practice runs | trace it leaves | name of the practice export const PRACTICES = Object.fromEntries(split(` P01|pipeline|machine|dependency component analysis P02|pipeline|machine|static analysis P03|pipeline|machine|build output signing P04|pipeline|machine|provenance generation P05|pipeline|machine|image scanning P06|repo|machine|second reviewer on merge request P07|repo|machine|branch protection rule P08|runtime|machine|short-lived credential from secret store P09|runtime|machine|key rotation job P10|runtime|machine|network segmentation rule P11|runtime|machine|admission controller P12|runtime|machine|application audit log P13|runtime|machine|alert rule set P14|manual|document|backup restore drill P15|manual|document|access review meeting P16|manual|document|incident response drill P17|manual|document|data classification exercise P18|manual|document|management review meeting `).map(([code, site, trace, name]) => [code, { code, site, trace, name }])); // control | practices that actually carry it out; empty cell: no engineering equivalent export const MAPPING = Object.fromEntries(split(` FF-01|P01,P04 FF-02|P08,P15 FF-03| FF-04|P12 FF-05|P13 FF-06|P16 FF-07|P14 FF-08|P01,P05 MS-01| MS-02| MS-03| MS-04| MS-05|P18 MS-06| CS-01|P06,P07 CS-02|P08 CS-03|P14 CS-04|P15 CS-05|P06,P07 CS-06|P01,P02,P05 CS-07| SR-01|P10,P11 SR-02|P17 SR-03| SR-04|P12 SR-05| `).map(([c, p]) => [c, p ? p.split(",") : []])); export const SOURCES = ["function-framework", "management-system", "audit-set", "sector"]; if (import.meta.url === `file://${process.argv[1]}`) { const practices = Object.values(PRACTICES); const links = Object.values(MAPPING).reduce((t, p) => t + p.length, 0); console.log(`model: ${CONTROLS.length} controls, ${practices.length} practices, ${links} links`); const y = gen([20, 10]); console.log("\n" + y("compliance source", "controls")); for (const u of SOURCES) console.log(y(u, CONTROLS.filter((c) => c.source === u).length)); const z = gen([20, 10, 16]); console.log("\n" + z("practice site", "practices", "trace left")); for (const site of ["pipeline", "repo", "runtime", "manual"]) { const p = practices.filter((x) => x.site === site); console.log(z(site, p.length, p[0].trace)); } } ``` ``` model: 26 controls, 18 practices, 25 links compliance source controls function-framework 8 management-system 6 audit-set 7 sector 5 practice site practices trace left pipeline 5 machine repo 2 machine runtime 6 machine manual 5 document ``` The model consists of two lists and the 25 links between them. The control side is evenly distributed: the largest source contributes eight items, the smallest five. The distribution on the practice side is not even, and the real information is there: of the eighteen practices, **thirteen** run in the pipeline, in the repo, or at runtime and leave a machine-readable trace; **five** run as a meeting or a drill and leave only a document behind. These two kinds of trace look equal in the matrix; in an audit they are not. #### Running the Matrix An empty cell in the mapping table is the unit of measure for this lesson: if a control has no practice opposite it, that item falls on none of the team's work. The second number is its reverse — how many work items an item falls on. The matrix can be read in two directions, and the two readings answer two separate questions. Read from control to practice, the question is **coverage**: is the item being carried out, and if so by how many work items. Read from practice to control, the question is **leverage**: how many items does this one piece of work satisfy at once. Both readings run over the same 25 links and bring different entries to the front; that is why both are written down. There is also a third reading, and it is mostly skipped: the practice that falls on no control. That column counts the work the control set does not see. ```js // compliance/matrix.mjs — the control-practice matrix is run and controls with no equivalent are counted. import { CONTROLS, PRACTICES, MAPPING, SOURCES, gen } from "./set.mjs"; const y = gen([18, 9, 15, 15, 18]); console.log(y("compliance source", "controls", "has equivalent", "no equivalent", "practices/control")); const rate = (k) => { const v = k.filter((c) => MAPPING[c.code].length > 0); const b = v.reduce((t, c) => t + MAPPING[c.code].length, 0); return [v.length, k.length - v.length, v.length ? (b / v.length).toFixed(2) : "-"]; }; for (const u of SOURCES) { const k = CONTROLS.filter((c) => c.source === u); console.log(y(u, k.length, ...rate(k))); } console.log(y("TOTAL", CONTROLS.length, ...rate(CONTROLS))); const none = CONTROLS.filter((c) => MAPPING[c.code].length === 0); console.log(`\n${none.length} controls with no engineering equivalent:`); for (const c of none) console.log(` ${c.code} ${c.text}`); const controlCount = (p) => Object.values(MAPPING).filter((l) => l.includes(p)).length; const ranked = Object.values(PRACTICES).map((p) => ({ ...p, n: controlCount(p.code) })) .sort((a, b) => b.n - a.n); const single = CONTROLS.filter((c) => MAPPING[c.code].length === 1).length; const heavy = CONTROLS.filter((c) => MAPPING[c.code].length > 2).map((c) => c.code); console.log(`\n${ranked.filter((p) => p.n > 1).length} practices fall on more than one control: ` + ranked.filter((p) => p.n > 1).map((p) => `${p.code} (${p.n})`).join(", ")); console.log(`controls resting on a single practice: ${single}; falling on more than two practices: ${heavy.join(", ")}`); console.log(`${ranked.filter((p) => p.n === 0).length} practices fall on no control: ` + ranked.filter((p) => p.n === 0).map((p) => `${p.code} ${p.name}`).join(", ")); ``` ``` compliance source controls has equivalent no equivalent practices/control function-framework 8 7 1 1.43 management-system 6 1 5 1.00 audit-set 7 6 1 1.67 sector 5 3 2 1.33 TOTAL 26 17 9 1.47 9 controls with no engineering equivalent: FF-03 data in transit and at rest is encrypted MS-01 scope statement is updated MS-02 applicability statement is justified MS-03 internal audit runs periodically MS-04 corrective action is closed MS-06 staff competence is recorded CS-07 system clocks are synchronized SR-03 data retention period is enforced SR-05 transfer to a third party is recorded 8 practices fall on more than one control: P01 (3), P05 (2), P06 (2), P07 (2), P08 (2), P12 (2), P14 (2), P15 (2) controls resting on a single practice: 10; falling on more than two practices: CS-06 2 practices fall on no control: P03 build output signing, P09 key rotation job ``` Of the twenty-six controls, **seventeen** have an equivalent and **nine** do not. The number nine is not a defect on its own; whether it is a defect is understood by looking at **where they gather**. Five of them are gathered in a single compliance source, the certifiable management system standard — the scope statement, the applicability statement, the internal audit, the corrective action, and staff competence. None of these items is the work of an application team, and their absence is not counted as a deficiency; their equivalent lies not in the team but in the organization's management layer. The matrix does not say this; it only shows an empty cell. **Reading an empty cell requires writing down separately whose job it is.** The remaining four are of a completely different class. FF-03, CS-07, SR-03, and SR-05 belong directly to the place where this application runs: data encryption, clock synchronization, the retention period, and transfer to a third party. All four are the team's responsibility, and for none of the four is there a practice running. CS-07 is the sneakiest of the four: if system clocks are not synchronized, all the records the previous topic produced cannot be ordered against one another — that is, the absence of an equivalent for one control also spoils the evidence of other controls that do have one. The number of practices per control is **1.47**, and ten controls rest on a single practice. A control that rests on a single practice is left without an equivalent the moment that practice stops; the matrix shows this only when it is run again. Read in the opposite direction, eight practices satisfy more than one control — dependency component analysis alone is enough for three items. The last line is the least discussed box of the matrix: two practices fall on no control. Build output signing and key rotation run in the team, spend resources, and have no equivalent in this control set. **This does not show that the practice is unnecessary; it shows that the control set does not ask for it** — a control set is not a list of engineering priorities. #### What Produces the Answer **AC6.** The answer to a control can come from three sources: by **query** from a machine-readable record, by **manual compilation** from a document collected at the end of the period, or by an **individual declaration** that rests on no record. The source has a second property: a query covers the whole period, while a document covers only the day it was produced. ```js // compliance/trace.mjs — the trace a practice leaves is examined: what answers the control's question. import { CONTROLS, PRACTICES, MAPPING, SOURCES, gen } from "./set.mjs"; // Three groups: if the practice leaves a machine-readable trace it is "machine-readable"; if it leaves // only a meeting or drill document it is "document-only"; if it has no practice at all it is "no trace". const group = (c) => { const p = MAPPING[c.code].map((x) => PRACTICES[x]); return !p.length ? "no trace" : p.some((x) => x.trace === "machine") ? "machine-readable" : "document-only"; }; const GROUPS = ["machine-readable", "document-only", "no trace"]; const y = gen([18, 18, 14, 9]); console.log(y("compliance source", ...GROUPS)); for (const u of SOURCES) { const k = CONTROLS.filter((c) => c.source === u); console.log(y(u, ...GROUPS.map((o) => k.filter((c) => group(c) === o).length))); } console.log(y("TOTAL", ...GROUPS.map((o) => CONTROLS.filter((c) => group(c) === o).length))); // Coverage of the answer: a machine trace is written throughout the period, a document belongs only to the day it was produced. const COVERAGE = { "machine-readable": "whole period", "document-only": "single day", "no trace": "none" }; const SOURCE = { "machine-readable": "query", "document-only": "manual compilation", "no trace": "self-declaration" }; const z = gen([16, 10, 18, 20]); console.log("\n" + z("group", "controls", "answer coverage", "answer source")); for (const o of GROUPS) { const k = CONTROLS.filter((c) => group(c) === o); console.log(z(o, k.length, COVERAGE[o], SOURCE[o])); } const doc = CONTROLS.filter((c) => group(c) === "document-only"); console.log(`\ndocument-only group: ${doc.map((c) => c.code).join(", ")}`); const machine = CONTROLS.filter((c) => group(c) === "machine-readable").length; console.log(`Of the 26 audit questions, ${machine} are answered by querying the record, ` + `${doc.length} by manual compilation at period end; ` + `${CONTROLS.length - machine - doc.length} are not answered at all`); ``` ``` compliance source machine-readable document-only no trace function-framework 5 2 1 management-system 0 1 5 audit-set 4 2 1 sector 2 1 2 TOTAL 11 6 9 group controls answer coverage answer source machine-readable 11 whole period query document-only 6 single day manual compilation no trace 9 none self-declaration document-only group: FF-06, FF-07, MS-05, CS-03, CS-04, SR-02 Of the 26 audit questions, 11 are answered by querying the record, 6 by manual compilation at period end; 9 are not answered at all ``` The fill rate of the matrix was 65%; looking at the source of the answer, the number falls to **11/26**, that is 42%. The six controls in between look filled in the matrix, are met with a document in the audit, and that document describes **a single day**. The backup restore drill ran on one day, the access review meeting was held on one day, the data classification exercise was finished once. The control's question, however, is about the whole period: was privileged access reviewed **throughout the period**. A one-day document does not fully answer this question; it turns it into a sampling. The three groups correspond to three separate kinds of work. For the nine controls with no trace, the work to be done is **setting up a practice** — four in the team's own area, five in another layer. For the six document-only controls, the work to be done is not setting up a practice; the practice is already running, and the work is **extracting a machine-readable trace from it**. For the eleven machine-readable controls, the work is only to compile the trace in the form the audit asks for. These three kinds of work carry very different costs, and a table that summarizes a compliance effort as "65% ready" writes all three into the same cell. #### Summary - Between twenty-six audit items and the eighteen practices of an application team there are 25 links; 17 controls have an engineering equivalent, 9 have none. - Five of the nine controls with no equivalent are gathered in a single compliance source and their equivalent lies not in the team but in the organization's management layer; the remaining four are directly in the team's area. Reading an empty cell requires writing down separately whose job it is. - A control with an equivalent falls on 1.47 practices on average and 10 controls rest on a single practice; eight practices satisfy more than one control, and the one with the most leverage is enough for three items. - Two practices fall on no control. This shows not that the practice is unnecessary, but that the control set is not a list of engineering priorities. - Looking at the source of the answer, the rate falls from 65% to 42%: 11 controls are answered by querying the record, 6 by a document describing a single day, and 9 are not answered at all. #### Next Step The matrix ran and three separate kinds of work were separated. Two of them stay outside this lesson: setting up practices for the controls with no trace is the job of an engineering plan, and those whose equivalent is in another layer are not even on the team's list. What remains is what can be measured — the **six document-backed controls** and the **eleven controls** that already leave a machine trace. The next lesson takes these seventeen items up again along the axis of evidence production and follows a single number: how far the number of items collected by hand at period end falls. Does generating evidence in the pipeline end manual compilation, which item never becomes automatic under any condition, and how many times the volume, how many personal data items, and which retention constraint does automatically produced evidence bring with it? ### Evidence Automation Source: https://academia.sh/en/courses/application-detection-and-compliance/evidence-automation Twenty-four audit evidence items are moved into the pipeline: the items touched by hand fall from 24 to 9, the annual compilation load falls from 213 hours to 96.2 hours, the 170-hour setup pays back in 1.46 years, and six items never become automatic under any condition. The cost appears in the opposite column: evidence records per period rise from 24 to 72,401, a factor of 3,017, the sampling rate falls from 100% to 0.63%, and the six items carrying a personal data type answer only 24.7% of the period because of a 90-day retention window. The previous lesson matched twenty-six audit items with the team's eighteen practices and separated three kinds of work. Two of them could not be measured: nine controls have no practice opposite them, and the equivalent of five of those lies in another layer. What remained measurable was seventeen controls — eleven leave a machine-readable trace, six leave only a document. This lesson follows the evidence of those seventeen controls. The question is not whether the practice exists, but **where the evidence is generated.** A practice running and the fact that it is running being provable are two separate jobs, and the second is mostly done at the end of the period, by hand. The unit of measure comes from here too: **the number of items touched by hand** and the annual compilation load of those items. #### The Evidence Item Set **AC7.** The evidence item set has 24 items and is tied to the 17 controls with an equivalent from the previous lesson; the set belongs to the fictional regional measurement network and is a **model**. **AC8.** Every item has a **kind**: a **trace** is a record that arises on its own in a system, a **decision** is a person's justification, and a **meeting** is the minutes of a drill or a meeting. **AC9.** Manually compiled evidence is produced on the day of compilation, and the interval it covers is not written on the record; evidence generated in the pipeline carries a period stamp — this distinction is an assumption of the model. **AC10.** The minutes are given per period **as data**, not measured; compilation is done three times a year, and this too is fiction. ```js // evidence/item.mjs — audit evidence items are a MODEL: fictional regional measurement network, one period. // The items tie to the 17 controls that had an equivalent in the previous lesson; the numbers are fiction. export const gen = (W) => (...h) => h.map((v, j) => (j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))).join(""); // code|control|kind|site|manual minutes|records produced in the period|personal data export const ITEMS = ` K01|FF-01|trace|pipeline|120|312|no|dependency inventory K02|FF-01|trace|pipeline|90|312|no|provenance information K03|FF-02|trace|runtime|150|8640|yes|short-lived credential issuance K04|FF-02|decision|runtime|240|24|yes|access review decision K05|FF-04|trace|runtime|180|54000|yes|application audit log K06|FF-05|trace|runtime|100|960|no|alert rule firing K07|FF-06|meeting|manual|300|2|no|incident response drill report K08|FF-06|decision|manual|120|6|no|role assignment decision K09|FF-07|meeting|manual|360|2|no|recovery objective measurement report K10|FF-08|trace|pipeline|110|312|no|image scan report K11|MS-05|meeting|manual|180|4|no|management review minutes K12|CS-01|trace|repo|130|486|no|change approval record K13|CS-01|decision|repo|90|18|no|emergency change justification K14|CS-02|trace|runtime|140|420|yes|production access issuance K15|CS-03|meeting|manual|300|2|no|restore drill report K16|CS-04|decision|runtime|280|24|yes|privileged access review K17|CS-05|trace|repo|120|486|no|code review approval K18|CS-05|trace|repo|80|486|no|branch protection rule status K19|CS-06|trace|pipeline|160|312|no|open scan result K20|CS-06|trace|pipeline|140|312|no|static analysis result K21|SR-01|trace|runtime|150|1440|no|segment rule status K22|SR-01|trace|runtime|130|1440|no|admission controller rejection record K23|SR-02|decision|manual|420|1|yes|data classification table K24|SR-04|trace|runtime|170|2400|yes|data subject request trace `.trim().split("\n").map((l) => { const [code, control, kind, site, min, records, personal, name] = l.trim().split("|"); return { code, control, kind, site, min: +min, records: +records, personal, name }; }); export const KINDS = ["trace", "decision", "meeting"]; export const COMPILATIONS = 3; // three evidence compilations a year (fiction) export const SETUP = { pipeline: 6, repo: 8, runtime: 12, manual: 0 }; // person-hours per item if (import.meta.url === `file://${process.argv[1]}`) { const y = gen([12, 8, 10, 12, 16, 16]); console.log(y("item kind", "items", "in system", "manual", "period minutes", "period records")); for (const t of KINDS) { const k = ITEMS.filter((x) => x.kind === t); console.log(y(t, k.length, k.filter((x) => x.site !== "manual").length, k.filter((x) => x.site === "manual").length, k.reduce((a, x) => a + x.min, 0), k.reduce((a, x) => a + x.records, 0))); } const min = ITEMS.reduce((a, x) => a + x.min, 0); console.log(y("TOTAL", ITEMS.length, ITEMS.filter((x) => x.site !== "manual").length, ITEMS.filter((x) => x.site === "manual").length, min, ITEMS.reduce((a, x) => a + x.records, 0))); console.log(`\ninitial state: ${ITEMS.length} of ${ITEMS.length} items are compiled by hand`); console.log(`per compilation ${(min / 60).toFixed(1)} hours, ${COMPILATIONS} compilations a year -> ` + `${(COMPILATIONS * min / 60).toFixed(1)} hours`); console.log(`items carrying a personal data type ` + `${ITEMS.filter((x) => x.personal === "yes").length}/${ITEMS.length}`); } ``` ``` item kind items in system manual period minutes period records trace 15 15 0 1970 72318 decision 5 3 2 1150 73 meeting 4 0 4 1140 10 TOTAL 24 18 6 4260 72401 initial state: 24 of 24 items are compiled by hand per compilation 71.0 hours, 3 compilations a year -> 213.0 hours items carrying a personal data type 7/24 ``` The second and third columns of the table set up the real tension of this lesson. Of the twenty-four items, **eighteen** already arise in a system — in the pipeline, in the repo, or at runtime. Even so, in the initial state the evidence of **all twenty-four** is compiled by hand. The record exists in the system, and at the end of the period a person exports it, puts it in a directory, and enters it into a table. The difference is not an engineering gap but a connection gap: a record existing does not mean it stands in the form the audit asks for. That there are more items than controls is also information: seventeen controls ask for twenty-four items. Some controls are not met with a single item — showing that a code change passed a second pair of eyes requires both the review approval and the status of the branch protection rule, because the first says the event took place and the second says its not taking place was prevented. The number of evidence items is determined not by the number of controls but by **how many parts the control's question consists of**. The third row also gives the limit of this. None of the four meeting items arises in a system, and this comes not from a lack of tooling but from the **nature** of the item: the result of a drill is written by a person. The minutes column, meanwhile, says the distribution is unbalanced — the four meeting items take 1,140 minutes, more than a quarter of the total load, yet exactly one sixth of the item count. #### Moving Generation into the Pipeline **AC11.** For a decision-kind item, the **record** becomes automatic — that the decision was made, the person who made it, and the timestamp are read from a system; the **justification text** stays manual. The remaining manual work was counted as 0.4 times the item's minutes, and this ratio is an assumption of the model. **AC12.** The setup cost is given in person-hours per item (pipeline 6, repo 8, runtime 12) and is **data**; the maintenance cost was not modeled, which makes the payback period look shorter than it is. ```js // evidence/automation.mjs — evidence generation is moved into the pipeline: how far the items collected by hand fall. import { ITEMS, COMPILATIONS, SETUP, gen } from "./item.mjs"; // A trace item arises in a system on its own: it is generated in the pipeline. The record of a decision // becomes automatic, its justification stays manual (0.4 times the item's minutes). A meeting does not become automatic. const status = (k) => k.site === "manual" ? "stays manual" : k.kind === "trace" ? "generated in pipeline" : k.kind === "decision" ? "semi-automatic" : "stays manual"; const remaining = (k) => status(k) === "generated in pipeline" ? 0 : status(k) === "semi-automatic" ? Math.round(0.4 * k.min) : k.min; const STATUSES = ["generated in pipeline", "semi-automatic", "stays manual"]; const y = gen([24, 8, 22, 16]); console.log(y("automation result", "items", "manual minutes left", "setup hours")); for (const d of STATUSES) { const k = ITEMS.filter((x) => status(x) === d); console.log(y(d, k.length, k.reduce((a, x) => a + remaining(x), 0), k.reduce((a, x) => a + SETUP[x.site], 0))); } const remainingMin = ITEMS.reduce((a, x) => a + remaining(x), 0); const setup = ITEMS.reduce((a, x) => a + (status(x) === "stays manual" ? 0 : SETUP[x.site]), 0); console.log(y("TOTAL", ITEMS.length, remainingMin, setup)); const before = COMPILATIONS * ITEMS.reduce((a, x) => a + x.min, 0) / 60; const after = COMPILATIONS * remainingMin / 60; const manualItems = ITEMS.filter((x) => status(x) !== "generated in pipeline"); console.log(`\nitems touched by hand ${ITEMS.length} -> ${manualItems.length} ` + `(${ITEMS.filter((x) => status(x) === "stays manual").length} full, ` + `${ITEMS.filter((x) => status(x) === "semi-automatic").length} partial)`); console.log(`annual compilation load ${before.toFixed(1)} -> ${after.toFixed(1)} hours; ` + `gain ${(before - after).toFixed(1)} hours/year`); console.log(`setup ${setup} hours -> payback ${(setup / (before - after)).toFixed(2)} years`); console.log(`items that never become automatic: ` + ITEMS.filter((x) => status(x) === "stays manual").map((x) => `${x.code} ${x.kind}`).join(", ")); ``` ``` automation result items manual minutes left setup hours generated in pipeline 15 0 138 semi-automatic 3 244 32 stays manual 6 1680 0 TOTAL 24 1924 170 items touched by hand 24 -> 9 (6 full, 3 partial) annual compilation load 213.0 -> 96.2 hours; gain 116.8 hours/year setup 170 hours -> payback 1.46 years items that never become automatic: K07 meeting, K08 decision, K09 meeting, K11 meeting, K15 meeting, K23 decision ``` The items touched by hand fall **from 24 to 9**: six fully manual, three partial. The annual compilation load falls from 213 hours to 96.2 hours and the gain is 116.8 hours a year. The setup takes 170 hours, so the investment pays back **in 1.46 years**. These three numbers must be read together: the sentence "automation cut the load by more than half" on its own is true, but the net balance of the first year is negative. The last line gives the ceiling of automation. Six items never become automatic under any condition, two of them of the decision kind and four of the meeting kind. These six items carry **1,680 minutes** of the remaining manual load, that is 87% of the 1,924 minutes. Automation took not most of the load but most of the **item count**; the remaining nine items are few in number and heavy in minutes. Where the next improvement should look is written in this column, and it is not a pipeline but the arrangement of a meeting. The three partially automated items are a separate class and the one most often misunderstood. An access review decision is marked in a system, but **why** that access is maintained is a sentence, and a pipeline cannot write that sentence. A record becoming automatic does not mean the decision becomes automatic; what becomes automatic is the proof that the decision **exists**, not its content. #### The Cost of Automatic Evidence **AC13.** The auditor inspects at most 25 records per item; the audit period is 365 days, and the retention window of an evidence item carrying a personal data type is 90 days. All three are **fictional data**, and the window decision comes from the data type the item carries. ```js // evidence/cost.mjs — the cost of automatic evidence: volume, sampling rate, and retention window. import { ITEMS, gen } from "./item.mjs"; const SAMPLE = 25, PERIOD_DAYS = 365, RETENTION_DAYS = 90; // all three are fictional data const automatic = (k) => k.site !== "manual" && k.kind !== "meeting"; const records = ITEMS.reduce((a, k) => a + k.records, 0); const inspected = ITEMS.reduce((a, k) => a + Math.min(k.records, SAMPLE), 0); const y = gen([26, 20, 22]); console.log(y("evidence volume", "before compilation", "pipeline-generated")); console.log(y("evidence records/period", ITEMS.length, records)); console.log(y("inspected by auditor", ITEMS.length, inspected)); console.log(y("sampling rate", "100%", `${(100 * inspected / records).toFixed(2)}%`)); // An evidence item carrying a personal data type is subject to the retention window; when the window is // shorter than the period, the item answers only a part of the period. const personal = ITEMS.filter((k) => automatic(k) && k.personal === "yes"); const full = ITEMS.filter((k) => automatic(k) && k.personal === "no"); const manual = ITEMS.filter((k) => !automatic(k)); const share = (100 * RETENTION_DAYS / PERIOD_DAYS).toFixed(1); const z = gen([32, 10, 16]); console.log("\n" + z("interval covered by evidence", "items", "period share")); console.log(z(`whole period (${PERIOD_DAYS} days)`, full.length, "100%")); console.log(z(`last ${RETENTION_DAYS} days (retention limit)`, personal.length, `${share}%`)); console.log(z("single day (manual compilation)", manual.length, "-")); console.log(`\nautomatic items carrying a personal data type: ` + personal.map((k) => k.code).join(", ")); console.log(`Of ${ITEMS.length} evidence items, ${full.length} answer the whole period, ` + `${personal.length} the last ${RETENTION_DAYS} days, and ${manual.length} a single day`); console.log(`evidence records ${ITEMS.length} -> ${records} (${Math.round(records / ITEMS.length)} times); ` + `inspected records ${inspected}`); ``` ``` evidence volume before compilation pipeline-generated evidence records/period 24 72401 inspected by auditor 24 458 sampling rate 100% 0.63% interval covered by evidence items period share whole period (365 days) 12 100% last 90 days (retention limit) 6 24.7% single day (manual compilation) 6 - automatic items carrying a personal data type: K03, K04, K05, K14, K16, K24 Of 24 evidence items, 12 answer the whole period, 6 the last 90 days, and 6 a single day evidence records 24 -> 72401 (3017 times); inspected records 458 ``` At first glance the first table reads against automation: evidence records rise from 24 to 72,401, a factor of **3,017**, and the sampling rate falls from 100% to 0.63%. This reading is wrong, and the reason it is wrong lies in the measure of the course. The 100% before compilation says all twenty-four documents were read; it does not say **which interval** those twenty-four documents describe. In the later state the 458 records inspected are a sample, but all 72,401 records behind them can be queried — the auditor's next question requires changing the query, not enlarging the sample. **More records mean less inspection; they do not have to mean fewer answers.** The condition of this conditional sentence shows in the second table. The six items carrying a personal data type — credential issuance, audit log, production access, data subject request, and two review decisions — are subject to a 90-day retention window. The audit period is 365 days. These six items answer **24.7%** of the period; every question about the first nine months of the period is unanswered, and that it is unanswered is understood only when the question is asked. Automation did not leave these items without evidence; **it made the lifetime of the evidence a constraint separate from evidence generation.** The queryability condition is not free either. For an evidence record generated in the pipeline to be useful in an audit, it must carry three fields: which control it belongs to, which moment it belongs to, and that it can be shown not to have been changed after it was generated. These three fields are inside the setup hours, and all three can be missing. A pile of automatic evidence with missing fields answers less than twenty-four manually compiled documents — its count is large, it cannot be queried, and it does not say which period it describes. The gain from automation comes not from volume but from **these three fields**. Read together, the three rows give this table: of the twenty-four items, **twelve** answer the whole period, **six** the last ninety days, and **six** a single day. Before automation these three classes could not be told apart, because all of them stood in the same form — as a document compiled at the end of the period. The first gain of measuring is not the number itself but **that the three classes have been separated.** #### Summary - Eighteen of the twenty-four evidence items already arose in a system, yet the evidence of all twenty-four was compiled by hand at the end of the period; the gap is not engineering but connection. - When generation is moved into the pipeline, the items touched by hand fall from 24 to 9 and the annual compilation load from 213 hours to 96.2 hours; the 170-hour setup pays back in 1.46 years and the balance of the first year is negative. - Six items never become automatic and carry 87% of the remaining manual load: automation took not most of the load but most of the item count. - The record of a decision becomes automatic and its justification stays manual; what becomes automatic is the proof that the decision exists, not its content. - The cost appears in the opposite column: evidence records rise by a factor of 3,017, the sampling rate falls to 0.63%, and the six items carrying a personal data type answer 24.7% of the period because of the 90-day retention window. #### Next Step Evidence generation moved into the pipeline and three classes were separated: items answering the whole period, the last ninety days, and a single day. All of these measurements were about **whether the evidence exists**. A part of the audit items asks not for existence but for a **number**: was the finding closed within its target time, and how long have those not closed been open. The next lesson computes these two numbers at the scale of an application team — mean time to remediate and open finding age. Both come from the same set of findings, go in opposite directions in the same period, and the ability to compute either depends on certain timestamps being present in the record. If a timestamp is missing, the item never enters the calculation; are the items that do not enter chosen at random, or do they always come from the same class? ### Application Security Metrics Source: https://academia.sh/en/courses/application-detection-and-compliance/application-security-metrics Of the 142 findings, the 121 that closed have a mean time to remediate of 17.0 days and a median of 6 days: the 30 items above the mean carry 75% of the total days. The same set gives opposite directions under two grouping rules — counted in the half it opened it falls from 23.6 to 7.6, counted in the half it closed it rises from 13.2 to 20.0 — and while the open queue falls from 24 items to 21, its mean age rises from 30.4 days to 59.8 days. Of eight questions, five are answered fully, one partially, and two are unanswered. Three interventions that improve the metric fix zero items: the mean goes to 12.6 days, the open finding age to 23.4 days, and the within-target rate to 94.2%. The previous lesson measured where evidence is produced and separated the items into those that answer the whole period, the last ninety days, or a single day; what was measured was the **existence** of evidence. Some of the audit items, however, ask not for existence but for a **number**: was the finding closed within its target time, and how long has the item that was not closed been open. This lesson calculates those two numbers — **mean time to remediate** and **open finding age** — at the scale of one application team. The Governance, Risk and Compliance course covered the same metrics at the program level; that queue is the input here. The question asked is narrow: what the metrics **say**, what they **do not say**, and how many questions what they do not say leaves unanswered. #### Finding Set and Question Set **AC14.** The finding set is 142 items, comes from five sources, belongs to a 180-day period, and is a **model** of the subscriber portal team of the fictional regional measurement network; the generator is in this block, and the seed is visible. **AC15.** An item can have three timestamps: **opened** exists on all of them, **verification** is never written in some sources, and **closure** exists only on an item that closed. **AC16.** The target durations of the severity classes are **data** (7, 21, 45, and 90 days) and are fictional. **AC17.** Midway through the period the priority rule changed: items from the pipeline were moved up, items from outside the pipeline were held back — this is an assumption of the model. ```js // metrics/finding.mjs — the finding set is a MODEL: fictional regional measurement network, a single 180-day period. // The generator is written here and the seed is visible; the set comes out the same on every run. export const gen = (B) => (...h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); export const PERIOD = 180, SEED = 20250, PIPELINE_SOURCES = ["dependency", "static", "image"]; let s = SEED; const rnd = () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; // source | findings | base days | triage days | probability the verification stamp is missing const SOURCES = [["dependency", 44, 16, 1, 0], ["static", 34, 26, 2, 0], ["image", 24, 22, 2, 0], ["audit", 16, 52, 7, .5], ["report", 24, 70, 11, .7]]; export const TARGET = { critical: 7, high: 21, medium: 45, low: 90 }; const FACTOR = { critical: .4, high: .7, medium: 1, low: 1.4 }; const ORDER = { pipeline: "medium low medium high low medium critical high".split(" "), external: "high medium critical high low medium high critical".split(" ") }; export const FINDINGS = []; for (const [source, total, base, triage, missing] of SOURCES) { for (let i = 0; i < total; i++) { const inPipeline = PIPELINE_SOURCES.includes(source); const severity = ORDER[inPipeline ? "pipeline" : "external"][Math.floor(rnd() * 8)]; const openedAt = 1 + Math.floor(rnd() * PERIOD); // The duration distribution is skewed: most items take a short time, a few take very long. let duration = base * FACTOR[severity] * (0.15 + 4 * Math.pow(rnd(), 3.5)); // The priority rule changed midway through the period: pipeline items moved up, the others held back. duration *= inPipeline ? (openedAt > 90 ? .45 : 1) : (openedAt > 90 ? 3.2 : 1.8); const kind = inPipeline && rnd() < .12 ? "invalid" : "fixed"; duration = Math.max(1, Math.round(kind === "invalid" ? duration * .15 : duration)); const closedAt = openedAt + duration <= PERIOD ? openedAt + duration : null; const d = rnd() < missing ? null : openedAt + Math.round(triage * (.4 + 1.2 * rnd())); FINDINGS.push({ source, severity, target: TARGET[severity], openedAt, closedAt, kind, verifiedAt: d === null ? null : Math.min(d, closedAt ?? PERIOD) }); } } export const closed = FINDINGS.filter((f) => f.closedAt !== null); export const unstamped = FINDINGS.filter((f) => f.verifiedAt === null); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model: ${FINDINGS.length} findings, ${SOURCES.length} sources, ${PERIOD} days, seed ${SEED}`); console.log(`${closed.length} items closed, ${FINDINGS.length - closed.length} items still open`); console.log(`verification stamp missing on ${unstamped.length} items: ` + SOURCES.map(([k]) => `${k} ${FINDINGS.filter((f) => f.source === k && !f.verifiedAt).length}/` + `${FINDINGS.filter((f) => f.source === k).length}`).join(", ")); // Questions to be asked at the end of the period, the record field each one needs, and the number of items carrying that field. const K = closed.length, D = FINDINGS.length - unstamped.length; const QUESTIONS = [["mean time to remediate", "opened+closed", K], ["within-target rate", "opened+closed", K], ["age of an open item", "opened", FINDINGS.length], ["how much of the duration was triage", "verification", D], ["how many of the closed were fixed", "closure kind", K], ["how many times an item was reopened", "state transition", 0], ["which change made the fix", "change id", 0], ["why the metric changed", "opened+source", K]]; const answer = (t) => t === 0 ? "none" : t >= K ? "full" : "partial"; const z = gen([36, 20, 10, 8]); console.log("\n" + z("question", "record field needed", "carrying", "answer")); for (const [m, d, t] of QUESTIONS) console.log(z(m, d, t, answer(t))); const n = (c) => QUESTIONS.filter(([, , t]) => answer(t) === c).length; console.log(`\nof ${QUESTIONS.length} questions, ${n("full")} are answered fully, ${n("partial")} partially, ` + `${n("none")} are unanswered`); } ``` ``` model: 142 findings, 5 sources, 180 days, seed 20250 121 items closed, 21 items still open verification stamp missing on 25 items: dependency 0/44, static 0/34, image 0/24, audit 10/16, report 15/24 question record field needed carrying answer mean time to remediate opened+closed 121 full within-target rate opened+closed 121 full age of an open item opened 142 full how much of the duration was triage verification 117 partial how many of the closed were fixed closure kind 121 full how many times an item was reopened state transition 0 none which change made the fix change id 0 none why the metric changed opened+source 121 full of 8 questions, 5 are answered fully, 1 partially, 2 are unanswered ``` The third line answers the question the previous lesson left behind. The 25 items missing the verification stamp are not randomly chosen: it is missing on none of the 102 items from the pipeline, on 10 of those from the audit log, and on 15 of those from the external report intake. **The place where the stamp is missing is the same as the place where the duration is longest.** The two unanswered questions are of the same class: **they are the questions that audit the metric.** A reopened item appears in the metric not as one long duration but as two short ones; if the change that closed it is unknown, it cannot be verified that the closure was a fix. The cost of making them answerable — assuming four state transitions per item — is 568 additional records and a link to be built between the finding and the change. #### Two Metrics, One Set **AC18.** Mean time to remediate is calculated only from **closed** items; an open item never enters the metric. Open age, on the other hand, is an **instantaneous** measurement. The base of the two metrics is therefore not the same set, and the two can move in opposite directions in the same period. ```js // metrics/metrics.mjs — two metrics are calculated from the same set: mean time to remediate and open finding age. import { FINDINGS, PERIOD, PIPELINE_SOURCES, closed, gen } from "./finding.mjs"; const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length; const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))]; const duration = (f) => f.closedAt - f.openedAt; const openAt = (t) => FINDINGS.filter((f) => f.openedAt <= t && (!f.closedAt || f.closedAt > t)); const age = (t) => openAt(t).map((f) => t - f.openedAt); const y = gen([26, 8, 10, 9, 8, 10]); console.log(y("metric (days)", "items", "mean", "median", "p90", "max")); for (const [name, a] of [["time to remediate", closed.map(duration)], ["open finding age (day 180)", age(PERIOD)]]) console.log(y(name, a.length, mean(a).toFixed(1), percentile(a, .5), percentile(a, .9), Math.max(...a))); const d = closed.map(duration), m = mean(d), above = d.filter((x) => x > m); console.log(`\n${above.length}/${d.length} items above the mean (${m.toFixed(1)}) carry ` + `${(100 * above.reduce((t, x) => t + x, 0) / d.reduce((t, x) => t + x, 0)).toFixed(0)}% of the total days`); // The same set under two grouping rules, then a snapshot of the open queue at two measurement points. const z = gen([30, 12, 12, 12]); console.log("\n" + z("grouping rule", "days 1-90", "days 91-180", "direction")); for (const [name, f] of [["counted in the half it opened", (x) => x.openedAt], ["counted in the half it closed", (x) => x.closedAt]]) { const a = mean(closed.filter((x) => f(x) <= 90).map(duration)); const c = mean(closed.filter((x) => f(x) > 90).map(duration)); console.log(z(name, a.toFixed(1), c.toFixed(1), c < a ? "falls" : "rises")); } for (const t of [90, PERIOD]) console.log(z(`open queue (day ${t})`, `${age(t).length} items`, `mean ${mean(age(t)).toFixed(1)}`, `median ${percentile(age(t), .5)}`)); const w = gen([16, 9, 18, 8, 22]); console.log("\n" + w("source group", "closed", "mean remediation", "open", "mean open finding age")); for (const [name, f] of [["pipeline", (x) => PIPELINE_SOURCES.includes(x.source)], ["non-pipeline", (x) => !PIPELINE_SOURCES.includes(x.source)]]) { const kk = closed.filter(f), aa = openAt(PERIOD).filter(f); console.log(w(name, kk.length, mean(kk.map(duration)).toFixed(1), aa.length, mean(aa.map((x) => PERIOD - x.openedAt)).toFixed(1))); } const h2 = FINDINGS.filter((x) => x.openedAt > 90 && !x.closedAt); console.log(`\n${h2.length} items opened between days 91 and 180 are still open and never enter the metric; ` + `${h2.filter((x) => !PIPELINE_SOURCES.includes(x.source)).length} of them come from non-pipeline sources. ` + `Removing the ${FINDINGS.filter((x) => x.kind === "invalid").length} items counted as invalid moves the mean ` + `${m.toFixed(1)} -> ${mean(closed.filter((x) => x.kind === "fixed").map(duration)).toFixed(1)} days`); ``` ``` metric (days) items mean median p90 max time to remediate 121 17.0 6 57 116 open finding age (day 180) 21 59.8 37 148 176 30/121 items above the mean (17.0) carry 75% of the total days grouping rule days 1-90 days 91-180 direction counted in the half it opened 23.6 7.6 falls counted in the half it closed 13.2 20.0 rises open queue (day 90) 24 items mean 30.4 median 19 open queue (day 180) 21 items mean 59.8 median 37 source group closed mean remediation open mean open finding age pipeline 97 13.7 5 10.0 non-pipeline 24 30.2 16 75.3 14 items opened between days 91 and 180 are still open and never enter the metric; 9 of them come from non-pipeline sources. Removing the 13 items counted as invalid moves the mean 17.0 -> 18.8 days ``` The first line is the central flaw of the metric: the mean is **17.0** days, the median is **6** days. The mean describes no item; 30 of the 121 items are above the mean and carry **75%** of the total waiting days. On the open side the distribution is also skewed, for a different reason: a long item never closes, so it does not enter the metric — it **accumulates**. The second table is the lesson's harshest number. The same 121 items, two grouping rules, two opposite directions: if an item is counted in the half it opened, the metric **falls** from 23.6 days to 7.6 days; if it is counted in the half it closed, it **rises** from 13.2 days to 20.0 days. Both are calculated correctly. **The answer to whether the metric improved depends not on the data but on the grouping rule.** The bottom two lines describe the queue: while the item count falls from 24 to 21, the mean age rises from 30.4 to 59.8 and the median from 19 to 37 — the queue got smaller, and what is in it got older. The source group table gives the reason: pipeline items close in 13.7 days and leave five open items, non-pipeline items close in 30.2 days and sixteen of them wait at 75.3 days of age. The mean is a mixture of these two groups; **when the mix ratio changes, the mean changes without any group getting faster.** The 14 items opened in the second half never enter the metric; when the 13 invalid items are removed, the mean rises to 18.8 days. #### Ways of Improving the Metric That Break the Measure **AC19.** All three interventions change only the **calculation rule** or a record field; none of them touches a line of code. The acceptance threshold is 90 days and is fictional. **AC20.** Reclassification writes the item into a lower severity class and widens its target duration. ```js // metrics/intervention.mjs — three interventions that improve the metric are run on the same set. import { FINDINGS, PERIOD, TARGET, closed, gen } from "./finding.mjs"; const mean = (a) => a.reduce((t, x) => t + x, 0) / a.length; const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(p * (a.length - 1))]; const duration = (f) => f.closedAt - f.openedAt; const openItems = FINDINGS.filter((f) => !f.closedAt), ages = openItems.map((f) => PERIOD - f.openedAt); // 1) The clock starts at verification, not at opening; an item without a stamp never enters the metric. const stamped = closed.filter((f) => f.verifiedAt !== null); const A = mean(closed.map(duration)), B = mean(stamped.map(duration)); const C = mean(stamped.map((f) => f.closedAt - f.verifiedAt)); // 2) Open items older than ninety days are closed as accepted risk (the threshold is fictional). const accepted = openItems.filter((f) => PERIOD - f.openedAt > 90); const remaining = openItems.filter((f) => PERIOD - f.openedAt <= 90).map((f) => PERIOD - f.openedAt); const D = mean([...closed.map(duration), ...accepted.map((f) => PERIOD - f.openedAt)]); // 3) An item that missed its target is written into a lower severity class; its target duration widens. const LOWER = { critical: "high", high: "medium", medium: "low", low: "low" }; const newTarget = (f) => duration(f) > f.target ? TARGET[LOWER[f.severity]] : f.target; const before = closed.filter((f) => duration(f) <= f.target).length; const after = closed.filter((f) => duration(f) <= newTarget(f)).length; const missing = closed.length - stamped.length, g = (v) => v.toFixed(1); const y = gen([34, 9, 22, 11, 11]); console.log(y("intervention", "items", "metric", "before", "after")); console.log(y("1 clock starts at verification", missing, "mean remediation", g(A), g(C))); console.log(y(" a) unstamped items drop out", missing, "mean remediation", g(A), g(B))); console.log(y(" b) clock is rolled back", stamped.length, "mean remediation", g(B), g(C))); console.log(y("2 aged items closed as accepted", accepted.length, "mean open finding age", g(mean(ages)), g(mean(remaining)))); console.log(y(" same intervention", accepted.length, "mean remediation", g(A), g(D))); console.log(y("3 missed items go one class down", after - before, "within-target rate", `${g(100 * before / closed.length)}%`, `${g(100 * after / closed.length)}%`)); console.log(`\nmedian open finding age ${percentile(ages, .5)} -> ${percentile(remaining, .5)}, oldest open item ` + `${Math.max(...ages)} -> ${Math.max(...remaining)} days`); console.log(`of the ${accepted.length} items closed as accepted, ` + `${accepted.filter((f) => f.severity === "critical").length} are critical severity; ` + `additional items actually fixed across the three interventions: 0`); ``` ``` intervention items metric before after 1 clock starts at verification 16 mean remediation 17.0 12.6 a) unstamped items drop out 16 mean remediation 17.0 14.6 b) clock is rolled back 105 mean remediation 14.6 12.6 2 aged items closed as accepted 7 mean open finding age 59.8 23.4 same intervention 7 mean remediation 17.0 23.3 3 missed items go one class down 15 within-target rate 81.8% 94.2% median open finding age 37 -> 16, oldest open item 176 -> 87 days of the 7 items closed as accepted, 2 are critical severity; additional items actually fixed across the three interventions: 0 ``` The first intervention brings the mean from 17.0 down to 12.6 days; the real information is in how the drop breaks down. Starting the clock at verification gains two days, from 14.6 to 12.6. The big part comes from **the 16 items without a verification stamp leaving the base of the metric**, and those items are from the slowest group. What improved is the narrowing of the measurement's scope. The second intervention sets the two metrics against each other. When the seven items older than ninety days are converted to accepted risk, the open finding age falls from 59.8 to 23.4 days, the median from 37 to 16, and the oldest item from 176 days to 87 days. If the same closures are counted in the remediation time, the mean rises to 23.3 days; if they are not counted, it does not move at all and both metrics seem to have improved at once. Two of the seven items are critical. The third intervention is the cheapest: when the 15 items that missed their target are written into a lower class, the within-target rate rises from 81.8% to 94.2%. The sum of the three interventions is zero additional items actually fixed. The rule follows from this: **a metric that can be improved without touching the thing it measures is not a decision tool but a report line.** Two more numbers are written next to the metric — the number of items that **do not enter** the calculation and the date the calculation rule last changed. #### Summary - Mean time to remediate is 17.0 days and the median is 6 days; the 30 items above the mean carry 75% of the total waiting days. The mean hides the queue. - Open age is calculated from a separate base: 21 items, a mean of 59.8 days, the oldest 176. While the queue falls from 24 items to 21, the mean age rises from 30.4 to 59.8. - The same set gives opposite directions under two grouping rules: it falls from 23.6 to 7.6 or rises from 13.2 to 20.0; a metric whose rule is not written down cannot be read. - Of eight questions, five are answered fully, one partially, and two are unanswered; all 25 items missing the verification stamp come from the two slowest sources. - The three interventions improve the metrics and fix zero items: the mean goes to 12.6 days, the open age to 23.4 days, and the within-target rate to 94.2%; two of the seven items converted to accepted risk are critical. #### Next Step Both metrics were calculated and shown to be breakable. What remains is the item the metrics hide: the queue that does not close. Sixteen open items come from non-pipeline sources, are 75.3 days old on average, and are not tied to a plan. The next lesson takes up this queue as **security debt** and runs the planned closure: in what order and with what capacity the debt items close, how long the **duration** of an accepted risk is, and where the point that will reassess the item when that duration expires is written. ### Security Debt Source: https://academia.sh/en/courses/application-detection-and-compliance/security-debt Of a 34-item debt ledger, 21 are open and 13 are accepted risk; only 5 of the acceptances carry a validity period and only 2 carry a reassessment point. Of six questions, one is answered fully, four partially, and one is unanswered; making the partial answers full requires writing 26 fields. A four-period queue with 48 person-days of capacity closes 11 items of the 209 person-day ledger, and the mean age of the remaining queue rises from 91 days to 360 days. The 8 accepted items whose validity was never written are never asked about again in any period; of the 3 expired acceptances, only one carries a reassessment point. The previous lesson computed two metrics and showed that both could be improved without touching the thing being measured. The cheapest improvement was converting the old items to accepted risk: the open finding age fell from 59.8 days to 23.4 days, and the number of items actually fixed stayed at zero. Those seven items went somewhere, and where they went is the subject of this lesson. Accepted risk and open debt together make up **security debt**. The Software Architecture curriculum measured technical debt; that measurement is not repeated here, and there is a single field that separates security debt from it: **an acceptance has a validity.** Technical debt does not pay itself off, but it does not expire either; an accepted risk whose validity was never written stops being debt and becomes a record. #### The Debt Ledger **AC21.** The debt ledger has 34 items: the 21 open items that remained unclosed in the previous lesson and 13 items that were converted to acceptance earlier. The set is a **model** of the subscriber portal team of the fictional regional measurement network; the generator is written in this block and the seed is visible. **AC22.** Each item has an **effort** (person-days) and a **risk score**; both are fictional data. **AC23.** An accepted item can carry three fields: a named **owner**, a **validity period**, and a **reassessment point**. **AC24.** The distribution of the fields is given **as data**: the first five acceptances have a validity written, the first two a reassessment point, and the first six an owner. ```js // debt/ledger.mjs — the security debt ledger is a MODEL: fictional regional measurement network, planning time. // The generator is written here and the seed is visible; the ledger comes out the same on every run. export const gen = (B) => (...h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); export const SEED = 41100, N_OPEN = 21, N_ACCEPTED = 13; let s = SEED; const rnd = () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; // Of the accepted items, the first five have a validity period, the first two a reassessment // point, and the first six a named owner; the rest have the field empty. This distribution is data. const VALIDITY = [90, 120, 180, 180, 365]; export const LEDGER = []; for (let i = 0; i < N_OPEN + N_ACCEPTED; i++) { const isAccepted = i >= N_OPEN, k = i - N_OPEN; LEDGER.push({ code: `D${String(i + 1).padStart(2, "0")}`, kind: isAccepted ? "accepted" : "open", source: rnd() < (isAccepted ? .75 : .70) ? "non-pipeline" : "pipeline", risk: 4 + Math.floor(rnd() * 21), // fictional risk score between 4 and 24 effort: 1 + Math.floor(rnd() * 13), // person-days age: isAccepted ? 40 + Math.floor(rnd() * 380) : 20 + Math.floor(rnd() * 160), validity: isAccepted && k < VALIDITY.length ? VALIDITY[k] : null, reassess: isAccepted && k < 2, owner: isAccepted ? k < 6 : true, }); } export const openItems = LEDGER.filter((d) => d.kind === "open"); export const accepted = LEDGER.filter((d) => d.kind === "accepted"); export const expired = (d) => d.validity !== null && d.age > d.validity; if (import.meta.url === `file://${process.argv[1]}`) { console.log(`model: ${LEDGER.length} debt items, ${openItems.length} open + ${accepted.length} accepted risk`); const y = gen([16, 8, 12, 12, 14, 14]); console.log("\n" + y("class", "items", "person-days", "risk total", "has validity", "expired")); for (const [name, k] of [["open debt", openItems], ["accepted risk", accepted]]) console.log(y(name, k.length, k.reduce((a, d) => a + d.effort, 0), k.reduce((a, d) => a + d.risk, 0), k.filter((d) => d.validity !== null).length, k.filter(expired).length)); // The questions asked of a debt item and the field each one needs. const QUESTIONS = [["is the item open or accepted", "class", LEDGER.length, LEDGER.length], ["who gave the acceptance", "owner", N_ACCEPTED, accepted.filter((d) => d.owner).length], ["how long is the acceptance valid", "validity", N_ACCEPTED, accepted.filter((d) => d.validity !== null).length], ["has the acceptance expired", "validity+age", N_ACCEPTED, accepted.filter((d) => d.validity !== null).length], ["when is the reassessment", "reassessment point", N_ACCEPTED, accepted.filter((d) => d.reassess).length], ["which period will the item close", "queue order", LEDGER.length, 0]]; const answer = (c, b) => c === 0 ? "none" : c === b ? "full" : "partial"; const z = gen([34, 20, 8, 10, 8]); console.log("\n" + z("question", "field needed", "base", "carrying", "answer")); for (const [q, f, b, c] of QUESTIONS) console.log(z(q, f, b, c, answer(c, b))); const n = (a) => QUESTIONS.filter(([, , b, c]) => answer(c, b) === a).length; const missing = (f) => N_ACCEPTED - accepted.filter(f).length; const toWrite = missing((d) => d.owner) + missing((d) => d.validity !== null) + missing((d) => d.reassess); console.log(`\nof ${QUESTIONS.length} questions, ${n("full")} fully answered, ${n("partial")} partially answered, ` + `${n("none")} unanswered`); console.log(`to make the partial answers full, ${toWrite} fields are written; of the ` + `${accepted.filter(expired).length} items with expired validity, ` + `${accepted.filter((d) => expired(d) && d.reassess).length} carries a reassessment point`); } ``` ``` model: 34 debt items, 21 open + 13 accepted risk class items person-days risk total has validity expired open debt 21 141 274 0 0 accepted risk 13 68 185 5 3 question field needed base carrying answer is the item open or accepted class 34 34 full who gave the acceptance owner 13 6 partial how long is the acceptance valid validity 13 5 partial has the acceptance expired validity+age 13 5 partial when is the reassessment reassessment point 13 2 partial which period will the item close queue order 34 0 none of 6 questions, 1 fully answered, 4 partially answered, 1 unanswered to make the partial answers full, 26 fields are written; of the 3 items with expired validity, 1 carries a reassessment point ``` The first table puts the two classes of the ledger side by side, and the real information is in the last two columns. Of the 13 accepted items, only **five** have a validity period written; for the other eight, the acceptance is a decision with no end. Of the five with a validity written, **three** have already expired at planning time, and because nobody asked, all three still sit in the ledger as accepted. The risk total column gives the scale: the accepted items carry **185** of the ledger's risk total, and the open debt carries 274. Acceptance does not remove the risk; it only moves it out of the column the metrics look at. In the question table, **one** of the six questions is answered fully. Of the remaining five, four concern only the accepted items and one concerns the whole ledger; four are partial and one is not answered at all. Making the partial ones full requires writing **26 fields**: seven owners, eight validity periods, and eleven reassessment points. This is the cheapest cost in the course, and it needs no pipeline setup or extra volume; what has to be written is the decision itself. The last line gives its counterpart: of the three expired acceptances, **only one** carries a point that will bring it up again. #### Running the Queue **AC25.** The capacity is 12 person-days per period, a period is 90 days, and the planning horizon is four periods; all three are **fictional data**. The ordering rule is risk per unit of effort, and an item that does not fit the capacity is left for the next period. **AC26.** Two runs are compared: a ledger where an accepted item never comes back, and a ledger where an expired acceptance, if it has a reassessment point, re-enters the queue. ```js // debt/queue.mjs — the debt is run for four periods as a fixed-capacity queue. import { LEDGER, openItems, accepted, expired, gen } from "./ledger.mjs"; const CAPACITY = 12, PERIODS = 4, PERIOD_DAYS = 90; // person-days, number of periods, period length // The ordering is risk per unit of effort; an item that does not fit the capacity is left for the next period. // With comeBack on: an expired acceptance that has a reassessment point enters the queue. const run = (comeBack) => { const queue = openItems.map((d) => ({ ...d })), pending = accepted.map((d) => ({ ...d })); const rows = []; for (let t = 1; t <= PERIODS; t++) { let back = 0; if (comeBack) for (const d of pending) if (!d.requeued && d.reassess && expired(d)) { d.requeued = true; queue.push(d); back++; } queue.sort((a, b) => b.risk / b.effort - a.risk / a.effort); let left = CAPACITY; for (const d of queue) if (!d.closed && d.effort <= left) { d.closed = true; left -= d.effort; } const stillOpen = queue.filter((d) => !d.closed); rows.push([t, back, queue.filter((d) => d.closed).length, CAPACITY - left, stillOpen.length, stillOpen.reduce((a, d) => a + d.effort, 0), Math.round(stillOpen.reduce((a, d) => a + d.age, 0) / stillOpen.length)]); for (const d of [...stillOpen, ...pending.filter((x) => !x.requeued)]) d.age += PERIOD_DAYS; } return { rows, stillOpen: queue.filter((d) => !d.closed), pending, back: rows.reduce((a, s) => a + s[1], 0) }; }; const y = gen([8, 10, 10, 12, 10, 14, 10]); const RUNS = [["A: acceptance never returns", false], ["B: expired acceptance returns", true]]; RUNS.forEach(([name, comeBack], i) => { const r = run(comeBack); console.log((i ? "\n" : "") + name); console.log(y("period", "returned", "closed", "spent", "left", "left person-days", "mean age")); for (const s of r.rows) console.log(y(...s)); const hidden = r.pending.filter((d) => !d.requeued); console.log(`debt visible in the ledger ${r.stillOpen.length} items / ` + `${r.stillOpen.reduce((a, d) => a + d.effort, 0)} person-days; ` + `accepted items that never entered the queue ${hidden.length} items / ` + `${hidden.reduce((a, d) => a + d.effort, 0)} person-days, risk total ` + `${hidden.reduce((a, d) => a + d.risk, 0)}`); }); console.log(`\ntotal capacity in four periods ${CAPACITY * PERIODS} person-days; the whole ledger ` + `${LEDGER.reduce((a, d) => a + d.effort, 0)} person-days`); console.log(`at planning time ${accepted.filter((d) => expired(d) && d.reassess).length} of the ` + `${accepted.filter(expired).length} expired acceptances carries a reassessment point; ` + `acceptances that returned to the queue over four periods ${run(true).back}`); console.log(`${accepted.filter((d) => d.validity === null).length} items with no validity ever written ` + `are never asked about again in any period`); ``` ``` A: acceptance never returns period returned closed spent leftleft person-days mean age 1 0 4 12 17 129 91 2 0 7 12 14 117 175 3 0 9 10 12 107 259 4 0 11 12 10 95 360 debt visible in the ledger 10 items / 95 person-days; accepted items that never entered the queue 13 items / 68 person-days, risk total 185 B: expired acceptance returns period returned closed spent leftleft person-days mean age 1 1 4 11 18 133 87 2 1 7 11 16 130 175 3 0 9 12 14 118 260 4 0 11 11 12 107 349 debt visible in the ledger 12 items / 107 person-days; accepted items that never entered the queue 11 items / 57 person-days, risk total 150 total capacity in four periods 48 person-days; the whole ledger 209 person-days at planning time 1 of the 3 expired acceptances carries a reassessment point; acceptances that returned to the queue over four periods 2 8 items with no validity ever written are never asked about again in any period ``` The queue closes **11 items** in four periods, and it closes the same number in both runs. The capacity is fixed: 48 person-days over four periods, against a ledger of **209 person-days**. This ratio writes the limit of the plan: with this capacity the ledger does not close, it is only ordered. In the third period the capacity spent stays at 10 person-days, because the next item does not fit the two remaining days; **a fixed-capacity queue cannot use its capacity in full**, and this loss is a consequence of the item size, not of the ordering rule. The real information is in the last two columns. While the number of remaining items falls from 17 to 10, the mean age rises from **91 days to 360 days**. The queue gets shorter and older: the rule of risk per unit of effort pulls the cheap and risky items forward, and the expensive items are carried over to the next period every time. The success of a debt plan is read not from the number of closed items but from **the age of the items that never reach the front of the queue.** The difference between the two runs is a single line, and it is the answer of the lesson. In run A, after four periods the ledger shows 10 items and 95 person-days; the 13 accepted items that never entered the queue carry 68 person-days and 185 risk points, and are never asked about in any period. In run B, the two expired acceptances come back, the ledger grows to 12 items, and the hidden load falls from 185 to 150. **The growth of the ledger is not a deterioration, it is the widening of the measurement.** The reason the number of returning items stays at two is also clear: in eight acceptances the validity was never written, and an acceptance with no validity is the one kind of item that no rule can bring back. #### Summary - Security debt consists of two classes and is separated from technical debt by a single field: an acceptance has a validity. Of the ledger's 34 items, 21 are open debt and 13 are accepted risk. - Only 5 of the acceptances have a validity written, 2 a reassessment point, and 6 a named owner; of the five with a validity written, three had already expired at planning time. - Of six questions, one is answered fully, four partially, and one is unanswered; the cost of making the partial ones full is 26 fields, and this cost needs no pipeline setup or extra volume. - Over four periods, 48 person-days of capacity close 11 items of the 209 person-day ledger; while the remaining queue falls from 17 items to 10, its mean age rises from 91 days to 360 days. - An acceptance with no validity and no reassessment point permanently drops out of the ledger: 11 of the 13 acceptances are never asked about in any period, and 150 risk points stay outside the measurement. #### Course Wrap-Up Over nine lessons a single question was asked: of the questions to be asked after an incident, how many can be answered, and what does it cost to make them answerable. Each lesson had its own set of items, its own set of questions, and its own cost column. | Lesson | How many questions | Answerable | Cost of making them answerable | |---|---|---|---| | Audit Log Design | 16 questions | 6 → 13 | 1.45 times the volume, 65→45 days retention, 3 personal data items | | Application-Specific Detection Scenarios | 18 abuse cases | 15 → 18 | 173 false positives, 101 off-hours pages in 30 days | | Application Breach Response | 12 steps | 5 → 12 from records | 630→151 minutes; notifications to 41,200 subscribers without records | | Trace in Short-Lived Workloads | 5 questions × 2,000 workloads | 6,604 / 10,000 pairs | 91,084 extra flushes, 33 core-minutes | | External Report Intake | 91 unique reports | unanswered 36 → 0, definite 15 → 20 | 5 fields, 1.45 times the volume, 20 days shorter retention | | Engineering Equivalent of Controls | 26 controls | 11 queries, 6 documents | a practice for 9 controls, a machine trace for 6 documents | | Evidence Automation | 24 evidence items | 12 for the whole period, 6 for the last 90 days | 170 hours of setup, 3,017 times the volume, 0.63% sampling rate | | Application Security Metrics | 8 questions | 5 full, 1 partial | 568 additional state transition records | | Security Debt | 6 questions | 1 full, 4 partial | 26 fields; 48 person-days of capacity for 209 person-days of debt | This course is the fifth and last course of the M25 Application Security and DevSecOps curriculum. Each of the five courses caught the same item at a different stage, and each counted in its own unit. | Course | Measure axis | Unit | |---|---|---| | K01 The Secure Development Lifecycle | catching stage and multiplier | stage | | K02 Secure Coding | coverage | point left outside | | K03 Security Automation and the Supply Chain | what the audit found, missed, and took from the pipeline | findings and minutes | | K04 Runtime Security and Secrets Management | holding at the most expensive stage | missed and wrongly cut | | K05 Detection, Response and Compliance | answerability of the question | unanswered question | The one question asked across the five courses was this: **which stage was an item caught at, and what did it cost at that stage.** Security is not a stage, it is a decision with a different price at every stage; and the most expensive stage is always the last one. The ledger of this course is the final form of that: an accepted risk is the recorded form of an item that was never caught at any stage, and when its validity is not written, the chance of its being caught drops out of the record as well. The rule of the curriculum is written once more: **an audit whose cost is not written is counted as unmeasured.** If the volume, the minutes, the personal data items, and the person-days are not written next to a record, an alert, an evidence item, or a metric, then the only thing known about that audit is that it was done — not what it cost and not what it missed.