--- title: 'Lessons Learned' source: 'https://academia.sh/en/courses/security-operations/lessons-learned' course: 'Security Operations and Incident Response' language: en updated: '2026-09-26T03:02:57+00:00' license: 'CC BY-SA 4.0' objectives: - 'Ties review questions to the record sources they require, and counts answered versus unanswered questions' - 'Splits the reason for being unanswered into three: no source, retention ran out, deleted at containment' - "Splits improvement items along the record-versus-preparation axis and works out each one's annual person-hour cost" --- # Lessons Learned Measuring the review along the axis of evidence and scope: eleven of sixteen questions being answered with nine record sources, five questions staying unanswered for three distinct reasons, five of twelve improvement items resting directly on a missing record, record-only improvements bringing the answered-question count to 16 for 136 person-hours while shaving 22 minutes off the chain, and preparation and process improvements gaining 55 minutes for 103 person-hours while opening no questions at all. The previous lesson brought the service back and lowered the expected footholds reaching production to 0.10. The incident is closed. What remains is reconstructing the incident itself: what happened, when it began, which decision was made at which minute, and which item, done differently, would have shortened the timeline. This lesson's measure is not the review's output but the review's **evidence base**. Opening, closing, and tracking action items was measured in the DevOps and Observability course; here, the step before that is what counts. A question set is built, the record sources needed to answer each question are written down, and three numbers are drawn out: how many questions are answered, how many stay unanswered **for lack of a record**, and how many of the improvement items opened rest directly on a missing record. ## The Question Set and Record Sources **IR34:** The review takes place twelve days after the incident; the first foothold was nine days ago. **IR35:** There are sixteen questions; each question requires one or two record sources, and which day it needs to look at is known. **IR36:** Each of the nine record sources has three properties — whether it exists, its retention period in days, and whether it persists past containment and recovery. **IR37:** A question is answered only when all the sources it requires are available; if a source does not exist, if its retention period does not cover the day the question needs, or if the source was deleted at containment, the question stays unanswered. **IR38:** There are twelve improvement items; each one either rests directly on a missing record, refreshes a preparation item, or ties in a process step. **IR39:** Each item's annual person-hour cost and the minutes it shaves off the decision chain are given; the minute values come from rerunning the chain. **IR40:** The decision chain's starting value is the full-preparation value from the first lesson: 182 minutes. ```js // lessons/model.mjs — lessons-learned model: no real incident record or review involved. // The question set, record sources and improvement items are data structures. export const REVIEW_DAY = 12, FIRST_FOOTHOLD = 9; // day the review is held; how many days ago the first foothold occurred // record source: name -> { does it exist, retention in days, does it persist past containment and recovery } export const SOURCES = { process: { name: "process-creation record", exists: true, retention: 14, persists: true }, identity: { name: "authentication log", exists: true, retention: 90, persists: true }, network: { name: "network flow data", exists: true, retention: 30, persists: true }, application: { name: "application audit log", exists: true, retention: 7, persists: true }, detection: { name: "detection pipeline log", exists: true, retention: 30, persists: true }, response: { name: "response log", exists: true, retention: 365, persists: true }, file: { name: "file integrity monitoring", exists: false, retention: 0, persists: false }, data: { name: "data access log", exists: false, retention: 0, persists: false }, memory: { name: "memory dump", exists: true, retention: 365, persists: false }, }; // question: text, required sources, how many days ago the question needs to look at export const QUESTIONS = [ ["which asset the first execution occurred on and when", ["process"], 9], ["which identity gained the first access", ["identity"], 9], ["which assets the service account was used on", ["identity", "network"], 9], ["whether collector-07 and -11 were actually affected", ["process"], 0], ["how much data volume was exfiltrated", ["network"], 0], ["which account read the billing archive", ["application"], 9], ["what the last day's application accesses were", ["application"], 3], ["when the persistence points were established", ["file"], 9], ["at what moment the configuration hooks were added", ["memory", "file"], 9], ["why the alert fired at minute 186", ["detection"], 0], ["whether a rule existed that could have fired earlier", ["detection", "process"], 0], ["what data the scope decision was based on", ["response"], 0], ["at what minute containment was applied", ["response"], 0], ["how long the legitimate work that was cut off lasted", ["response", "application"], 0], ["when the missed points were established", ["memory", "file"], 9], ["which subscriber records were read", ["data"], 9], ]; // why a source cannot answer the question export function reason(k, day) { const s = SOURCES[k]; if (!s.exists) return "no source"; if (!s.persists) return "deleted at containment"; if (day > s.retention) return "retention ran out"; return null; } export function answer(question, added = []) { const [, required, day] = question; const missing = required.filter((k) => !added.includes(k) && reason(k, day)); return { answered: missing.length === 0, reasons: missing.map((k) => `${k}: ${reason(k, day)}`) }; } export const answered = (added = []) => QUESTIONS.filter((s) => answer(s, added).answered).length; // improvement item: name, the gap it rests on (source key or preparation item), type, // annual person-hours, minutes it shaves off the decision chain export const IMPROVEMENTS = [ ["turning on file integrity monitoring", "file", "record", 34, 0], ["turning on the data access log", "data", "record", 46, 0], ["extending application retention to 30 days", "application", "record", 12, 0], ["capturing memory before containment", "memory", "record", 18, 0], ["process-creation logging for collector-07..12", "process", "record", 26, 22], ["keeping the asset inventory current", "-", "preparation", 24, 10], ["refreshing the communication tree", "-", "preparation", 6, 6], ["refreshing the authorization matrix", "-", "preparation", 8, 8], ["mapping out record sources", "-", "preparation", 12, 7], ["a quarterly tabletop exercise", "-", "preparation", 30, 9], ["tying containment actions to a procedure", "-", "process", 14, 8], ["turning the scope criterion into a dashboard metric", "-", "process", 9, 7], ]; export const CHAIN_BASE = 182, ALERT = 186; // the previous lessons' full-preparation decision chain // access curve: [minute, asset] (same data as the previous lessons) export const ACCESS = [[0, "collector-03"], [26, "collector-03"], [58, "validator-01"], [95, "collector-07"], [134, "collector-11"], [171, "validator-02"], [214, "billing-01"], [263, "portal-01"], [318, "workorder-01"], [372, "endpoint-01"], [441, "storage-01"], [505, "billing-01"], [588, "collector-09"], [676, "console-01"], [762, "portal-02"], [845, "storage-01"], [951, "billing-02"], [1088, "console-01"]]; export const itemsReached = (min) => ACCESS.filter(([t]) => t <= min).length; // selected improvement set: answered questions, chain minutes, containment moment, items reached export function apply(selected) { const k = IMPROVEMENTS.filter(([name]) => selected.includes(name)); const added = k.filter((x) => x[2] === "record").map((x) => x[1]); const chain = CHAIN_BASE - k.reduce((s, x) => s + x[4], 0); return { items: k.length, answered: answered(added), chain, containment: ALERT + chain, reached: itemsReached(ALERT + chain), hours: k.reduce((s, x) => s + x[3], 0) }; } ``` The measurement file first matches sources to questions, then applies the improvement sets and works out the answered-question count together with the chain minutes. ```js // lessons/measure.mjs — which question stays unanswered for lack of a record, what the reason // for that is, and how many of the improvement items opened rest directly on a missing record. import { REVIEW_DAY, SOURCES, QUESTIONS, IMPROVEMENTS, CHAIN_BASE, reason, answer, answered, apply } from "./model.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); console.log(`review takes place ${REVIEW_DAY} days after the incident; ${QUESTIONS.length} questions, ` + `${Object.keys(SOURCES).length} record sources`); console.log("\n1. record sources"); const A = [-26, 7, 10, 9, 10, 12]; write(A, "source", "exists", "retention", "persists", "required", "blocked"); for (const [k, s] of Object.entries(SOURCES)) { const required = QUESTIONS.filter(([, g]) => g.includes(k)); const blocked = required.filter(([, , day]) => reason(k, day)); write(A, s.name, s.exists ? "yes" : "no", s.exists ? s.retention + " days" : "-", s.persists ? "yes" : "no", required.length, blocked.length); } console.log("\n2. unanswered questions and their reason"); const B = [-4, -50, -48]; for (const s of QUESTIONS) { const y = answer(s); if (!y.answered) write(B, "", s[0], y.reasons.join("; ")); } console.log(` answered ${answered()}/${QUESTIONS.length}`); console.log("\n3. improvement items"); const C = [-52, 12, 18, 12, 11]; write(C, "item", "type", "gap it closes", "annual p-h", "chain min"); for (const [name, gap, type, hours, min] of IMPROVEMENTS) write(C, name, type, gap === "-" ? "-" : SOURCES[gap].name.split(" ")[0], hours, min || "-"); const recordItems = IMPROVEMENTS.filter((x) => x[2] === "record").length; console.log(` items resting directly on a missing record: ${recordItems}/${IMPROVEMENTS.length}`); console.log("\n4. when improvement sets are applied"); const D = [-24, 8, 10, 11, 12, 11, 12]; write(D, "set", "items", "answered", "chain min", "containment", "reached", "annual p-h"); const names = (t) => IMPROVEMENTS.filter((x) => t.includes(x[2])).map((x) => x[0]); const SETS = { "none": [], "record only": names(["record"]), "preparation + process": names(["preparation", "process"]), "all": names(["record", "preparation", "process"]) }; for (const [k, selected] of Object.entries(SETS)) { const r = apply(selected); write(D, k, r.items, `${r.answered}/${QUESTIONS.length}`, r.chain, r.containment, `${r.reached}/18`, r.hours); } console.log("\n5. where the two axes do not convert into each other"); const recordR = apply(SETS["record only"]), prepR = apply(SETS["preparation + process"]); const E = [-23, 12, 17, 15, -22]; write(E, "set", "annual p-h", "questions opened", "minutes gained", " p-h per unit"); write(E, "record only", recordR.hours, recordR.answered - answered(), CHAIN_BASE - recordR.chain, ` ${(recordR.hours / (recordR.answered - answered())).toFixed(1)} / question`); write(E, "preparation + process", prepR.hours, prepR.answered - answered(), CHAIN_BASE - prepR.chain, ` ${(prepR.hours / (CHAIN_BASE - prepR.chain)).toFixed(1)} / minute`); const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfour sets and the question scan within the 500 ms band: ${ms < 500 ? "yes" : "no"}`); ``` ``` review takes place 12 days after the incident; 16 questions, 9 record sources 1. record sources source exists retention persists required blocked process-creation record yes 14 days yes 3 0 authentication log yes 90 days yes 2 0 network flow data yes 30 days yes 2 0 application audit log yes 7 days yes 3 1 detection pipeline log yes 30 days yes 2 0 response log yes 365 days yes 3 0 file integrity monitoring no - no 3 3 data access log no - no 1 1 memory dump yes 365 days no 2 2 2. unanswered questions and their reason which account read the billing archive application: retention ran out when the persistence points were established file: no source at what moment the configuration hooks were added memory: deleted at containment; file: no source when the missed points were established memory: deleted at containment; file: no source which subscriber records were read data: no source answered 11/16 3. improvement items item type gap it closes annual p-h chain min turning on file integrity monitoring record file 34 - turning on the data access log record data 46 - extending application retention to 30 days record application 12 - capturing memory before containment record memory 18 - process-creation logging for collector-07..12 record process-creation 26 22 keeping the asset inventory current preparation - 24 10 refreshing the communication tree preparation - 6 6 refreshing the authorization matrix preparation - 8 8 mapping out record sources preparation - 12 7 a quarterly tabletop exercise preparation - 30 9 tying containment actions to a procedure process - 14 8 turning the scope criterion into a dashboard metric process - 9 7 items resting directly on a missing record: 5/12 4. when improvement sets are applied set items answered chain min containment reached annual p-h none 0 11/16 182 368 9/18 0 record only 5 16/16 160 346 9/18 136 preparation + process 7 11/16 127 313 8/18 103 all 12 16/16 105 291 8/18 239 5. where the two axes do not convert into each other set annual p-h questions opened minutes gained p-h per unit record only 136 5 22 27.2 / question preparation + process 103 0 55 1.9 / minute four sets and the question scan within the 500 ms band: yes ``` The numbers belong to the **measurement** class; their inputs are the assumptions above. ## Eleven Have an Answer The first table counts the sources. Six sources exist and persist: the process-creation record, the authentication log, network flow data, the application audit log, the detection pipeline log, and the response log. The last column gives how many questions each source **blocks**. File integrity monitoring is required for three questions and blocks all three, because it does not exist at all. The memory dump is required for two questions and blocks both, because it does not persist. The application audit log is required for three questions and blocks only one — that question looks nine days back, while retention is seven days. The second table lists the five unanswered questions along with their reason, and the reason falls into three distinct kinds. **No source:** the questions of when the persistence points were established and which subscriber records were read are unanswered because file integrity monitoring and the data access log were never turned on at all. **Retention ran out:** which account read the billing archive is unanswered because it falls outside the application record's seven-day window — the source existed, the record existed, but it had been deleted. **Deleted at containment:** the questions of when the configuration hooks and the missed points were established are unanswered because the assets were pulled off the network before a memory dump was taken. This third reason comes straight out of this topic. In the third lesson, short-term containment stopped access at minute 393 and gained five items; in the fourth lesson, the same decision made memory examination impossible on seven assets and caused three configuration hooks to be missed. Here the same decision's third counterpart appears: two review questions can never be answered. **The containment decision's cost is paid in three separate lessons, in three separate units.** ## Five Rest on a Record The third table lists the twelve improvement items. Five rest directly on a missing record: turning on file integrity monitoring, turning on the data access log, extending application retention to thirty days, capturing memory before containment, and adding logging to the six collectors that were not forwarding process-creation records. Five refresh preparation items, and two tie in a process step. **Five of the twelve items are the direct consequence of a missing record.** The fourth table applies the sets. Record-only improvements cost five items and 136 person-hours a year; the answered-question count rises from 11 to **16** and the chain drops from 182 to 160 minutes — the only minute gain comes from adding process-creation logging. Preparation and process improvements cost seven items and 103 person-hours; the chain drops to 127 minutes, containment to minute 313, and the items reached fall from nine to eight, but the answered-question count **stays at 11**. All together cost 239 person-hours, give a 105-minute chain, and answer 16/16. ## Two Axes The fifth table shows that the two sets do not convert into each other. Record improvements cost 27.2 person-hours per question and shave only 22 minutes off the chain. Preparation and process improvements cost 1.9 person-hours per minute and open no questions at all. The same budget buys two different things: one buys **stopping the next incident faster**, the other buys **telling this incident's story better**. Ranking the two against each other is not this model's job, but one observation can be made. One of the record improvements — capturing memory before containment — opens two questions for eighteen person-hours and adds no minutes to the chain at all; in exchange, it delays containment, and that delay's cost was measured in the third lesson. If an improvement item carries a cost on another axis beyond the question it opens, the item has been measured incompletely unless that cost is also written into the list. ## Summary - Eleven of the sixteen review questions are answered with nine record sources; five stay unanswered. - The three reasons for being unanswered are counted separately: two questions have no source at all, one question's retention period was not enough, and two questions' sources were deleted at containment. - The containment decision's cost is five items in the third lesson, three missed footholds in the fourth lesson, and two unanswered questions here. - Five of the twelve improvement items rest directly on a missing record; five refresh a preparation item, and two tie in a process step. - Record-only improvements bring the answered-question count to 16/16 for 136 person-hours and shave 22 minutes off the chain; preparation and process improvements gain 55 minutes for 103 person-hours and open no questions at all. ## Next Step The review has drawn out its answers, written down the improvement items, and counted their costs. All of this stayed inside the organization. There is one more face of the incident that looks outward: who needs to be notified, at what threshold, within what time. The next lesson measures that as time and stakeholder count — how many stakeholders there are, at what threshold notification begins, how many corrections a notification made before scope is finalized produces, and how many hours the delay costs.