Lessons Learned

The previous lesson brought the service back and lowered the expected footholds reaching production to 0.10. The incident is closed. What remains is reconstructing the incident itself: what happened, when it began, which decision was made at which minute, and which item, done differently, would have shortened the timeline.

This lesson’s measure is not the review’s output but the review’s evidence base. Opening, closing, and tracking action items was measured in the DevOps and Observability course; here, the step before that is what counts. A question set is built, the record sources needed to answer each question are written down, and three numbers are drawn out: how many questions are answered, how many stay unanswered for lack of a record, and how many of the improvement items opened rest directly on a missing record.

The Question Set and Record Sources

IR34: The review takes place twelve days after the incident; the first foothold was nine days ago. IR35: There are sixteen questions; each question requires one or two record sources, and which day it needs to look at is known. IR36: Each of the nine record sources has three properties — whether it exists, its retention period in days, and whether it persists past containment and recovery. IR37: A question is answered only when all the sources it requires are available; if a source does not exist, if its retention period does not cover the day the question needs, or if the source was deleted at containment, the question stays unanswered. IR38: There are twelve improvement items; each one either rests directly on a missing record, refreshes a preparation item, or ties in a process step. IR39: Each item’s annual person-hour cost and the minutes it shaves off the decision chain are given; the minute values come from rerunning the chain. IR40: The decision chain’s starting value is the full-preparation value from the first lesson: 182 minutes.

// lessons/model.mjs — lessons-learned model: no real incident record or review involved.
// The question set, record sources and improvement items are data structures.

export const REVIEW_DAY = 12, FIRST_FOOTHOLD = 9;   // day the review is held; how many days ago the first foothold occurred

// record source: name -> { does it exist, retention in days, does it persist past containment and recovery }
export const SOURCES = {
  process: { name: "process-creation record", exists: true, retention: 14, persists: true },
  identity: { name: "authentication log", exists: true, retention: 90, persists: true },
  network: { name: "network flow data", exists: true, retention: 30, persists: true },
  application: { name: "application audit log", exists: true, retention: 7, persists: true },
  detection: { name: "detection pipeline log", exists: true, retention: 30, persists: true },
  response: { name: "response log", exists: true, retention: 365, persists: true },
  file: { name: "file integrity monitoring", exists: false, retention: 0, persists: false },
  data: { name: "data access log", exists: false, retention: 0, persists: false },
  memory: { name: "memory dump", exists: true, retention: 365, persists: false },
};

// question: text, required sources, how many days ago the question needs to look at
export const QUESTIONS = [
  ["which asset the first execution occurred on and when", ["process"], 9],
  ["which identity gained the first access", ["identity"], 9],
  ["which assets the service account was used on", ["identity", "network"], 9],
  ["whether collector-07 and -11 were actually affected", ["process"], 0],
  ["how much data volume was exfiltrated", ["network"], 0],
  ["which account read the billing archive", ["application"], 9],
  ["what the last day's application accesses were", ["application"], 3],
  ["when the persistence points were established", ["file"], 9],
  ["at what moment the configuration hooks were added", ["memory", "file"], 9],
  ["why the alert fired at minute 186", ["detection"], 0],
  ["whether a rule existed that could have fired earlier", ["detection", "process"], 0],
  ["what data the scope decision was based on", ["response"], 0],
  ["at what minute containment was applied", ["response"], 0],
  ["how long the legitimate work that was cut off lasted", ["response", "application"], 0],
  ["when the missed points were established", ["memory", "file"], 9],
  ["which subscriber records were read", ["data"], 9],
];

// why a source cannot answer the question
export function reason(k, day) {
  const s = SOURCES[k];
  if (!s.exists) return "no source";
  if (!s.persists) return "deleted at containment";
  if (day > s.retention) return "retention ran out";
  return null;
}
export function answer(question, added = []) {
  const [, required, day] = question;
  const missing = required.filter((k) => !added.includes(k) && reason(k, day));
  return { answered: missing.length === 0,
    reasons: missing.map((k) => `${k}: ${reason(k, day)}`) };
}
export const answered = (added = []) => QUESTIONS.filter((s) => answer(s, added).answered).length;

// improvement item: name, the gap it rests on (source key or preparation item), type,
// annual person-hours, minutes it shaves off the decision chain
export const IMPROVEMENTS = [
  ["turning on file integrity monitoring", "file", "record", 34, 0],
  ["turning on the data access log", "data", "record", 46, 0],
  ["extending application retention to 30 days", "application", "record", 12, 0],
  ["capturing memory before containment", "memory", "record", 18, 0],
  ["process-creation logging for collector-07..12", "process", "record", 26, 22],
  ["keeping the asset inventory current", "-", "preparation", 24, 10],
  ["refreshing the communication tree", "-", "preparation", 6, 6],
  ["refreshing the authorization matrix", "-", "preparation", 8, 8],
  ["mapping out record sources", "-", "preparation", 12, 7],
  ["a quarterly tabletop exercise", "-", "preparation", 30, 9],
  ["tying containment actions to a procedure", "-", "process", 14, 8],
  ["turning the scope criterion into a dashboard metric", "-", "process", 9, 7],
];
export const CHAIN_BASE = 182, ALERT = 186;  // the previous lessons' full-preparation decision chain
// access curve: [minute, asset] (same data as the previous lessons)
export const ACCESS = [[0, "collector-03"], [26, "collector-03"], [58, "validator-01"],
  [95, "collector-07"], [134, "collector-11"], [171, "validator-02"], [214, "billing-01"],
  [263, "portal-01"], [318, "workorder-01"], [372, "endpoint-01"], [441, "storage-01"],
  [505, "billing-01"], [588, "collector-09"], [676, "console-01"], [762, "portal-02"],
  [845, "storage-01"], [951, "billing-02"], [1088, "console-01"]];
export const itemsReached = (min) => ACCESS.filter(([t]) => t <= min).length;

// selected improvement set: answered questions, chain minutes, containment moment, items reached
export function apply(selected) {
  const k = IMPROVEMENTS.filter(([name]) => selected.includes(name));
  const added = k.filter((x) => x[2] === "record").map((x) => x[1]);
  const chain = CHAIN_BASE - k.reduce((s, x) => s + x[4], 0);
  return { items: k.length, answered: answered(added), chain, containment: ALERT + chain,
    reached: itemsReached(ALERT + chain), hours: k.reduce((s, x) => s + x[3], 0) };
}

The measurement file first matches sources to questions, then applies the improvement sets and works out the answered-question count together with the chain minutes.

// lessons/measure.mjs — which question stays unanswered for lack of a record, what the reason
// for that is, and how many of the improvement items opened rest directly on a missing record.
import { REVIEW_DAY, SOURCES, QUESTIONS, IMPROVEMENTS, CHAIN_BASE, reason, answer, answered,
  apply } from "./model.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const t0 = process.hrtime.bigint();

console.log(`review takes place ${REVIEW_DAY} days after the incident; ${QUESTIONS.length} questions, ` +
  `${Object.keys(SOURCES).length} record sources`);

console.log("\n1. record sources");
const A = [-26, 7, 10, 9, 10, 12];
write(A, "source", "exists", "retention", "persists", "required", "blocked");
for (const [k, s] of Object.entries(SOURCES)) {
  const required = QUESTIONS.filter(([, g]) => g.includes(k));
  const blocked = required.filter(([, , day]) => reason(k, day));
  write(A, s.name, s.exists ? "yes" : "no", s.exists ? s.retention + " days" : "-",
    s.persists ? "yes" : "no", required.length, blocked.length);
}

console.log("\n2. unanswered questions and their reason");
const B = [-4, -50, -48];
for (const s of QUESTIONS) {
  const y = answer(s);
  if (!y.answered) write(B, "", s[0], y.reasons.join("; "));
}
console.log(`  answered ${answered()}/${QUESTIONS.length}`);

console.log("\n3. improvement items");
const C = [-52, 12, 18, 12, 11];
write(C, "item", "type", "gap it closes", "annual p-h", "chain min");
for (const [name, gap, type, hours, min] of IMPROVEMENTS)
  write(C, name, type, gap === "-" ? "-" : SOURCES[gap].name.split(" ")[0], hours, min || "-");
const recordItems = IMPROVEMENTS.filter((x) => x[2] === "record").length;
console.log(`  items resting directly on a missing record: ${recordItems}/${IMPROVEMENTS.length}`);

console.log("\n4. when improvement sets are applied");
const D = [-24, 8, 10, 11, 12, 11, 12];
write(D, "set", "items", "answered", "chain min", "containment", "reached", "annual p-h");
const names = (t) => IMPROVEMENTS.filter((x) => t.includes(x[2])).map((x) => x[0]);
const SETS = { "none": [], "record only": names(["record"]),
  "preparation + process": names(["preparation", "process"]),
  "all": names(["record", "preparation", "process"]) };
for (const [k, selected] of Object.entries(SETS)) {
  const r = apply(selected);
  write(D, k, r.items, `${r.answered}/${QUESTIONS.length}`, r.chain, r.containment,
    `${r.reached}/18`, r.hours);
}

console.log("\n5. where the two axes do not convert into each other");
const recordR = apply(SETS["record only"]), prepR = apply(SETS["preparation + process"]);
const E = [-23, 12, 17, 15, -22];
write(E, "set", "annual p-h", "questions opened", "minutes gained", "  p-h per unit");
write(E, "record only", recordR.hours, recordR.answered - answered(), CHAIN_BASE - recordR.chain,
  `  ${(recordR.hours / (recordR.answered - answered())).toFixed(1)} / question`);
write(E, "preparation + process", prepR.hours, prepR.answered - answered(), CHAIN_BASE - prepR.chain,
  `  ${(prepR.hours / (CHAIN_BASE - prepR.chain)).toFixed(1)} / minute`);

const ms = Number(process.hrtime.bigint() - t0) / 1e6;   // the raw measurement is not printed, only the band is checked
console.log(`\nfour sets and the question scan within the 500 ms band: ${ms < 500 ? "yes" : "no"}`);

review takes place 12 days after the incident; 16 questions, 9 record sources

1. record sources
source                     exists retention persists  required     blocked
process-creation record       yes   14 days      yes         3           0
authentication log            yes   90 days      yes         2           0
network flow data             yes   30 days      yes         2           0
application audit log         yes    7 days      yes         3           1
detection pipeline log        yes   30 days      yes         2           0
response log                  yes  365 days      yes         3           0
file integrity monitoring      no         -       no         3           3
data access log                no         -       no         1           1
memory dump                   yes  365 days       no         2           2

2. unanswered questions and their reason
    which account read the billing archive            application: retention ran out
    when the persistence points were established      file: no source
    at what moment the configuration hooks were added memory: deleted at containment; file: no source
    when the missed points were established           memory: deleted at containment; file: no source
    which subscriber records were read                data: no source
  answered 11/16

3. improvement items
item                                                        type     gap it closes  annual p-h  chain min
turning on file integrity monitoring                      record              file          34          -
turning on the data access log                            record              data          46          -
extending application retention to 30 days                record       application          12          -
capturing memory before containment                       record            memory          18          -
process-creation logging for collector-07..12             record  process-creation          26         22
keeping the asset inventory current                  preparation                 -          24         10
refreshing the communication tree                    preparation                 -           6          6
refreshing the authorization matrix                  preparation                 -           8          8
mapping out record sources                           preparation                 -          12          7
a quarterly tabletop exercise                        preparation                 -          30          9
tying containment actions to a procedure                 process                 -          14          8
turning the scope criterion into a dashboard metric      process                 -           9          7
  items resting directly on a missing record: 5/12

4. when improvement sets are applied
set                        items  answered  chain min containment    reached  annual p-h
none                           0     11/16        182         368       9/18           0
record only                    5     16/16        160         346       9/18         136
preparation + process          7     11/16        127         313       8/18         103
all                           12     16/16        105         291       8/18         239

5. where the two axes do not convert into each other
set                      annual p-h questions opened minutes gained  p-h per unit
record only                     136                5             22  27.2 / question
preparation + process           103                0             55  1.9 / minute

four sets and the question scan within the 500 ms band: yes

The numbers belong to the measurement class; their inputs are the assumptions above.

Eleven Have an Answer

The first table counts the sources. Six sources exist and persist: the process-creation record, the authentication log, network flow data, the application audit log, the detection pipeline log, and the response log. The last column gives how many questions each source blocks. File integrity monitoring is required for three questions and blocks all three, because it does not exist at all. The memory dump is required for two questions and blocks both, because it does not persist. The application audit log is required for three questions and blocks only one — that question looks nine days back, while retention is seven days.

The second table lists the five unanswered questions along with their reason, and the reason falls into three distinct kinds. No source: the questions of when the persistence points were established and which subscriber records were read are unanswered because file integrity monitoring and the data access log were never turned on at all. Retention ran out: which account read the billing archive is unanswered because it falls outside the application record’s seven-day window — the source existed, the record existed, but it had been deleted. Deleted at containment: the questions of when the configuration hooks and the missed points were established are unanswered because the assets were pulled off the network before a memory dump was taken.

This third reason comes straight out of this topic. In the third lesson, short-term containment stopped access at minute 393 and gained five items; in the fourth lesson, the same decision made memory examination impossible on seven assets and caused three configuration hooks to be missed. Here the same decision’s third counterpart appears: two review questions can never be answered. The containment decision’s cost is paid in three separate lessons, in three separate units.

Five Rest on a Record

The third table lists the twelve improvement items. Five rest directly on a missing record: turning on file integrity monitoring, turning on the data access log, extending application retention to thirty days, capturing memory before containment, and adding logging to the six collectors that were not forwarding process-creation records. Five refresh preparation items, and two tie in a process step. Five of the twelve items are the direct consequence of a missing record.

The fourth table applies the sets. Record-only improvements cost five items and 136 person-hours a year; the answered-question count rises from 11 to 16 and the chain drops from 182 to 160 minutes — the only minute gain comes from adding process-creation logging. Preparation and process improvements cost seven items and 103 person-hours; the chain drops to 127 minutes, containment to minute 313, and the items reached fall from nine to eight, but the answered-question count stays at 11. All together cost 239 person-hours, give a 105-minute chain, and answer 16/16.

Two Axes

The fifth table shows that the two sets do not convert into each other. Record improvements cost 27.2 person-hours per question and shave only 22 minutes off the chain. Preparation and process improvements cost 1.9 person-hours per minute and open no questions at all. The same budget buys two different things: one buys stopping the next incident faster, the other buys telling this incident’s story better.

Ranking the two against each other is not this model’s job, but one observation can be made. One of the record improvements — capturing memory before containment — opens two questions for eighteen person-hours and adds no minutes to the chain at all; in exchange, it delays containment, and that delay’s cost was measured in the third lesson. If an improvement item carries a cost on another axis beyond the question it opens, the item has been measured incompletely unless that cost is also written into the list.

Summary

• Eleven of the sixteen review questions are answered with nine record sources; five stay unanswered.
• The three reasons for being unanswered are counted separately: two questions have no source at all, one question’s retention period was not enough, and two questions’ sources were deleted at containment.
• The containment decision’s cost is five items in the third lesson, three missed footholds in the fourth lesson, and two unanswered questions here.
• Five of the twelve improvement items rest directly on a missing record; five refresh a preparation item, and two tie in a process step.
• Record-only improvements bring the answered-question count to 16/16 for 136 person-hours and shave 22 minutes off the chain; preparation and process improvements gain 55 minutes for 103 person-hours and open no questions at all.

Next Step

The review has drawn out its answers, written down the improvement items, and counted their costs. All of this stayed inside the organization. There is one more face of the incident that looks outward: who needs to be notified, at what threshold, within what time. The next lesson measures that as time and stakeholder count — how many stakeholders there are, at what threshold notification begins, how many corrections a notification made before scope is finalized produces, and how many hours the delay costs.