--- title: 'Observability and Operations' source: 'https://academia.sh/en/courses/devops-observability' language: en updated: '2026-09-13T05:08:21+00:00' license: 'CC BY-SA 4.0' lessons: 18 outcomes: - 'Sets up the metric, log, and trace collection pipeline end to end' - 'Designs alerts to be symptom-based and actionable' - 'Manages incident response with process and documentation' --- # Observability and Operations Deciding which question is to be answered and collecting the data to fit that question: pull-based and push-based metric collection, label cardinality and retention, producing indicators through queries and aggregation, log collection through the agent, buffer, and indexing layers, parsing the log into searchable fields, the balance between level of detail and cost, distributed tracing infrastructure, a shared telemetry convention, dashboards that answer a question, threshold-based, rate-based, and anomaly-based alert rules, alert fatigue, synthetic monitoring, a sustainable on-call rotation, incident classification and escalation, runbooks, blameless post-incident review, regulating the pace of change with an error budget, and monitoring capacity and cost. ## The Metric and Log Pipeline The path the data travels: the scope and latency difference between pull-based and push-based collection, the multiplicative growth of label cardinality in a time series and its retention, queries and aggregation for producing indicators from metrics, log collection through the agent, buffer, and indexing layers, parsing the log into searchable fields, and the balance between level of detail and cost. ### Metric Collection Models Source: https://academia.sh/en/courses/devops-observability/metric-collection-models Pull-based and push-based collection run in the same window and are applied to a set of twelve questions genuinely asked in operations: which model answers how many questions, why short-lived jobs slip through in pull, why resource identity gets confused in push, and how large the discovery delay and the cost per question are. The previous course closed at the point where the infrastructure was tied to a record, and every change passed through a review and a gate. But once past the gate, no one could see what the system did while it was running. This course's first question picks up from there and takes one step further: **who decides what gets measured?** This course's unit of measure is not data, it is the **question**. Whether a collection setup is good is measured not by how many gigabytes it accumulates, but by how many of the questions genuinely asked in operations it answers. This lesson builds that set of questions and runs two collection models — **pull-based** and **push-based** — in the same window, applying both to the set. The measurement is again on the **regional measurement network**: software that collects readings from water meters, validates the readings, turns them into bills, and opens work orders for field crews. Both the network and the measured numbers are **fictional**; no real collector is run, what gets counted is a **model** built with `node`. **ML1.** The twelve questions are fictional and written in advance, as is the resource class, metric, and labels each question requires. **ML2.** The inventory is fictional: 29 long-lived instances across three environments, 11 restarts within a one-hour window, and 42 nightly validation runs. **ML3.** The pull-based collector scrapes every 15 seconds and refreshes its target inventory every 30 seconds; the push-based sender flushes every 10 seconds in a long-lived process, and the batch job sends at the end of every step. **ML4.** A record is 16 bytes, and a series descriptor is 120 bytes. **ML5.** A resource class is counted as covered only if at least two measurements can be taken from at least 90% of its lifetimes; a single measurement does not show a change. **ML6.** Restart times and run durations are produced with our own generator, seed `20260901`. #### The Twelve Questions Asked in Operations The question set is a product of operations, not of the measurement tool: these are sentences genuinely asked during on-call, in the morning report, and during incident review. Each question is tied to three things — which **resource class** the data must come from, which **metric**, and which **labels**. ```js // network.mjs — regional measurement network model and the operational question set. Both are fictional. export const INSTANCES = { // environment -> service -> long-lived instances production: { "reading-collector": 8, validator: 6, billing: 4, "work-order": 3 }, staging: { "reading-collector": 2, validator: 2, billing: 1, "work-order": 1 }, development: { "reading-collector": 1, validator: 1, billing: 0, "work-order": 0 } }; export const LONG_COUNT = Object.values(INSTANCES) .reduce((t, o) => t + Object.values(o).reduce((a, b) => a + b, 0), 0); export const METRICS_LONG = { "request-count": { result: 5 }, "request-duration": { bucket: 12 }, "queue-depth": { queue: 3 }, "processed-reading": { region: 11, result: 5 }, "process-start-time": {}, "gc-pause": { bin: 8 }, "file-descriptor-count": {} }; export const METRICS_SHORT = { "job-step-duration": { step: 6 }, "job-processed-records": {} }; export const seriesCount = (o) => Object.values(o) .reduce((t, e) => t + Object.values(e).reduce((c, k) => c * k, 1), 0); // the twelve questions genuinely asked in operations: [id, text, resource class, metric, required labels] export const QUESTIONS = [ ["Q1", "which service slowed down", "long", "request-duration", ["service"]], ["Q2", "what is the tail bucket of latency", "long", "request-duration", ["service", "bucket"]], ["Q3", "which environment saw a rise in errors", "long", "request-count", ["environment", "result"]], ["Q4", "which version broke it", "long", "request-count", ["version"]], ["Q5", "how many readings did the nightly job process", "short", "job-processed-records", ["job"]], ["Q6", "why did the nightly job finish late yesterday", "short", "job-step-duration", ["job", "step"]], ["Q7", "which region's readings were not validated", "long", "processed-reading", ["region", "result"]], ["Q8", "which instance restarted", "long", "request-count", ["instance"]], ["Q9", "which validation run was slow", "short", "job-step-duration", ["job", "instance"]], ["Q10", "which error code rose", "long", "request-count", ["result"]], ["Q11", "why did the work-order queue back up", "long", "queue-depth", ["service", "queue"]], ["Q12", "which reading a bill was generated from", "request", "-", ["transaction-id"]], ]; export const answer = (scope, labels) => QUESTIONS.filter(([, , k, , a]) => scope.includes(k) && a.every((e) => labels.includes(e))); export const makeRng = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648; export const RECORD_BYTES = 16, SERIES_BYTES = 120; export const mb = (b) => (b / 1024 ** 2).toFixed(1); ``` There are three resource classes. The **long-lived** class is a service instance running for hours. The **short-lived** class is every run of the nightly batch job; it lives half a minute on average. The **request** class is the single transaction itself, and no metric can carry it, because the cardinality of a transaction ID is unbounded. Q12 is therefore marked from the start: a metric pipeline cannot answer it no matter how much data it collects. #### Pull and Push in the Same Window The two models run in the same one-hour window. The difference collects in three places: **what gets seen** (coverage), **when it gets seen** (discovery delay), and **whose reading it is recorded as** (resource identity). ```js // collect.mjs — pull-based and push-based collection run in the same window and are applied to the question set. import { INSTANCES, LONG_COUNT, METRICS_LONG, METRICS_SHORT, seriesCount, QUESTIONS, answer, makeRng, RECORD_BYTES, SERIES_BYTES, mb } from "./network.mjs"; const WINDOW = 3600, SCRAPE = 15, INVENTORY = 30, PUSH = 10; // seconds const rand = makeRng(20260901); // seed made visible const lifetimes = []; // 29 instances from window start, 11 restarts for (let i = 0; i < LONG_COUNT; i++) lifetimes.push({ class: "long", start: 0, end: WINDOW }); for (let i = 0; i < 11; i++) { const h = lifetimes[Math.floor(rand() * LONG_COUNT)], t = 300 + Math.floor(rand() * 3000); h.end = t; lifetimes.push({ class: "long", start: t + 20, end: WINDOW }); } const runs = Array.from({ length: 42 }, (_, i) => { // nightly validation runs const start = 60 + i * 80 + Math.floor(rand() * 30); return { class: "short", start, end: start + 6 + Math.floor(rand() * 33) }; }); const all = [...lifetimes, ...runs]; const pull = (o) => { // scrapes after the inventory refreshes const t = [], visible = Math.ceil(o.start / INVENTORY) * INVENTORY; for (let s = Math.ceil(visible / SCRAPE) * SCRAPE; s < o.end; s += SCRAPE) t.push(s); return t; }; const push = (o) => { // process sends itself; no discovery round const t = []; if (o.class === "short") { // batch job sends at the end of every step for (let a = 1; a <= 6; a++) t.push(o.start + Math.round(((o.end - o.start) * a) / 6)); return t; } for (let s = o.start + PUSH; s < o.end; s += PUSH) t.push(s); return t; }; const covered = (f, s) => all.filter((o) => o.class === s && f(o).length >= 2).length; const total = (s) => all.filter((o) => o.class === s).length; const discovery = (f) => { // only lifetimes that start inside the window const g = lifetimes.filter((o) => o.start > 0).map((o) => f(o)[0] - o.start); // restarted instances return [(g.reduce((a, b) => a + b, 0) / g.length).toFixed(1), Math.max(...g)]; }; const LABELS = ["service", "environment", "version", "result", "bucket", "queue", "region", "job", "step"]; const MODELS = [["pull", pull, LONG_COUNT, covered(pull, "short")], // identity comes from the inventory ["push", push, lifetimes.length, 3]]; // identity is the process's own report, job template is fixed console.log(`regional measurement network (fictional): ${LONG_COUNT} long-lived instances, ${Object.keys(INSTANCES).length} environments,` + ` ${lifetimes.length - LONG_COUNT} restarts, ${runs.length} nightly validation runs`); console.log(`${seriesCount(METRICS_LONG)} series per instance, ${seriesCount(METRICS_SHORT)} series per run; window ${WINDOW} s\n`); console.log("model long-lived short-lived discovery avg/max identity long identity short"); const summary = []; for (const [name, f, identityLong, identityShort] of MODELS) { const [avg, max] = discovery(f), lc = covered(f, "long"), sc = covered(f, "short"); const measured = all.reduce((t, o) => t + f(o).length * seriesCount(o.class === "long" ? METRICS_LONG : METRICS_SHORT), 0); const seriesTotal = identityLong * seriesCount(METRICS_LONG) + identityShort * seriesCount(METRICS_SHORT); summary.push([name, lc, sc, measured, seriesTotal, identityLong === LONG_COUNT]); // does identity survive a restart console.log(`${name.padEnd(7)}${String(`${lc}/${total("long")}`).padStart(12)}` + `${String(`${sc}/${total("short")}`).padStart(13)}${String(`${avg} / ${max} s`).padStart(19)}` + `${String(`${identityLong}/${lifetimes.length}`).padStart(15)}${String(`${identityShort}/${runs.length}`).padStart(16)}`); } console.log("\nagainst the question set (12 questions)"); console.log("model answered bytes/hour per question left unanswered"); for (const [name, lc, sc, measured, seriesTotal, stable] of summary) { const scope = [lc / total("long") >= 0.9 && "long", sc / total("short") >= 0.9 && "short"].filter(Boolean); const labels = stable ? [...LABELS, "instance"] : LABELS; // no instance label if identity is unstable const y = answer(scope, labels), bytes = measured * RECORD_BYTES + seriesTotal * SERIES_BYTES; console.log(`${name.padEnd(7)}${String(`${y.length}/${QUESTIONS.length}`).padStart(10)}` + `${String(`${mb(bytes)} MB`).padStart(12)}${String(`${mb(bytes / y.length)} MB`).padStart(14)} ` + QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" ")); } const unused = Object.keys(METRICS_LONG).filter((a) => !QUESTIONS.some(([, , , o]) => o === a)); const unusedSeries = unused.reduce((t, a) => t + seriesCount({ [a]: METRICS_LONG[a] }), 0); console.log(`\nmetrics answering no question: ${unused.join(", ")}`); console.log(` ${unusedSeries}/${seriesCount(METRICS_LONG)} series per instance (${(100 * unusedSeries / seriesCount(METRICS_LONG)).toFixed(1)}%),` + ` ${mb(LONG_COUNT * unusedSeries * (WINDOW / SCRAPE) * RECORD_BYTES)} MB/hour under pull`); const missedRuns = runs.length - covered(pull, "short"), REPORT = 7 * 3600, START = 2 * 3600; const wait = runs.reduce((t, o) => t + (REPORT - START - o.end), 0) / runs.length / 3600; console.log(`runs that produce no series under pull: ${missedRuns}: the failure surfaces only in the morning report,` + ` ${wait.toFixed(1)} hours later on average`); ``` ``` regional measurement network (fictional): 29 long-lived instances, 3 environments, 11 restarts, 42 nightly validation runs 85 series per instance, 7 series per run; window 3600 s model long-lived short-lived discovery avg/max identity long identity short pull 40/40 13/42 14.6 / 27 s 29/40 13/42 push 40/40 42/42 10.0 / 10 s 40/40 3/42 against the question set (12 questions) model answered bytes/hour per question left unanswered pull 8/12 9.7 MB 1.2 MB Q5 Q6 Q9 Q12 push 9/12 14.5 MB 1.6 MB Q8 Q9 Q12 metrics answering no question: process-start-time, gc-pause, file-descriptor-count 10/85 series per instance (11.8%), 1.1 MB/hour under pull runs that produce no series under pull: 29: the failure surfaces only in the morning report, 4.5 hours later on average ``` On the long-lived side the two models are equal: all forty of the forty lifetimes are covered under both models. The split opens on the short-lived side. The pull-based collector can take two or more measurements from only **thirteen** of the 42 validation runs. The reason is that two delays stack on top of each other: the inventory refreshes every thirty seconds, the scrape happens every fifteen seconds, and a run's average lifetime is under thirty seconds. A run ends before the round that would have seen it. Twenty-nine runs pass without producing a single series, and **their passing is invisible too** — a missing series is not an error, only an absence. The push side has no such problem, because no one has to discover the run: a run sends its own data and sends once more at the end of every step. All forty-two of the forty-two runs are covered. The identity columns flip the split. In pull, identity comes from **the inventory**; when an instance restarts, the inventory entry stays the same, so the series stays the same too — forty lifetimes collect under twenty-nine identities, and a restart shows up as a gap in the series. In push, identity is **the process's own report**; every new process generates itself a new identity, so forty lifetimes create forty separate identities, and a restarted instance cannot be told apart from a brand-new one never seen before. On the short-lived side the same mechanism breaks in the opposite direction: because the batch template writes a fixed job name, forty-two runs merge under **three identities** and individual runs blur into each other. #### The Questions the Two Models Leave Unanswered Pull answers eight questions, push answers nine. What is left unanswered is not the same, and that is the most important line in the table for this lesson. Under pull, Q5, Q6, and Q9 drop out: how many readings the nightly job processed, why it finished late yesterday, and which run was slow. All three depend on the short-lived resource class, and that class is out of coverage. The cost of this is in the last line: when one of the twenty-nine runs that produce no series fails, the only signal that shows it is the morning reconciliation report; the job runs between two and three at night, and the report is not checked until seven — **four and a half hours on average**. When a validation run at the head of the billing chain drops silently, that four and a half hours multiplies directly into the number of records that need correcting. Under push, Q8 and Q9 drop out: which instance restarted and which validation run was slow. Both depend on the reliability of a single label — `instance`. The label technically exists and is written on every record; what is not reliable is the guarantee that the same instance will carry the same value at two different times. This shows that collection can close off a question **without losing any data at all**: the data is not missing, it is unmeasurable. Q12 is unanswered under both models and stays that way for the rest of this lesson. Asking which reading a bill was generated from means asking for a single transaction's path across four services; a metric pipeline cannot carry that path, because carrying it would require turning the transaction ID into a label, and that label's cardinality would equal the transaction count. #### The Metric That Answers No Question The last block counts which portion of the collected data has no payoff. Three of the seven metrics — process start time, garbage-collection pause, and file descriptor count — appear in none of the twelve questions. That is **ten** of the eighty-five series per instance, twelve percent. Under the pull model that is 1.1 megabytes an hour, more than eight hundred megabytes over a thirty-day retention, and it closes not a single question in return. These three metrics are not bad; they are a cost because they have no question of their own. If garbage-collection pause enters a question set tomorrow, the same bytes become meaningful. What justifies collecting something is not the metric itself, but the existence of a question that asks for it. Cost per question lets the two models be compared on the same scale: 1.2 megabytes for pull, 1.6 for push. Push answers more questions but costs thirty percent more per question, because a ten-second send interval is more frequent than a fifteen-second scrape, and identity multiplication spreads the series count across forty identities. The answer to "which is better" is not in a single number; it is in reading these two columns together. #### Summary - The question set consists of twelve questions; each question is tied to a resource class, a metric, and a set of labels. The unit of measure is not the bytes collected, but the questions answered. - Pull-based collection covers 13 of the 42 short-lived runs; because the inventory-refresh (30 s) and scrape (15 s) delays stack on top of each other, a run ends before the round that would have seen it. - Push-based collection covers 42/42 of the short-lived class, but because the process reports its own identity, 40 lifetimes produce 40 separate identities; on the short-lived side, 42 runs merge under 3 identities. - Pull answers 8/12 questions, push answers 9/12; what is left unanswered does not overlap — Q5, Q6, Q9 under pull, Q8, Q9 under push. Q12 (which reading a bill was generated from) is unanswered under both models. - 10/85 series (11.8%) of the collected metrics answer no question; that is 1.1 MB an hour under pull. Cost per question is 1.2 MB under pull and 1.6 MB under push. - The cost of an uncovered run ties to a number: the failure of one of the 29 runs that produce no series surfaces in the morning report an average of 4.5 hours later. #### Next Step This lesson treated the collected data as a box: measurement count, series count, bytes. But the series itself is a structure, and inside that structure sits a decision — which labels exist. Each of the eight or nine questions answered depended on the presence of a specific label; when the `instance` label dropped, two questions closed. The next lesson measures this link in the opposite direction: what question does **adding** a label open, how many series and bytes does it cost, and which label opens no question at all when added. ### Time Series Data Source: https://academia.sh/en/courses/devops-observability/time-series-data What question adding a label opens is counted against the question set: for every label, series, bytes, and the question opened are measured together, marginal bytes per question are derived, a label that opens no question is named, and how much the retention window opens each question for is worked out separately. The previous lesson measured collection like a box: how many measurements, how many series, how many bytes. But each of the eight or nine questions answered depended on the presence of a specific label, and when the `instance` label became unreliable, two questions closed at once. This lesson measures that link in the opposite direction: what question does **adding** a label open, and how many bytes does that question cost to open? Why a label grows the series count multiplicatively rather than additively was measured in the Cluster Observability lesson of the **Container Orchestration** course: a base of ten thousand series rose to 6.8x with a single seven-valued label, and to 28.3x when a four-valued label was added to every metric. That measurement is **not repeated** here; the multiplication rule is this lesson's **input**. What is measured here is something else — **which question opens** in exchange for the multiplier. The series set and the numbers are both **fictional**; what gets measured is a **model** built with `node`. **ML7.** The label set, the cardinalities, and which label applies to which metric are fictional and written in advance. **ML8.** The `instance` label resolves within a service; at most eight instances per service are assumed (the 29 instances from the previous lesson spread across four services). **ML9.** Every question has a backward-looking **horizon**: how many days of data must be looked at for it to be answerable is written in advance. **ML10.** Sampling is 15 seconds, a record is 16 bytes, and a series descriptor is 120 bytes (the same as the previous lesson). **ML11.** For a question to count as opened, **all** the labels it requires must be present in the set; no single label is sufficient on its own. **ML12.** The metric set derives from the previous lesson's metrics; the three metrics that answer no question were counted there and are not carried into this set. #### The Decision Inside the Series A time series consists of a name and a label set, and in storage every distinct combination of that pair is a separate series. The label set is therefore not a storage setting, it is a **question decision**: which breakdown you can perform later is fixed at the moment of collection. ```js // series.mjs — the time series set, label cardinalities, and the label/horizon requirement of questions. export const METRICS = { "request-count": "long", "request-duration": "long", "queue-depth": "long", "processed-reading": "long", "job-step-duration": "short", "job-processed-records": "short" }; export const LABELS = { // name: [cardinality, metrics it applies to] service: [4, ["request-count", "request-duration", "queue-depth", "processed-reading"]], job: [3, ["job-step-duration", "job-processed-records"]], environment: [3, Object.keys(METRICS)], version: [4, Object.keys(METRICS)], instance: [8, Object.keys(METRICS)], result: [5, ["request-count", "processed-reading"]], bucket: [12, ["request-duration"]], queue: [3, ["queue-depth"]], region: [11, ["processed-reading"]], step: [6, ["job-step-duration"]], "node-name": [9, Object.keys(METRICS)], }; // question: [id, required labels, required horizon (days)] export const QUESTIONS = [["Q1", ["service"], 1], ["Q2", ["service", "bucket"], 1], ["Q3", ["environment", "result"], 1], ["Q4", ["version"], 30], ["Q5", ["job"], 7], ["Q6", ["job", "step"], 7], ["Q7", ["region", "result"], 365], ["Q8", ["instance"], 7], ["Q9", ["job", "instance"], 7], ["Q10", ["result"], 1], ["Q11", ["service", "queue"], 1], ["Q12", ["transaction-id"], 1]]; export const seriesCount = (open) => Object.keys(METRICS).reduce((t, m) => t + open.reduce((c, e) => c * (LABELS[e][1].includes(m) ? LABELS[e][0] : 1), 1), 0); export const bytes = (series, days, interval = 15) => series * Math.floor(86400 / interval) * days * 16 + series * 120; export const gb = (b) => (b / 1024 ** 3).toFixed(2); ``` Labels are not spread evenly across metrics. `region` applies only to the processed-reading metric; `environment`, `version`, `instance`, and `node-name` apply to every metric. This split determines the cost: an eleven-valued label added to a single metric can be cheaper than a four-valued label added to every metric. #### The Label Ladder The ladder starts from an empty set — only service and job names — and adds labels one at a time. At every rung, four things are written together: series, bytes, questions answered, and **the marginal byte cost of the question opened at that rung**. ```js // labels.mjs — the question a label opens when added, the question closed by removing it, and the retention window. import { METRICS, LABELS, QUESTIONS, seriesCount, bytes, gb } from "./series.mjs"; const opened = (a) => QUESTIONS.filter(([, e]) => e.every((x) => a.includes(x))).map(([k]) => k); const LADDER = ["environment", "result", "bucket", "queue", "region", "step", "version", "instance", "node-name"]; let open = ["service", "job"], prevQuestions = opened(open), prevBytes = bytes(seriesCount(open), 30); console.log(`${Object.keys(METRICS).length} metrics, 15 s sampling, 30-day retention; base labels ${open.join(" + ")}`); console.log("added label cardinality series GB answered opened question marginal GB/question"); const print = (name, card) => { const s = seriesCount(open), b = bytes(s, 30), y = opened(open); const added = y.filter((q) => !prevQuestions.includes(q)); const marginal = added.length ? gb((b - prevBytes) / added.length) : "-"; console.log(`${name.padEnd(15)}${String(card).padStart(13)}${s.toLocaleString("en-US").padStart(9)}` + `${gb(b).padStart(10)}${String(`${y.length}/12`).padStart(13)} ${(added.join(" ") || "-").padEnd(17)}${marginal.padStart(22)}`); prevQuestions = y; prevBytes = b; }; print("base", "-"); for (const e of LADDER) { open = [...open, e]; print(`+ ${e}`, LABELS[e][0]); } const FULL = open, fullBytes = bytes(seriesCount(FULL), 30); console.log(`\nwhen a label is removed (full set ${FULL.length} labels,` + ` ${seriesCount(FULL).toLocaleString("en-US")} series, ${gb(fullBytes)} GB)`); console.log("removed label gained GB closed question GB/closed question"); for (const e of FULL) { const remaining = FULL.filter((x) => x !== e), gain = fullBytes - bytes(seriesCount(remaining), 30); const closed = opened(FULL).filter((q) => !opened(remaining).includes(q)); console.log(`${e.padEnd(18)}${gb(gain).padStart(13)} ${(closed.join(" ") || "-").padEnd(17)}` + `${(closed.length ? gb(gain / closed.length) : "-").padStart(21)}`); } const PRUNED = FULL.filter((e) => e !== "node-name"); console.log(`\nretention window (pruned set ${PRUNED.length} labels,` + ` ${seriesCount(PRUNED).toLocaleString("en-US")} series)`); console.log("window GB answered in horizon GB/question question the window opens"); let prevHorizon = []; for (const day of [7, 30, 90, 395]) { const y = opened(PRUNED).filter((q) => QUESTIONS.find(([k]) => k === q)[2] <= day); const b = bytes(seriesCount(PRUNED), day); console.log(`${String(`${day} d`).padEnd(9)}${gb(b).padStart(9)}${String(`${y.length}/12`).padStart(24)}` + `${gb(b / y.length).padStart(13)} ${y.filter((q) => !prevHorizon.includes(q)).join(" ") || "-"}`); prevHorizon = y; } ``` ``` 6 metrics, 15 s sampling, 30-day retention; base labels service + job added label cardinality series GB answered opened question marginal GB/question base - 22 0.06 2/12 - - + environment 3 66 0.17 2/12 - - + result 5 162 0.42 4/12 Q3 Q10 0.12 + bucket 12 294 0.76 5/12 Q2 0.34 + queue 3 318 0.82 6/12 Q11 0.06 + region 11 918 2.36 7/12 Q7 1.55 + step 6 963 2.48 8/12 Q6 0.12 + version 4 3,852 9.92 9/12 Q4 7.44 + instance 8 30,816 79.35 11/12 Q8 Q9 34.72 + node-name 9 277,344 714.17 11/12 - - when a label is removed (full set 11 labels, 277,344 series, 714.17 GB) removed label gained GB closed question GB/closed question service 500.59 Q1 Q2 Q11 166.86 job 31.15 Q5 Q6 Q9 10.38 environment 476.11 Q3 476.11 result 427.17 Q3 Q7 Q10 142.39 bucket 97.89 Q2 97.89 queue 17.80 Q11 17.80 region 444.97 Q7 444.97 step 33.37 Q6 33.37 version 535.63 Q4 535.63 instance 624.90 Q8 Q9 312.45 node-name 634.82 - - retention window (pruned set 10 labels, 30,816 series) window GB answered in horizon GB/question question the window opens 7 d 18.52 9/12 2.06 Q1 Q2 Q3 Q5 Q6 Q8 Q9 Q10 Q11 30 d 79.35 10/12 7.94 Q4 90 d 238.05 10/12 23.80 - 395 d 1044.76 11/12 94.98 Q7 ``` The first half of the ladder is cheap. The result label opens two questions for 0.12 gigabytes, the queue label opens one question for 0.06 gigabytes, the step label opens one question for 0.12 gigabytes. These are labels that apply to a single metric or carry low cardinality; the product grows by a small multiplier. The second half is expensive. The version label is four-valued, but because it applies to **every metric** it quadruples the series and opens a single question — which version broke it — for 7.44 gigabytes. The instance label opens two questions, at 34.72 gigabytes per question. Between two different labels that each open two questions in the same set there is a **three-hundred-fold** difference, and the source of that difference is not the importance of the question but how many metrics the label applies to. The `environment` row must be read separately: it opens no question at the rung where it is added. But Q3 requires both the environment and result labels; without environment, the result label alone could not have opened Q3. A label "opening no question" and a label "not being sufficient on its own" are different things, and the ladder cannot distinguish between them. #### Removing a Label, and Pruning The second table is what makes the distinction: every label is **removed** from the full set and the questions that close are counted. When `environment` is removed, Q3 closes — so it does have a share, and that share costs 476 gigabytes. When `node-name` is removed, **no question closes**. The bytes gained are 634.82 gigabytes: eighty-nine percent of the full set. This label's name needs to be written down because it looks harmless. It is nine-valued, short, easy to read, and the collector adds it on its own — the node name is already known. The problem is that none of the twelve questions ask for the node name. Whichever node the pod runs on, what operations asks for is the service, the environment, the version, and the instance. Until a label is tied to a question, it is not information, it is a multiplier. The pruning order also comes from the table. First the label with no payoff is dropped; this single step brings the series from 277,344 down to 30,816. After that, the marginal cost of every remaining label drops to one-ninth — the version label costs 59 gigabytes instead of 535, the instance label costs 69 instead of 624. A label with a payoff often looks expensive not because of itself, but because of the payoff-less label sitting next to it. #### The Product Is an Upper Bound All the numbers on the ladder came out of a product, and a product assumes that every label combination will actually occur. The inventory does not confirm that: the four services do not have equal instance counts, two services never run in the development environment, and only two services emit the processed-reading metric. The actual series set can be counted from the inventory. ```js // actual.mjs — the product is an upper bound: the actual series count and the correction of the measure. import { METRICS, LABELS, QUESTIONS, seriesCount, bytes, gb } from "./series.mjs"; const INSTANCES = { "reading-collector": [8, 2, 1], validator: [6, 2, 1], // production, staging, development billing: [4, 1, 0], "work-order": [3, 1, 0] }; const EMITTERS = { "processed-reading": ["reading-collector", "validator"] }; // others on all four services const RUNNERS = 3 * 2; // three nightly jobs, runner in two environments const PRUNED = ["service", "job", "environment", "version", "instance", "result", "bucket", "queue", "region", "step"]; const BASE = ["service", "job", "environment", "instance"]; // the triple actually counted from inventory const actual = Object.keys(METRICS).reduce((t, m) => { const base = METRICS[m] === "short" ? RUNNERS : (EMITTERS[m] || Object.keys(INSTANCES)).reduce((s, v) => s + INSTANCES[v].reduce((a, b) => a + b, 0), 0); return t + base * PRUNED.filter((e) => !BASE.includes(e) && LABELS[e][1].includes(m)) .reduce((c, e) => c * LABELS[e][0], 1); }, 0); const theoretical = seriesCount(PRUNED), answered = QUESTIONS.filter(([, e]) => e.every((x) => PRUNED.includes(x))).length; console.log("metric actual theoretical overcount"); for (const m of Object.keys(METRICS)) { const base = METRICS[m] === "short" ? RUNNERS : (EMITTERS[m] || Object.keys(INSTANCES)).reduce((s, v) => s + INSTANCES[v].reduce((a, b) => a + b, 0), 0); const multiplier = PRUNED.filter((e) => !BASE.includes(e) && LABELS[e][1].includes(m)) .reduce((c, e) => c * LABELS[e][0], 1); const k = PRUNED.reduce((c, e) => c * (LABELS[e][1].includes(m) ? LABELS[e][0] : 1), 1); console.log(`${m.padEnd(22)}${(base * multiplier).toLocaleString("en-US").padStart(9)}${k.toLocaleString("en-US").padStart(13)}` + `${String(`${(k / (base * multiplier)).toFixed(1)}x`).padStart(14)}`); } console.log(`${"total".padEnd(22)}${actual.toLocaleString("en-US").padStart(9)}` + `${theoretical.toLocaleString("en-US").padStart(13)}${String(`${(theoretical / actual).toFixed(1)}x`).padStart(14)}`); console.log(`\n30-day retention theoretical ${gb(bytes(theoretical, 30))} GB` + ` actual ${gb(bytes(actual, 30))} GB`); console.log(`for ${answered}/12 questions, per question theoretical ${gb(bytes(theoretical, 30) / answered)} GB` + ` actual ${gb(bytes(actual, 30) / answered)} GB`); ``` ``` metric actual theoretical overcount request-count 580 1,920 3.3x request-duration 1,392 4,608 3.3x queue-depth 348 1,152 3.3x processed-reading 4,400 21,120 4.8x job-step-duration 144 1,728 12.0x job-processed-records 24 288 12.0x total 6,888 30,816 4.5x 30-day retention theoretical 79.35 GB actual 17.74 GB for 11/12 questions, per question theoretical 7.21 GB actual 1.61 GB ``` The product overcounts by four and a half times overall, and by twelve times on the nightly-job metrics. The reason differs per metric: of the theoretical ninety-six combinations of the service-environment-instance triple, only twenty-nine exist; only two of the four services emit processed-reading; and the nightly job runs a single runner in two environments rather than eight instances across three environments. This correction changes the decision. A retention discussion that looks at the theoretical number prices thirty days at 79 gigabytes, 7.21 gigabytes per question; the actual set counted from the inventory brings the same thirty days down to 17.74 gigabytes, 1.61 gigabytes per question. The difference is not a saving, it is a **measurement error** — a cost inflated four and a half times leads not to skipping a label that should never have been added, but to skipping a label that is actually needed. #### The Retention Window The label set determines **which breakdown** a question can perform; the retention window determines **how far back** it can look. The third table measures the same pruned set across four windows; the numbers are over the theoretical series set, and in the actual set all of them divide by four and a half. The seven-day window holds eighteen and a half gigabytes and answers nine questions — 2.06 gigabytes per question. Thirty days opens Q4: a version comparison must cover the time both versions stayed in production, and deployment frequency requires thirty days for that. Its cost is 7.94 gigabytes per question. The ninety-day window opens no new question. Three times the bytes, zero questions — the cost per question rises to 23.80 gigabytes. This line is the exact price of a decision that stretches retention on the reasoning that it should be kept a little longer. Three hundred ninety-five days opens a single question: which region's readings did not pass validation, compared against the same period the year before. Its cost is 965 gigabytes of extra retention. The cost of leaving it unanswered can also be counted: when the seasonal comparison cannot be made, which region is chronically problematic cannot be told apart, and the field crew has to close work orders for all **eleven regions** instead of just one. Which number is larger depends on the regional measurement network's own cost structure; what carries the decision is that the two numbers stand in the same table. Q12 appears in none of these tables. Neither adding a label nor extending retention opens it; the question of which reading a bill was generated from cannot be solved with a label, because the breakdown it requires is per transaction. #### Summary - A label's cost is not its cardinality, it is the product of its cardinality and the number of metrics it applies to: the 11-valued `region` opens a question for 1.55 GB, while the 4-valued `version` opens a question for 7.44 GB. - As the ladder climbs from 22 series to 277,344 series, questions answered climb from 2/12 to 11/12; marginal bytes per question range from 0.06 GB to 34.72 GB, a three-hundred-fold spread. - `node-name` opens no question and closes no question; removing it gains 634.82 GB, 89% of the full set. Pruning is the first step; after it, the cost of every other label drops to one-ninth. - A label that opens no question at the rung where it is added is different from a label with no share in any question: `environment` opens no question on its own, but removing it closes Q3. - The product is an upper bound: the actual set counted from the inventory is 6,888 series, a quarter of the theoretical 30,816; cost per question drops from 7.21 GB to 1.61 GB. - The retention window sets the question's horizon: 7 days answers 9/12 questions at 2.06 GB/question, 30 days answers 10/12 at 7.94 GB/question, 395 days answers 11/12 at 94.98 GB/question. The 90-day window opens no new question in exchange for three times the bytes. #### Next Step Up to this point the series set was treated like a warehouse: how many series exist, how many bytes they hold, which breakdown they store. But no question is answered by the series itself; the answer comes out of an **aggregation** — average, percentile, total, rate. The next lesson answers the same question with different aggregations and shows that the answer changes: which aggregation hides what, which detail pre-aggregation erases irrecoverably, and what it buys back in query cost in return. ### Query and Aggregation Source: https://academia.sh/en/courses/devops-observability/query-and-aggregation The same question is answered with four aggregations, and the answer is shown to change: the average hides, the percentile shows, the total misleads, and the count and the rate point to different services. The detail pre-aggregation loses irrecoverably and the query cost it buys back are counted in the same table. The previous lesson measured which breakdown the series set stores. But no question is answered by the series itself: a series is hundreds of thousands of numbers, and the answer must be a single number. **Aggregation** steps in between — average, percentile, max, total — and this lesson shows that this step is not settled: from the same data, for the same question, different aggregations produce **different answers**. The second measure is pre-aggregation. Storing a minute-level summary instead of the raw measurement drops query cost by two orders of magnitude, and erases a detail irrecoverably in return. This lesson's measure is counting both sides in the same table. The measurement is again on the **regional measurement network** and is **fictional**; no real traffic is run, what gets counted is a **model** produced with `node`. **ML13.** The request rate and latency profile of the four services are fictional and written in advance. **ML14.** Measurements are produced with our own generator, seed `20260902`; 90,000 requests in a one-hour window. **ML15.** Six percent of the billing service's requests wait on a lock and take 950–1100 ms of extra latency; the other three services have a smooth distribution. A fictional 90-second degradation is also added to the validator (+900 ms), and the event-free profile values are taken as the threshold baseline. **ML16.** Pre-aggregation keeps four fields per minute: count, total, max, and the minute's p95 value. **ML17.** The overall p95 computed from the summary is the request-count-weighted average of the minute p95s — the assumption that percentiles can be combined is applied deliberately. **ML18.** Duration measurement depends on the environment; only the result of the band check is printed to the output, not the raw measurement. #### The Raw Measurement Set ```js // measurements.mjs — one hour of request measurements produced with our own generator; profiles are fictional. export const makeRng = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648; export const WINDOW = 3600; export const PROFILE = { // service: [req/s, base, spread, tail share, tail ms, error rate] "reading-collector": [12, 150, 110, 0, 0, 0.012], validator: [8, 90, 40, 0, 0, 0.006], billing: [3, 40, 30, 0.06, 950, 0.035], "work-order": [2, 200, 40, 0, 0, 0.009], }; export const measurements = (seed = 20260902) => { const rand = makeRng(seed), o = {}; for (const [service, [rate, base, spread, tailShare, tailMs, errorRate]] of Object.entries(PROFILE)) { const d = []; for (let sec = 0; sec < WINDOW; sec++) for (let i = 0; i < rate; i++) { const queued = rand() < tailShare; d.push({ sec, ms: Math.round(base + rand() * spread + (queued ? tailMs + rand() * 150 : 0)), error: rand() < errorRate }); } o[service] = d; } return o; }; export const percentile = (a, p) => { const s = [...a].sort((x, y) => x - y); return s[Math.min(s.length - 1, Math.ceil(p * s.length) - 1)]; }; export const maxName = (o) => Object.entries(o).sort((a, b) => b[1] - a[1])[0][0]; ``` The profiles are not distinguished by their average alone. The reading collector is high-volume with a smooth distribution; billing is low-volume and has a tail — six percent of its requests finish above one second. These two shapes are the two latency patterns most commonly encountered in real operations, and which aggregation sees the difference between them is not obvious from the start. #### The Same Question, Four Aggregations ```js // query.mjs — the same question is answered with different aggregations; the cost of pre-aggregation is counted. import { PROFILE, WINDOW, measurements, percentile, maxName } from "./measurements.mjs"; const raw = measurements(); const services = Object.keys(PROFILE); const totalRequests = services.reduce((t, s) => t + raw[s].length, 0); console.log(`one hour of measurement (fictional model): ${totalRequests.toLocaleString("en-US")} requests, ${services.length} services`); console.log("\nQ1 'which service slowed down' — answer by aggregation"); console.log("aggregation " + services.map((s, i) => s.padStart([19, 11, 9, 12][i])).join("") + " answer"); const AGGREGATIONS = [["average ms", (d) => d.reduce((t, x) => t + x.ms, 0) / d.length], ["p95 ms", (d) => percentile(d.map((x) => x.ms), 0.95)], ["max ms", (d) => Math.max(...d.map((x) => x.ms))], ["total s", (d) => d.reduce((t, x) => t + x.ms, 0) / 1000]]; for (const [name, f] of AGGREGATIONS) { const v = Object.fromEntries(services.map((s) => [s, f(raw[s])])); console.log(`${name.padEnd(14)}` + services.map((s, i) => v[s].toFixed(0).padStart([19, 11, 9, 12][i])).join("") + ` ${maxName(v)}`); } console.log("\nQ10 'which error code rose' — count or rate"); const errorCount = Object.fromEntries(services.map((s) => [s, raw[s].filter((x) => x.error).length])); const errorRate = Object.fromEntries(services.map((s) => [s, 100 * errorCount[s] / raw[s].length])); console.log("measure " + services.map((s, i) => s.padStart([19, 11, 9, 12][i])).join("") + " answer"); console.log("error count " + services.map((s, i) => String(errorCount[s]).padStart([19, 11, 9, 12][i])).join("") + ` ${maxName(errorCount)}`); console.log("error rate % " + services.map((s, i) => errorRate[s].toFixed(2).padStart([19, 11, 9, 12][i])).join("") + ` ${maxName(errorRate)}`); ``` ``` one hour of measurement (fictional model): 90,000 requests, 4 services Q1 'which service slowed down' — answer by aggregation aggregation reading-collector validator billing work-order answer average ms 204 110 125 220 work-order p95 ms 255 128 1042 238 billing max ms 260 130 1166 240 billing total s 8834 3162 1350 1582 reading-collector Q10 'which error code rose' — count or rate measure reading-collector validator billing work-order answer error count 523 168 428 66 reading-collector error rate % 1.21 0.58 3.96 0.92 billing ``` One question, three separate aggregations that produce four different answers. The numbers all belong to the same 90,000 requests, and none of them is wrong. **The average hides.** Billing's average is 125 milliseconds, the second lowest of the four services. The six percent slice waiting in the tail pulls the average up by only seventy milliseconds, because the average divides extreme values across the whole volume. A dashboard that looks at the average never shows billing at all. **The percentile shows.** The same service's p95 is 1042 milliseconds — between four and eight times the others. The tail cannot hide in the p95, because the six percent slice is above the five percent threshold. If one request in twenty crosses one second, that shows up only in the percentile. **The total misleads.** Looking at the total duration, the answer is the reading collector: 8834 seconds. But what this number says is not latency, it is **volume** — the reading collector takes 43,200 requests an hour, billing takes 10,800. The total measures workload when what is wanted is speed. It is correct for a capacity question and misleading for a latency question. The error table sets up the same split a second time. Error **count** points to the reading collector (523), error **rate** points to billing (3.96%). Both are correct, and they answer two different questions: "which service produces the most errors" versus "which service is the most broken." When the on-call person is shown the wrong one, the investigation starts in the logs of the service whose error rate is three times lower. #### The Query Window Does the Same Thing Aggregation is not only which function is applied, it is also **which interval** it is applied to. The same average measures different things when applied to a minute versus an hour. The block below places a short degradation on the validator and asks the same two aggregations across four windows. ```js // window.mjs — when the query window changes, the same question gives a different answer. import { PROFILE, WINDOW, measurements, percentile } from "./measurements.mjs"; const raw = measurements(); const EVENT = { service: "validator", start: 1500, duration: 90, extra: 900 }; // fictional degradation for (const x of raw[EVENT.service]) if (x.sec >= EVENT.start && x.sec < EVENT.start + EVENT.duration) x.ms += EVENT.extra; const d = raw[EVENT.service]; const baseline = { avg: 110, p95: 128 }; // values of the event-free profile console.log(`${EVENT.service} service has a ${EVENT.duration} s degradation of +${EVENT.extra} ms` + ` (${(100 * EVENT.duration / WINDOW).toFixed(1)}% of the hour); threshold is twice the baseline value`); console.log("query window windows highest avg ms over threshold highest p95 ms over threshold time to notice"); for (const w of [60, 300, 900, 3600]) { const windowCount = WINDOW / w, avg = [], p95 = []; for (let i = 0; i < windowCount; i++) { const g = d.filter((x) => x.sec >= i * w && x.sec < (i + 1) * w).map((x) => x.ms); avg.push(g.reduce((a, b) => a + b, 0) / g.length); p95.push(percentile(g, 0.95)); } console.log(`${String(`${w / 60} min`).padEnd(14)}${String(windowCount).padStart(9)}` + `${Math.max(...avg).toFixed(0).padStart(17)}${String(`${avg.filter((v) => v > 2 * baseline.avg).length}/${windowCount}`).padStart(16)}` + `${String(Math.max(...p95)).padStart(16)}${String(`${p95.filter((v) => v > 2 * baseline.p95).length}/${windowCount}`).padStart(16)}` + `${String(`${w / 60} min`).padStart(16)}`); } console.log(`\nrecords per resolution (one hour, four services): ` + [60, 300, 900, 3600].map((w) => `${w / 60} min ${4 * (WINDOW / w) * 4}`).join(" ")); ``` ``` validator service has a 90 s degradation of +900 ms (2.5% of the hour); threshold is twice the baseline value query window windows highest avg ms over threshold highest p95 ms over threshold time to notice 1 min 60 1009 2/60 1028 2/60 1 min 5 min 12 380 1/12 1023 1/12 5 min 15 min 4 200 0/4 1008 1/4 15 min 60 min 1 132 0/1 129 0/1 60 min records per resolution (one hour, four services): 1 min 960 5 min 192 15 min 64 60 min 16 ``` The degradation is two and a half percent of the hour, and it looks four different ways across four windows. In the one-minute window, both the average and the percentile rise above one thousand milliseconds, and two windows cross the threshold. At five minutes the average drops to 380 but is still above the threshold. **At fifteen minutes the average dies**: the highest value is 200 milliseconds, which stays below the two-hundred-twenty threshold — no window is flagged. In the same window the percentile still sees 1008 milliseconds and flags one window. At one hour both vanish: 132 milliseconds for the average, 129 for the percentile — no trace of the event remains. The record loss and its cost sit in the same line. One-minute resolution holds 960 records an hour, one-hour resolution holds 16 — a sixty-fold difference. Time to notice is the window itself: a window cannot be queried before it closes, so fifteen-minute resolution means a fifteen-minute delay at best. If the degradation lasts ninety seconds, that fifteen minutes falls entirely after the event has already ended. #### The Two Faces of Pre-Aggregation Storing the raw measurement per request is expensive. The common response is to keep a minute-level summary: count, total, max, and the minute's p95 value for every minute. The block below actually produces the summary and asks the same questions of both stores. ```js // preagg.mjs — the detail pre-aggregation loses and the query cost it buys back. import { PROFILE, measurements, percentile } from "./measurements.mjs"; const raw = measurements(), services = Object.keys(PROFILE); const summary = {}; // minute summary: count, total, max, minute p95 for (const s of services) { summary[s] = Array.from({ length: 60 }, (_, d) => { const g = raw[s].filter((x) => Math.floor(x.sec / 60) === d).map((x) => x.ms); return { count: g.length, total: g.reduce((a, b) => a + b, 0), max: Math.max(...g), p95: percentile(g, 0.95) }; }); } const rawAll = services.flatMap((s) => raw[s].map((x) => x.ms)); const rawRecords = rawAll.length, summaryRecords = services.length * 60 * 4; const rawP95 = percentile(rawAll, 0.95); const summaryP95 = services.reduce((t, s) => t + summary[s].reduce((a, d) => a + d.p95 * d.count, 0), 0) / rawRecords; const rawAvg = rawAll.reduce((a, b) => a + b, 0) / rawRecords; const summaryAvg = services.reduce((t, s) => t + summary[s].reduce((a, d) => a + d.total, 0), 0) / rawRecords; const deviation = (o, h) => `${(100 * Math.abs(o - h) / h).toFixed(1)}%`; console.log(`raw ${rawRecords.toLocaleString("en-US")} records, minute summary ${summaryRecords} records` + ` (${(rawRecords / summaryRecords).toFixed(0)}x smaller)`); console.log("\nquestion raw answer summary answer deviation"); const ROWS = [["overall average ms", rawAvg.toFixed(1), summaryAvg.toFixed(1), deviation(summaryAvg, rawAvg)], ["overall p95 ms", String(rawP95), summaryP95.toFixed(1), deviation(summaryP95, rawP95)], ["max ms", String(Math.max(...rawAll)), String(Math.max(...services.flatMap((s) => summary[s].map((d) => d.max)))), "0.0%"], ["requests over 1 s", String(rawAll.filter((x) => x > 1000).length), "cannot be computed", "-"]]; for (const [a, h, o, s] of ROWS) console.log(`${a.padEnd(26)}${h.padStart(12)}${o.padStart(21)}${s.padStart(11)}`); // query cost: the same p95 query is run against both stores REPEATS times const REPEATS = 40; const expectedRatio = rawRecords / summaryRecords; const measure = (f) => { const b = process.hrtime.bigint(); for (let i = 0; i < REPEATS; i++) f(); return Number(process.hrtime.bigint() - b); }; const rawTime = measure(() => services.map((s) => percentile(raw[s].map((x) => x.ms), 0.95))); const summaryTime = measure(() => services.map((s) => percentile(summary[s].map((d) => d.p95), 0.95))); const measuredRatio = rawTime / summaryTime; console.log(`\np95 query run ${REPEATS} times`); console.log(` records scanned raw ${(rawRecords * REPEATS).toLocaleString("en-US")}` + ` summary ${(summaryRecords * REPEATS).toLocaleString("en-US")} expected ratio ${expectedRatio.toFixed(0)}x`); console.log(` is the measured time ratio between 5 and 20000x: ${measuredRatio > 5 && measuredRatio < 20000 ? "yes" : "no"}`); ``` ``` raw 90,000 records, minute summary 960 records (94x smaller) question raw answer summary answer deviation overall average ms 165.9 165.9 0.0% overall p95 ms 251 288.1 14.8% max ms 1166 1166 0.0% requests over 1 s 729 cannot be computed - p95 query run 40 times records scanned raw 3,600,000 summary 38,400 expected ratio 94x is the measured time ratio between 5 and 20000x: yes ``` The table lays three separate behaviors side by side. The average is **preserved exactly**: the deviation is zero, because the average can be reconstructed from the total and the count — the sum of the minute totals is the hour's total. The max is also preserved, for the same reason: the max of the maxes is the real max. The percentile is not preserved. The overall p95 is 251 in the raw data and 288 milliseconds from the summary; the deviation is 14.8%. The reason is one sentence: **the average of percentiles is not a percentile.** The minute p95 values are the tail of each minute's own distribution; weighting them by request count and summing them does not give the tail of the hour's distribution. The direction is not a coincidence, either — in this data set the summary skews upward, because billing's tail pulls every minute's p95 up, and those minutes are not diluted in the total. In a different data set the same method could skew downward; the only thing that can be predicted is that a deviation exists. The last line is a total loss. The raw answer to "how many requests crossed one second" is 729; the summary has no answer, and not even an approximate one, because the count of requests above the threshold sits in none of the summary's four fields. The detail has been erased, and it cannot be recovered. Nothing makes this visible, either: when this query is written against the summary, the store does not raise an error, it just becomes clear that the query cannot be written. The gain side is in the same table too. The same p95 query scans 3.6 million records in the raw store, 38,400 in the summary store — **94 times** fewer. Duration measurement depends on the environment and is not printed here; whether the measured ratio matches the expected order of magnitude is run through a band check, and the result is written as `yes`. The decision is a collision between two numbers: a ninety-four-times cheaper query, a percentile skewed by fifteen percent, and a threshold question that is completely lost. The summary is close to free for dashboards that ask for the average and the max; it is the wrong foundation for alert rules that ask for the percentile and the threshold. #### Summary - The same question gives three different answers across four aggregations: the average points to `work-order`, the p95 and max point to `billing`, the total points to `reading-collector`. None is wrong; each measures something different. - The average hides because it divides the extreme slice across the volume (billing 125 ms), the percentile shows it (1042 ms); the total measures volume instead of latency. - Error count and error rate point to different services: 523 errors for `reading-collector`, a 3.96% rate for `billing`. - The query window is also an aggregation: a 90-second degradation shows up in both the average and the percentile at a 1-minute window, only in the percentile at 15 minutes, and in neither at 60 minutes. As resolution drops from 960 records to 16, time to notice rises from 1 minute to 60 minutes. - Pre-aggregation preserves the average and the max exactly (0.0%), skews the p95 by 14.8%, and erases the count of over-threshold requests entirely — 729 requests cannot be computed from the summary. - The same query scans 3.6 million records in the raw store, 38,400 in the summary store: 94 times cheaper. The gain and the loss are two faces of the same decision. #### Next Step Up to this point every answer was a number: how many milliseconds, how many requests, how many errors. But no number in the table answers the question of **why**. It is known that billing's p95 is 1042 milliseconds; it is not known which lock, which query, or which region's reading it was waiting on. That information does not live in a metric, it lives in the text the process writes itself. The next lesson takes on the log stream and counts, layer by layer, how many lines of that text are lost as it passes through the collection pipeline. ### Log Collection Source: https://academia.sh/en/courses/devops-observability/log-collection The log stream is run through the agent, buffer, and indexing layers, and the records lost at every layer are counted: buffer overflow, truncation, and indexing delay. When the load doubles, which layer breaks first is measured, and the question the loss leaves unanswered is written down together with the cost per question. Every answer in the previous lesson was a number: how many milliseconds, how many requests, how many errors. But no number answers the question of **why** — it is known that billing's p95 is 1042 milliseconds, it is not known which lock it was waiting on. That information does not live in a metric, it lives in the text the process writes itself. A log line is a different kind of thing from a metric: not a predefined series, but text written at the moment of the event, with unbounded cardinality. That is where its cost comes from. A metric pipeline carries a few thousand numbers a second; a log pipeline carries tens of thousands of lines a second. This lesson splits that pipeline into three layers — the **log agent**, the **buffer**, and the **indexer** — and counts how many records are lost at each layer. The stream is the **regional measurement network's** own stream and is **fictional**; no real log store is run, what gets counted is a **model** run second by second with `node`. **ML19.** Line rates, the mix, and the layer limits are fictional and written in advance; there is no randomness in the simulation, the same input gives the same number on every run. **ML20.** The log agent reads 1400 lines a second and holds a local buffer of 30,000 lines; the queue holds 300,000 lines, and the indexer processes 1000 lines a second. **ML21.** Truncation happens at 512 bytes. An error line is 620 bytes, and its last 108 bytes hold the error code and the transaction ID the process generated. **ML22.** When the buffer fills, the type of the dropped lines matches that second's composition; the queue makes no distinction by type. **ML23.** If a line is indexed more than 60 seconds after it was written, it is counted as "late-visible" and is treated as absent for a question asked at event time. **ML24.** A question is answered if the loss on the line type it requires is under the accepted margin: 10% for rate questions, 1% for count questions. Q11 is never answered at any load because its metered value is not written to the log, and Q12 is never answered because there is no shared identity across the chain of services. #### Three Layers and Their Limits ```js // stream.mjs — the log stream is run through the agent, buffer, and indexing layers; loss is counted at every layer. export const BASE = 232, MIX = { request: 0.86, error: 0.02, "job-step": 0.12 }; export const BYTES = { request: 180, error: 620, "job-step": 240 }; export const JOB_START = 600, JOB_END = 1200, JOB_EXTRA = 900; // nightly batch window, all job-step export const LIMITS = { agentRead: 1400, agentBuffer: 30000, queue: 300000, indexer: 1000, truncate: 512, visibility: 60 }; export const WINDOW = 3600; export const run = (load, s = LIMITS) => { let buffer = 0, queue = 0; const d = { produced: 0, agentDropped: 0, queueDropped: 0, indexed: 0, lateVisible: 0, maxDelay: 0, type: {}, droppedType: {} }; for (const t of Object.keys(BYTES)) { d.type[t] = 0; d.droppedType[t] = 0; } for (let sec = 0; sec < WINDOW; sec++) { const input = {}; // this second's line composition for (const t of Object.keys(BYTES)) input[t] = load * BASE * MIX[t]; if (sec >= JOB_START && sec < JOB_END) input["job-step"] += load * JOB_EXTRA; const total = Object.values(input).reduce((a, b) => a + b, 0); for (const t of Object.keys(BYTES)) d.type[t] += input[t]; d.produced += total; buffer += total; // layer 1: agent const read = Math.min(buffer, s.agentRead); buffer -= read; if (buffer > s.agentBuffer) { const dropped = buffer - s.agentBuffer; // dropped lines carry this second's composition for (const t of Object.keys(BYTES)) d.droppedType[t] += dropped * input[t] / total; d.agentDropped += dropped; buffer = s.agentBuffer; } queue += read; // layer 2: buffer queue if (queue > s.queue) { // queue makes no type distinction, loss spreads over all types const dropped = queue - s.queue; for (const t of Object.keys(BYTES)) d.droppedType[t] += dropped * input[t] / total; d.queueDropped += dropped; queue = s.queue; } const written = Math.min(queue, s.indexer); // layer 3: indexer queue -= written; d.indexed += written; const delay = buffer / s.agentRead + queue / s.indexer; d.maxDelay = Math.max(d.maxDelay, delay); if (delay > s.visibility) d.lateVisible += written; } return d; }; ``` The three layers break in three distinct ways, and that distinction is what matters for this lesson. The agent and the queue **drop lines** — the record ceases to exist. The indexer does not drop lines, it **delays** them — the record arrives eventually, but it is not there at the moment the question is asked. Truncation neither drops nor delays; it leaves the line where it is and **cuts off its tail**. #### Layer-by-Layer Loss at Two Loads ```js // layers.mjs — layer-by-layer loss at two loads, and the question the loss leaves unanswered. import { BASE, MIX, BYTES, JOB_START, JOB_END, JOB_EXTRA, LIMITS, WINDOW, run } from "./stream.mjs"; const fmt = (x) => Math.round(x).toLocaleString("en-US"); console.log(`log stream (fictional, second by second): ${WINDOW} s, base ${BASE} lines/s` + ` (${Object.entries(MIX).map(([t, p]) => `${t} ${p * 100}%`).join(", ")})`); console.log(`nightly batch job ${JOB_START}-${JOB_END} s window, +${JOB_EXTRA} lines/s, all job-step`); console.log(`limits: agent read ${LIMITS.agentRead}/s buffer ${fmt(LIMITS.agentBuffer)};` + ` queue ${fmt(LIMITS.queue)}; indexer ${LIMITS.indexer}/s; truncate ${LIMITS.truncate} bytes;` + ` visibility threshold ${LIMITS.visibility} s\n`); console.log("load produced agent dropped queue dropped truncated max delay late-visible loss %"); const results = {}; for (const load of [1, 2]) { const d = run(load); const truncated = Object.entries(d.type).reduce((t, [type, n]) => t + (BYTES[type] > LIMITS.truncate ? n : 0), 0); results[load] = { ...d, truncated }; console.log(`${`${load}x`.padEnd(6)}${fmt(d.produced).padStart(9)}${fmt(d.agentDropped).padStart(14)}` + `${fmt(d.queueDropped).padStart(14)}${fmt(truncated).padStart(10)}` + `${`${d.maxDelay.toFixed(0)} s`.padStart(15)}${fmt(d.lateVisible).padStart(13)}` + `${`${(100 * d.agentDropped / d.produced).toFixed(1)}%`.padStart(9)}`); } console.log("\nthe type of loss and the question it leaves unanswered"); console.log("loss affected lines question left unanswered"); const double = results[2], ROWS = [ ["agent buffer filled (2x)", `job-step ${fmt(double.droppedType["job-step"])} lines` + ` (${(100 * double.droppedType["job-step"] / double.type["job-step"]).toFixed(1)}%)`, "Q5 Q6"], ["truncation (both loads)", `last ${BYTES.error - LIMITS.truncate} bytes of error lines (100%)`, "Q10 Q12"], ["indexing delay (2x)", `${fmt(double.lateVisible)} lines became visible more than ${LIMITS.visibility} s late`, "every question asked at event time"]]; for (const [a, b, c] of ROWS) console.log(`${a.padEnd(26)}${b.padEnd(50)}${c}`); // against the question set: [id, line type, does the field truncate in the queue, accepted loss share] const QUESTIONS_LOG = [["Q1", "request", false, 0.10], ["Q2", "request", false, 0.10], ["Q3", "request", false, 0.10], ["Q4", "request", false, 0.10], ["Q5", "job-step", false, 0.01], ["Q6", "job-step", false, 0.01], ["Q7", "request", false, 0.10], ["Q8", "request", false, 0.10], ["Q9", "job-step", false, 0.01], ["Q10", "error", true, 0.10]]; const mb = (b) => (b / 1024 ** 2).toFixed(0); console.log("\nagainst the question set (Q11's value is never written to the log, Q12 has no identity across the chain)"); console.log("load stored MB lost MB request loss job-step loss answered MB/question unanswered"); for (const load of [1, 2]) { const d = results[load], lossRate = (k) => d.droppedType[k] / d.type[k]; const produced = Object.keys(BYTES).reduce((t, k) => t + d.type[k] * BYTES[k], 0); const stored = Object.keys(BYTES).reduce((t, k) => t + (d.type[k] - d.droppedType[k]) * Math.min(BYTES[k], LIMITS.truncate), 0); const answered = QUESTIONS_LOG.filter(([, type, inQueue, tolerance]) => lossRate(type) <= tolerance && !(inQueue && BYTES[type] > LIMITS.truncate)); console.log(`${`${load}x`.padEnd(6)}${mb(stored).padStart(11)}${mb(produced - stored).padStart(9)}` + `${`${(100 * lossRate("request")).toFixed(1)}%`.padStart(14)}${`${(100 * lossRate("job-step")).toFixed(1)}%`.padStart(15)}` + `${`${answered.length}/12`.padStart(11)}${mb(stored / answered.length).padStart(13)} ` + [...QUESTIONS_LOG.filter((s) => !answered.includes(s)).map(([k]) => k), "Q11", "Q12"].join(" ")); } ``` ``` log stream (fictional, second by second): 3600 s, base 232 lines/s (request 86%, error 2%, job-step 12%) nightly batch job 600-1200 s window, +900 lines/s, all job-step limits: agent read 1400/s buffer 30,000; queue 300,000; indexer 1000/s; truncate 512 bytes; visibility threshold 60 s load produced agent dropped queue dropped truncated max delay late-visible loss % 1x 1,375,200 0 0 16,704 79 s 170,000 0.0% 2x 2,750,400 488,400 0 33,408 261 s 895,000 17.8% the type of loss and the question it leaves unanswered loss affected lines question left unanswered agent buffer filled (2x) job-step 400,315 lines (31.3%) Q5 Q6 truncation (both loads) last 108 bytes of error lines (100%) Q10 Q12 indexing delay (2x) 895,000 lines became visible more than 60 s late every question asked at event time against the question set (Q11's value is never written to the log, Q12 has no identity across the chain) load stored MB lost MB request loss job-step loss answered MB/question unanswered 1x 278 2 0.0% 0.0% 9/12 31 Q10 Q11 Q12 2x 449 111 6.0% 31.3% 6/12 75 Q5 Q6 Q9 Q10 Q11 Q12 ``` At single-fold load, not a single line drops. The agent can read 1400 lines a second, and the peak rate is 1132; the queue does not even come close to its capacity of three hundred thousand lines. But the pipeline still does not run cleanly: the indexer processes 1000 lines a second, and during the batch job window the input rises above that. The buildup stays in the queue, there is no loss — only a **delay**. The maximum delay is 79 seconds, and 170,000 lines are indexed above the sixty-second visibility threshold. Those lines have not been lost; they are not there at the moment they are asked for. #### The Silence of Truncation Truncation does the same thing at both loads, and it is the smallest-looking number in the table: two megabytes at single-fold load. But the share of lines it affects is one hundred percent. An error line is 620 bytes, the truncation limit is 512; the last 108 bytes of every error line are cut off, and two fields sit there — the error code and the transaction ID the process generated. The shape of the loss also matters. A dropped line makes itself known: counters do not add up, the line sequence skips. A truncated line **gives nothing away**. The store holds 16,704 error lines, all of them readable, all of them with their timestamp and service name intact. Only the error code field is empty, and it is empty not because the error code was never written, but because the line was cut off. The two cases cannot be told apart on the query side; both return the same empty cell. The cost is Q10: the question of which error code rose cannot be answered from the log. In the previous lesson the same question was answered from a metric — the `result` label was five-valued, and the error rate could be computed. But the `result` label on the metric splits into five classes; the error code in the log carries hundreds of distinct values, and that is what actually narrows the investigation. #### When the Load Doubles The second row answers the question: **the agent breaks first.** The peak rate rises to 2264 lines a second, the agent's read rate is 1400, and the 864 lines a second in between build up in the buffer; the thirty-thousand-line buffer fills in thirty-five seconds, and for the remaining five hundred sixty-five seconds the excess drops — 488,400 lines, 17.8% of what was produced. The queue never breaks, because the agent is already not passing the excess through. This is the characteristic behavior of a layered pipeline: **the narrowest layer protects the others**, and because it protects them, the others' limits are never tested. Increasing queue capacity does not gain this pipeline a single line. The distribution of the dropped lines is not a coincidence either. The moment the buffer overflows is the batch job window, and in that window eighty-two percent of the input is job-step lines; of the 488,400 dropped lines, 400,315 are job-step — 31.3% of all job-step lines. How many readings the nightly job processed and why it finished late can no longer be asked from the log. But the loss does not stop there. There is no rule choosing which line drops when the buffer overflows; six percent of request lines drop too. Request lines that have nothing to do with the batch job go missing only because they pass through the same buffer. One layer overflowing corrupts questions that had nothing to do with causing the overflow — for count questions, six percent is already more than enough; for rate questions, the result is not wrong, but it is **skewed**, and the skew is invisible. The last table writes this course's rule in its clearest form. When the load doubles, the stored data rises from 278 megabytes to 449 megabytes — a sixty-one percent increase. Questions answered drop from 9 to 6. Cost per question rises from 31 megabytes to 75, **two and a half times**. More data was collected, fewer questions were answered; the two are faces of the same event. #### Which Limit Change Brings Which Question Back Now that the layer that breaks is known, the question is this: how many questions does widening a single limit bring back? ```js // mitigation.mjs — single limit changes at 2x load: which question comes back, at how many megabytes. import { BYTES, LIMITS, run } from "./stream.mjs"; const QUESTIONS_LOG = [["Q1", "request", false, 0.10], ["Q2", "request", false, 0.10], ["Q3", "request", false, 0.10], ["Q4", "request", false, 0.10], ["Q5", "job-step", false, 0.01], ["Q6", "job-step", false, 0.01], ["Q7", "request", false, 0.10], ["Q8", "request", false, 0.10], ["Q9", "job-step", false, 0.01], ["Q10", "error", true, 0.10]]; const mb = (b) => (b / 1024 ** 2).toFixed(0), fmt = (x) => Math.round(x).toLocaleString("en-US"); const CHANGES = [["no change", {}], ["buffer 30k -> 300k", { agentBuffer: 300000 }], ["agent read 1400 -> 2400", { agentRead: 2400 }], ["indexer 1000 -> 2400", { indexer: 2400 }], ["truncate 512 -> 1024 bytes", { truncate: 1024 }], ["agent 2400 + indexer 2400", { agentRead: 2400, indexer: 2400 }]]; console.log("single limit changes at 2x load"); console.log("change agent dropped queue dropped delay stored MB answered MB/question"); for (const [name, changes] of CHANGES) { const limits = { ...LIMITS, ...changes }, d = run(2, limits), lossRate = (k) => d.droppedType[k] / d.type[k]; const stored = Object.keys(BYTES).reduce((t, k) => t + (d.type[k] - d.droppedType[k]) * Math.min(BYTES[k], limits.truncate), 0); const answered = QUESTIONS_LOG.filter(([, type, inQueue, tolerance]) => lossRate(type) <= tolerance && !(inQueue && BYTES[type] > limits.truncate)); console.log(`${name.padEnd(31)}${fmt(d.agentDropped).padStart(15)}${fmt(d.queueDropped).padStart(15)}` + `${`${d.maxDelay.toFixed(0)} s`.padStart(9)}${mb(stored).padStart(13)}` + `${`${answered.length}/12`.padStart(12)}${mb(stored / answered.length).padStart(13)}`); } ``` ``` single limit changes at 2x load change agent dropped queue dropped delay stored MB answered MB/question no change 488,400 0 261 s 449 6/12 75 buffer 30k -> 300k 218,400 69,000 454 s 495 6/12 83 agent read 1400 -> 2400 0 459,400 299 s 455 6/12 76 indexer 1000 -> 2400 488,400 0 21 s 449 6/12 75 truncate 512 -> 1024 bytes 488,400 0 261 s 452 7/12 65 agent 2400 + indexer 2400 0 0 0 s 556 9/12 62 ``` Three of the four single steps bring back no question. Multiplying the buffer by ten cuts the agent's loss in half, but it **moves the loss to the queue** and raises the delay from 261 seconds to 454: the buffer does not prevent loss, it delays it, and a delay is a loss too. Raising the agent's read rate zeroes out the agent's loss, but in exchange 459,400 lines drop in the queue — the total loss is nearly the same, only the layer has changed. Speeding up the indexer alone saves not a single line; it only cuts the delay from 261 seconds to 21, which is a real gain, but on a different line item. The one cheap step that brings a question back is the truncation limit: raising it from 512 to 1024 bytes costs three extra megabytes and brings back Q10 — cost per question drops from 75 megabytes to 65. The only way to bring the other questions back is to change two limits **together**: loss zeroes out, delay drops to zero, questions answered rises to 9/12, and cost per question falls to 62 megabytes — the lowest of the six rows. The pipeline's bottleneck is not in a single layer, it is in the ratio between two layers. #### Summary - The three layers break in three distinct ways: the agent and queue drop lines, the indexer delays them, truncation leaves the line and cuts off its tail. - At single-fold load no line drops, but the indexing delay rises to 79 seconds and 170,000 lines become visible past the 60-second visibility threshold — data left unanswered without being lost. - Truncation is independent of load and erases the last 108 bytes on one hundred percent of error lines; an empty field and a truncated field cannot be told apart in a query, and Q10 stays unanswered. - When the load doubles, the **agent** breaks first: 488,400 lines drop (17.8%), the queue never overflows — the narrowest layer never lets the others' limits be tested. - The overflow is not selective: 400,315 of the dropped lines are job-step (31.3%), but 6% of request lines drop too; questions unrelated to the batch job get corrupted as well. - As stored data rises from 278 to 449 MB, questions answered drop from 9/12 to 6/12; cost per question rises from 31 MB to 75 MB. Four of the single limit increases bring back no question; the buffer moves the loss to the queue, agent speed just moves the loss to a different layer, and the solution is in the ratio between two limits. #### Next Step Up to this point the log line was counted as a whole: how many lines produced, how many lines dropped, how many bytes stored. But the line that reaches the store is still **text**, and very few questions can be answered on text alone — you can search for a string, you can count how many lines it appears in; you cannot ask which service's error rate rose in which environment. The next lesson parses real log lines and turns them into searchable fields: how many questions unstructured text answers, how many questions become answerable once fields are produced, and what the line the parsing rule silently misses costs. ### Log Parsing and Structuring Source: https://academia.sh/en/courses/devops-observability/log-parsing-and-structuring Real log lines are parsed and turned into searchable fields, and two store formats are counted against the question set: how many questions unstructured text answers, how many questions open up once fields are produced. When the format changes, the line the parsing rule silently misses is counted, and the cost of the wrong answer that results is written down. The previous lesson counted the line as a whole: how many lines produced, how many lines dropped, how many bytes stored. But the line that reaches the store is still **text**. On text, you can search for a string and count how many lines it appears in; you cannot ask which region's readings failed validation in which environment, because that requires the line to be broken into parts. This lesson measures that step. Real lines are run through a parsing rule, turned into searchable fields, and two store formats — raw text and fields — are counted against the same question set. The third measure is the line the rule **misses**: when a service changes format, fields stay silently empty, and a query returns the wrong answer without raising an error. The lines come from the **regional measurement network's** nightly validation stream and are **fictional**; the ground truth for every line is kept separately, and the measurement is made against it. **ML25.** The 900 lines of the validation stream are produced with our own generator, seed `20260903`; the structured ground truth for every line is stored separately, and correctness is checked against it. **ML26.** The validator service switches to a new version after the quarter mark of the lines and changes its field names: `zone` instead of `region`, `status` instead of `result`, `duration_ms=…` instead of `duration=…ms`. **ML27.** The parsing rule is written against the old format and is not changed after the version switch. **ML28.** Most of region 07's failed readings are written by the validator; region 13's failures are written by the other services. **ML29.** Unstructured text can only count a single string known in advance; grouping and numeric operations require a field. **ML30.** A field must be filled in at least 80% of lines to be usable in a grouping or numeric question. #### Real Log Lines ```js // lines.mjs — real log lines are produced; each line's ground truth is kept separately. export const makeRng = (t) => () => (t = (t * 1103515245 + 12345) % 2147483648) / 2147483648; export const REGION = ["01", "03", "05", "07", "09", "11", "13", "15", "17", "19", "21"]; export const DROP = { "07": [0.80, 0.10], "13": [0.12, 0.26] }; export const SERVICE = ["reading-collector", "validator", "billing", "work-order"]; const two = (n) => String(n).padStart(2, "0"); // the validator changes its field names after switching to the new version: region->zone, result->status export const generate = (count = 900, seed = 20260903) => { const rand = makeRng(seed), lines = [], records = []; for (let i = 0; i < count; i++) { const service = SERVICE[Math.floor(rand() * SERVICE.length)]; const newFormat = service === "validator" && i > count * 0.25; // version switch const r = REGION[Math.floor(rand() * REGION.length)]; // drop rate by region: [at the validator, at the other services] const [dv, do_] = DROP[r] ?? [0.10, 0.07]; const failed = rand() < (service === "validator" ? dv : do_); const duration = 60 + Math.floor(rand() * 320), instance = `u${two(Math.floor(rand() * 29))}`; const time = `2026-08-02T02:${two(Math.floor(i / 15))}:${two(i % 60)}Z`; records.push({ service, region: r, result: failed ? "failed" : "passed", duration, instance, newFormat }); lines.push(newFormat ? `${time} environment=production service=${service} instance=${instance} level=info event=reading` + ` zone=${r} status=${failed ? "failed" : "passed"} duration_ms=${duration}` : `${time} production ${service}[${instance}] INFO reading processed region=${r}` + ` result=${failed ? "failed" : "passed"} duration=${duration}ms`); } return { lines, records }; }; ``` Both formats carry the same information; the only difference is in the names. This is the most common outcome of a logging convention change, and it is not an error for either service — both lines are valid. #### Parsing and the Missed Line ```js // parse.mjs — unstructured text versus searchable fields, compared against the question set. import { generate, SERVICE } from "./lines.mjs"; const { lines, records } = generate(); console.log(`log lines produced: ${lines.length}`); console.log("sample lines"); for (const i of [0, 1, 227, 232]) console.log(` ${lines[i]}`); // parsing rule written against the old format const RULE = /^(\S+) (\w+) ([\w-]+)\[(\w+)\] (\w+) (.*?) region=(\d+) result=(\w+) duration=(\d+)ms$/; const fields = lines.map((s) => { const m = RULE.exec(s); return m ? { time: m[1], environment: m[2], service: m[3], instance: m[4], region: m[7], result: m[8], duration: +m[9] } : { time: s.slice(0, 20).trim(), environment: "", service: "", instance: "", region: "", result: "", duration: null }; }); const matched = fields.filter((a) => a.region).length; console.log(`\nparsing rule matched ${matched}/${lines.length} lines;` + ` ${lines.length - matched} lines have silently empty fields (${(100 * (lines.length - matched) / lines.length).toFixed(1)}%)`); // Q7: which region's readings were not validated const tally = (source, regionOf, resultOf) => { const t = {}; for (const k of source) { const r = regionOf(k); if (!r) continue; t[r] ??= [0, 0]; t[r][0]++; if (resultOf(k) === "failed") t[r][1]++; } return t; }; const worst = (t) => Object.entries(t).sort((a, b) => b[1][1] - a[1][1])[0]; const actual = tally(records, (k) => k.region, (k) => k.result), parsed = tally(fields, (a) => a.region, (a) => a.result); const [actualRegion, actualCounts] = worst(actual), [parsedRegion, parsedCounts] = worst(parsed); console.log("\nQ7 'which region's readings were not validated'"); console.log(` actual region ${actualRegion} ${actualCounts[1]} failed / ${actualCounts[0]} readings`); console.log(` from parsing region ${parsedRegion} ${parsedCounts[1]} failed / ${parsedCounts[0]} readings` + ` deviation ${(100 * Math.abs(parsedCounts[1] - actualCounts[1]) / actualCounts[1]).toFixed(1)}% same answer: ${actualRegion === parsedRegion ? "yes" : "no"}`); console.log(` the rule never saw ${actual[actualRegion][1] - (parsed[actualRegion]?.[1] ?? 0)} failed readings for region ${actualRegion}`); // unstructured text can only count a single known phrase const PHRASE = "reading processed", matchCount = lines.filter((x) => x.includes(PHRASE)).length; console.log(`\nQ5 'how many readings did the nightly job process'`); console.log(` actual ${records.length} lines containing "${PHRASE}" in raw text ${matchCount}` + ` deviation ${(100 * (records.length - matchCount) / records.length).toFixed(1)}%`); // Q1: does the service average come out correctly from the fields const avgDuration = (l, s, f) => { const g = l.filter((x) => x.service === s).map(f); return g.reduce((a, b) => a + b, 0) / g.length; }; const slowest = (l, f) => SERVICE.map((s) => [s, avgDuration(l, s, f)]).sort((a, b) => b[1] - a[1])[0]; const [actualSlow, actualSlowVal] = slowest(records, (k) => k.duration), [parsedSlow, parsedSlowVal] = slowest(fields.filter((a) => a.service), (a) => a.duration); console.log(`Q1 'which service slowed down' actual ${actualSlow} ${actualSlowVal.toFixed(0)} ms` + ` from parsing ${parsedSlow} ${parsedSlowVal.toFixed(0)} ms same answer: ${actualSlow === parsedSlow ? "yes" : "no"}`); // the six questions within this stream's scope; the others sit on other line types const QUESTIONS = [["Q1", ["service", "duration"], "numeric", actualSlow === parsedSlow], ["Q2", ["duration"], "numeric", true], ["Q3", ["environment", "result"], "group", true], ["Q5", ["phrase"], "count", matchCount === records.length], ["Q7", ["region", "result"], "group", actualRegion === parsedRegion], ["Q8", ["instance"], "group", true]]; const coverage = (a) => fields.filter((x) => x[a] !== "" && x[a] !== null).length / fields.length; const bytes = lines.reduce((t, x) => t + x.length, 0); const fieldBytes = fields.reduce((t, a) => t + Object.values(a).join("").length + Object.keys(a).length * 2, 0); console.log("\nstore format bytes answerable correct answer bytes/correct question answered wrong"); const print = (name, b, y) => { const correct = y.filter(([, , , ok]) => ok); console.log(`${name.padEnd(18)}${b.toLocaleString("en-US").padStart(9)}${`${y.length}/6`.padStart(15)}` + `${`${correct.length}/6`.padStart(16)}${(correct.length ? Math.round(b / correct.length).toLocaleString("en-US") : "-").padStart(24)} ` + (y.filter((q) => !correct.includes(q)).map(([k]) => k).join(" ") || "-")); }; print("raw text", bytes, QUESTIONS.filter(([, , i]) => i === "count")); print("fields + index", bytes + fieldBytes, QUESTIONS.filter(([, req, , ]) => req.every((g) => g === "phrase" || coverage(g) >= 0.80))); ``` ``` log lines produced: 900 sample lines 2026-08-02T02:00:00Z production validator[u02] INFO reading processed region=11 result=passed duration=210ms 2026-08-02T02:00:01Z production reading-collector[u21] INFO reading processed region=01 result=passed duration=217ms 2026-08-02T02:15:47Z environment=production service=validator instance=u21 level=info event=reading zone=05 status=passed duration_ms=250 2026-08-02T02:15:52Z environment=production service=validator instance=u08 level=info event=reading zone=11 status=passed duration_ms=83 parsing rule matched 736/900 lines; 164 lines have silently empty fields (18.2%) Q7 'which region's readings were not validated' actual region 07 29 failed / 97 readings from parsing region 13 18 failed / 51 readings deviation 37.9% same answer: no the rule never saw 21 failed readings for region 07 Q5 'how many readings did the nightly job process' actual 900 lines containing "reading processed" in raw text 736 deviation 18.2% Q1 'which service slowed down' actual validator 225 ms from parsing validator 238 ms same answer: yes store format bytes answerable correct answer bytes/correct question answered wrong raw text 103,440 1/6 0/6 - Q5 fields + index 159,828 6/6 4/6 39,957 Q5 Q7 ``` The rule matches 736 of the 900 lines. The remaining 164 lines are **not rejected**; when parsing fails, the line is not discarded, only its fields stay empty. The store holds 900 records, all searchable, all with their timestamp in place. Only in 164 of them are the region, result, service, and duration fields empty, and that leaves no trace in a query's result. #### The Answer That Changes Silently Q7 carries this lesson's most expensive number. In reality, the region with the most failed readings is **07**: twenty-nine of ninety-seven readings failed validation. The answer that comes out of the parsed fields is **13**. The deviation is not merely 37.9 percent — **the answer is different**. The reason lies in the design from the first section. Most of region 07's failed readings are written by the validator, which has switched to the new format that does not match the rule; the rule has **never seen** twenty-one failed readings for region 07. Region 13's failures are written by the other services, still on the old format, and all get counted. The field crew goes to the wrong region: work orders open for region 13, region 07's meters stay unread, and no bill is issued. This is a loss type not seen on the metric side. A missing metric breaks the series, visibly. An empty field lets **the query still run** — over fewer lines, without saying so. The signal of the loss disappears along with the loss itself. Q5 shows the same phenomenon in a plainer form. How many readings the nightly job processed can be answered by counting the string `reading processed` in the raw text — the only kind of question unstructured text can genuinely answer. But the new format does not write that string; the count comes out to 736 instead of 900, 18.2 percent short, wrong by the same amount whether taken by grep or from parsing. Q1, by contrast, gives the correct answer: the slowest service is the validator in both calculations. But its value shifts — 225 in the actual average, 238 milliseconds from parsing. The same missing data changes the answer to one question and only shifts the number in another; which of the two happens depends on the data's distribution and is not known while the query is written. #### Text, Fields, and Cost The last table places the two store formats side by side. Raw text is 103,440 bytes and can answer **one** of the six questions — and it answers it wrong. Once fields are produced, the bytes rise to 159,828, a 54.5 percent increase, and **all six of the six questions** become answerable. What grows is not the lines, it is the field names, the separated values, and the search index. But the table's real column is the third one: only **four** of the six answerable questions are answered **correctly**. This two-question gap between "answerable" and "correct" is what this lesson measures. The cost per correct answer is 39,957 bytes; if all six questions had been correct, it would have been 26,638 bytes. The difference is the price of the rule being left unmaintained, and there is no alert that would trigger on it — the 18.2 percent non-matching line share appears nowhere unless it is collected as a metric in its own right. #### Five Ways to Fix the Rule Five parsing options are applied to the same set of lines; matched lines, correct answers, and bytes are measured together. ```js // options.mjs — the lines matched, the correct answers, and the byte cost of each parsing option. import { generate } from "./lines.mjs"; const { lines, records } = generate(); const OLD = /^(\S+) (\w+) ([\w-]+)\[(\w+)\] \w+ .*? region=(\d+) result=(\w+) duration=(\d+)ms$/; const NEW = /^(\S+) environment=(\w+) service=([\w-]+) instance=(\w+) .*? zone=(\d+) status=(\w+) duration_ms=(\d+)$/; const keyValue = (s) => Object.fromEntries([...s.matchAll(/(\w+)=(\S+)/g)].map((m) => [m[1], m[2]])); const FIELD_MAP = { zone: "region", status: "result", duration_ms: "duration" }; const remap = (o) => Object.fromEntries(Object.entries(o).map(([k, v]) => [FIELD_MAP[k] ?? k, v])); const fromMatch = (m) => ({ environment: m[2], service: m[3], instance: m[4], region: m[5], result: m[6], duration: +m[7] }); const empty = { environment: "", service: "", instance: "", region: "", result: "", duration: null }; const OPTIONS = [ ["single rule (old format)", (s) => { const m = OLD.exec(s); return m ? fromMatch(m) : empty; }], ["two rules (old + new)", (s) => { const m = OLD.exec(s) || NEW.exec(s); return m ? fromMatch(m) : empty; }], ["key-value, unmapped", (s) => ({ ...empty, ...keyValue(s) })], ["key-value + field map", (s) => ({ ...empty, ...remap(keyValue(s)) })], ["structured at the source", null], // every line is written under a single convention ]; const structured = records.map((k) => `time=2026-08-02T02:00:00Z environment=production service=${k.service}` + ` instance=${k.instance} event=reading region=${k.region} result=${k.result} duration=${k.duration}`); const worst = (l) => { const t = {}; for (const a of l) { if (!a.region) continue; t[a.region] ??= [0, 0]; t[a.region][0]++; if (a.result === "failed") t[a.region][1]++; } return Object.entries(t).sort((a, b) => b[1][1] - a[1][1])[0]; }; const [actualRegion] = worst(records), rawBytes = lines.reduce((t, x) => t + x.length, 0); console.log("option matched Q7 answer Q7 correct Q5 correct bytes bytes/correct question"); for (const [name, f] of OPTIONS) { const source = f ? lines : structured; const fields = f ? lines.map(f) : structured.map((s) => ({ ...empty, ...keyValue(s) })); const filled = fields.filter((a) => a.region).length; const [region] = worst(fields), q7 = region === actualRegion, q5 = filled === records.length; const bytes = source.reduce((t, x) => t + x.length, 0) + fields.reduce((t, a) => t + Object.values(a).join("").length + Object.keys(a).length * 2, 0); const correct = 4 + (q7 ? 1 : 0) + (q5 ? 1 : 0); // Q1 Q2 Q3 Q8 correct under every option console.log(`${name.padEnd(31)}${String(`${filled}/900`).padStart(9)}${String(region).padStart(11)}` + `${(q7 ? "yes" : "no").padStart(12)}${(q5 ? "yes" : "no").padStart(12)}` + `${bytes.toLocaleString("en-US").padStart(9)}${Math.round(bytes / correct).toLocaleString("en-US").padStart(24)}`); } console.log(`\nraw text ${rawBytes.toLocaleString("en-US")} bytes; actual worst region ${actualRegion}`); ``` ``` option matched Q7 answer Q7 correct Q5 correct bytes bytes/correct question single rule (old format) 736/900 13 no no 140,028 35,007 two rules (old + new) 900/900 07 yes yes 145,424 24,237 key-value, unmapped 736/900 13 no no 132,560 33,140 key-value + field map 900/900 07 yes yes 131,576 21,929 structured at the source 900/900 07 yes yes 188,368 31,395 raw text 103,440 bytes; actual worst region 07 ``` The third row is the unexpected one. The key-value parser reads both formats and rejects no line; even so, the Q7 answer is still 13. The gap is between parsing succeeding and the field coming out under **the right name**: the new format writes the region under the `zone` key, the query looks for `region`, and the two do not match. A format-independent parser solves the format problem, it **does not solve the naming problem**. The fourth row is the cheapest correct option: a three-line field map is added to the same parser, and the cost per correct answer drops from 35,007 bytes to 21,929. The two-rule option is also correct, but every new format demands one more rule; the mapping table only grows by one line. The fifth row solves the problem at the source: every service writes under a single convention, and no parsing rule is needed at all. It cannot break, and it needs no maintenance — and it is **the most expensive**: because the field names repeat on every line, the bytes rise from 131,576 to 188,368, a 43.2 percent increase. Parsing is a byte saving, and the price of that saving is maintenance load; which one is cheaper depends on line volume and how often the format changes. #### Summary - The parsing rule matches 736 of the 900 lines; the remaining 164 lines are not discarded, only their fields stay empty (18.2%), and this leaves no trace in any query result. - Unstructured text can answer only one of six questions (counting a single known string); once fields are produced, all six become answerable, and bytes rise by 54.5%. - Being answerable is not the same as being correct: four of the six questions are answered correctly. On Q7, the answer comes out as region 13 instead of region 07 — the rule has never seen 21 failed readings for region 07. - The same missing data does not change the answer on Q1, it only shifts the value from 225 to 238 milliseconds; which question breaks depends on the distribution of the data, and it is not known while the query is being written. - The cost per correct answer is 39,957 bytes; if the rule had been kept up to date, it would have been 26,638 bytes. The 18.2% non-matching line share appears nowhere unless it is collected as a metric. - A format-independent parser solves the format problem, not the naming problem (still 736/900); once a field map is added, the cost per correct answer comes to 21,929 bytes. Writing structured at the source cannot break, but it takes 43.2% more bytes. #### Next Step Up to this point, metrics and logs were collected, parsed, and counted against the question set separately. But in every measurement, retention was held fixed, and cost was computed at a single level of detail. A real pipeline does not work that way: as data ages, its detail is reduced, it is downsampled, summarized, and eventually discarded. The next lesson scans through these tiers and counts, at every tier, the number of questions answerable together with the bytes: where the cost per question goes tier by tier, and which question the cheapest tier loses. ### Retention and Cost Source: https://academia.sh/en/courses/devops-observability/retention-and-cost Three retention tiers are scanned, and at every tier the number of answerable questions is counted together with the bytes: cost per question is derived tier by tier, one tier's marginal cost differs by a thousandfold, and which questions the cheapest tier loses, and why, is written down by name. In the five measurements so far, retention was always held fixed. The metric pipeline was priced over thirty days, the log pipeline over one hour, and every lesson measured at a single level of detail. A real pipeline does not work that way: as data ages, its resolution is lowered, its labels are downsampled, its lines are discarded, and it eventually collapses into a summary. This lesson scans through those tiers. The measure does not change — at every tier, the **number of answerable questions** is counted together with the **bytes**, and cost per question is derived tier by tier. In the end, two numbers stand side by side: the marginal cost of adding a tier, and the name of the question that tier closes. The numbers come from the **regional measurement network's** pipeline and are **fictional**; what gets measured is a **model** built with `node`, and its inputs come from this topic's previous lessons. **ML31.** The age range, resolution, dropped labels, and kept log line types of the three tiers are fictional and written in advance. **ML32.** The metric series count is taken from the second lesson's actual set: 6,888 series. At the summary tier, the `instance` label is rolled up to the service level, and the `bucket` label is dropped entirely. **ML33.** Log volume derives from the fourth lesson's stream: a base of 232 lines/second, 540,000 lines from the nightly job; an error line is 512 bytes, the truncation limit. **ML34.** Every question's horizon comes from the second lesson, its resolution requirement from the third, and its required log line type from the fourth and fifth. **ML35.** A tier is counted as answering a question if its age range covers the question's horizon, its resolution is sufficient, it has not dropped the required label, and it keeps the required line type. **ML36.** Q12 is answered at no tier; the data it requires is not present in any tier's content. #### Three Tiers and Their Distinguishing Measures ```js // tiers.mjs — retention tiers, the questions answered at each tier, and the cost per question. export const METRICS = { // metric: [instance base, service base, other-label multiplier, bucket] "request-count": [29, 4, 4 * 5, 1], "request-duration": [29, 4, 4, 12], "queue-depth": [29, 4, 4 * 3, 1], "processed-reading": [20, 2, 4 * 11 * 5, 1], "job-step-duration": [6, 3, 4 * 6, 1], "job-processed-records": [6, 3, 4, 1], }; export const seriesCount = (summary) => Object.values(METRICS).reduce((t, [a, b, c, d]) => t + (summary ? b : a) * c * (summary ? 1 : d), 0); // summary rolls instance up to service, drops bucket entirely export const LOG_LINES = { request: 232 * 0.86 * 86400, error: 232 * 0.02 * 86400, "job-step": 232 * 0.12 * 86400 + 540000 }; // lines/day; nightly job adds 540,000 lines export const LINE_BYTES = { request: 180, error: 512, "job-step": 240 }; // tier: [name, start day, end day, resolution (s), is summary, log line types it keeps] export const TIERS = [["raw", 0, 7, 15, false, ["request", "error", "job-step"]], ["downsampled", 7, 90, 300, false, ["error", "job-step"]], ["summary", 90, 395, 3600, true, []]]; // question: [id, required horizon (days), required resolution (s), required labels, required line type] export const QUESTIONS = [["Q1", 1, 300, ["service"], null], ["Q2", 1, 15, ["bucket"], null], ["Q3", 1, 3600, ["environment", "result"], null], ["Q4", 30, 3600, ["version"], null], ["Q5", 7, null, [], "job-step"], ["Q6", 7, null, [], "job-step"], ["Q7", 365, 3600, ["region", "result"], null], ["Q8", 7, 300, ["instance"], null], ["Q9", 7, 300, ["instance"], "job-step"], ["Q10", 1, null, [], "error"], ["Q11", 1, 300, ["queue"], null], ["Q12", 1, null, [], "tracing"]]; export const DROPPED = ["instance", "bucket"]; // labels dropped at the summary tier export const fits = (tier, question, checkAge = true) => { // does the tier answer the question const [, start, end, resolution, summary, types] = tier, [, horizon, reqResolution, reqLabels, reqType] = question; if (checkAge && !(start <= horizon && horizon <= end)) return false; if (reqResolution !== null && resolution > reqResolution) return false; if (summary && reqLabels.some((e) => DROPPED.includes(e))) return false; return reqType === null || types.includes(reqType); }; export const gb = (b) => (b / 1024 ** 3).toFixed(2); ``` Four measures define a tier, and all four can close off a question: **age range** (does it cover the question's horizon), **resolution** (the query-window measurement from the third lesson), **dropped label** (the label ladder from the second lesson), and **kept line type** (the layer loss from the fourth lesson). Designing a tier means deciding on all four of these at once. #### Bytes and Questions, Tier by Tier ```js // cost.mjs — bytes tier by tier, questions answered, and cost per question. import { METRICS, seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, DROPPED, fits, gb } from "./tiers.mjs"; const metricBytes = (b) => { const [, start, end, resolution, summary] = b; return seriesCount(summary) * Math.floor(86400 / resolution) * (end - start) * 16 + seriesCount(summary) * 120; }; const logBytes = (b) => (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0); console.log(`metric series: raw/downsampled ${seriesCount(false).toLocaleString("en-US")},` + ` summary ${seriesCount(true).toLocaleString("en-US")} (dropped labels: ${DROPPED.join(", ")})`); console.log("\ntier days resolution metric GB log GB total GB within age by detail GB/question"); const rows = []; for (const b of TIERS) { const m = metricBytes(b), l = logBytes(b), t = m + l; const withinAge = QUESTIONS.filter((s) => fits(b, s)), byDetail = QUESTIONS.filter((s) => fits(b, s, false)); rows.push([b, t, withinAge, byDetail]); console.log(`${b[0].padEnd(14)}${`${b[1]}-${b[2]}`.padEnd(9)}${`${b[3]} s`.padStart(12)}` + `${gb(m).padStart(11)}${gb(l).padStart(9)}${gb(t).padStart(11)}${`${withinAge.length}/12`.padStart(13)}` + `${`${byDetail.length}/12`.padStart(14)}${gb(t / (withinAge.length || 1)).padStart(13)}`); } console.log("\ncumulative (tiers stacked on top of each other)"); console.log("tiers total GB answered GB/question marginal GB/new question newly opened"); let total = 0, prev = []; for (let i = 0; i < rows.length; i++) { const tierBytes = rows[i][1], name = TIERS.slice(0, i + 1).map((b) => b[0]).join(" + "); total += tierBytes; const answered = [...new Set(rows.slice(0, i + 1).flatMap(([, , withinAge]) => withinAge))]; const opened = answered.filter((s) => !prev.includes(s)); console.log(`${name.padEnd(29)}${gb(total).padStart(10)}${`${answered.length}/12`.padStart(13)}` + `${gb(total / answered.length).padStart(13)}${(opened.length ? gb(tierBytes / opened.length) : "-").padStart(26)} ` + (opened.map(([k]) => k).join(" ") || "-")); prev = answered; } const cheapest = rows[rows.length - 1]; const lost = QUESTIONS.filter((s) => !cheapest[3].includes(s) && s[0] !== "Q12"); console.log(`\ncheapest tier: ${cheapest[0][0]}, ${gb(cheapest[1])} GB, ${cheapest[3].length}/12 questions by detail` + `, ${gb(cheapest[1] / cheapest[3].length)} GB per question`); console.log("questions it loses and why"); for (const s of lost) { const reasons = [cheapest[0][3] > (s[2] ?? Infinity) && `resolution ${cheapest[0][3]} s`, s[3].some((e) => DROPPED.includes(e)) && `${s[3].filter((e) => DROPPED.includes(e)).join("/")} label dropped`, s[4] && !cheapest[0][5].includes(s[4]) && `${s[4]} lines not kept`].filter(Boolean); console.log(` ${s[0].padEnd(4)} ${reasons.join("; ")}`); } ``` ``` metric series: raw/downsampled 6,888, summary 668 (dropped labels: instance, bucket) tier days resolution metric GB log GB total GB within age by detail GB/question raw 0-7 15 s 4.14 26.18 30.31 9/12 11/12 3.37 downsampled 7-90 300 s 2.45 70.51 72.96 5/12 10/12 14.59 summary 90-395 3600 s 0.07 0.00 0.07 1/12 3/12 0.07 cumulative (tiers stacked on top of each other) tiers total GB answered GB/question marginal GB/new question newly opened raw 30.31 9/12 3.37 3.37 Q1 Q2 Q3 Q5 Q6 Q8 Q9 Q10 Q11 raw + downsampled 103.28 10/12 10.33 72.96 Q4 raw + downsampled + summary 103.35 11/12 9.40 0.07 Q7 cheapest tier: summary, 0.07 GB, 3/12 questions by detail, 0.02 GB per question questions it loses and why Q1 resolution 3600 s Q2 resolution 3600 s; bucket label dropped Q5 job-step lines not kept Q6 job-step lines not kept Q8 resolution 3600 s; instance label dropped Q9 resolution 3600 s; instance label dropped; job-step lines not kept Q10 error lines not kept Q11 resolution 3600 s ``` The two columns measure separate things, and seeing that distinction is the key to this table. The **within age** column counts questions whose horizon falls inside that tier's age window; the **by detail** column counts how many questions that tier's detail could answer if age were disregarded entirely. The raw tier carries eleven questions by detail but answers only nine, because the horizons of the version comparison and the seasonal region comparison fall outside seven days. Keeping the detail is not enough; the detail has to sit alongside old enough data. The byte split is also different from what might be expected. Of the raw tier's 30.31 gigabytes, 26.18 is log, and only 4.14 is metric. At the downsampled tier, even though request lines are discarded, the log still holds 70.51 gigabytes — because error and job-step lines also accumulate over an eighty-three-day window. The summary tier is close to free on both sides: three hundred five days of retention comes down to 0.07 gigabytes. #### The Reversal of Marginal Cost The cumulative table gives the number that carries the decision. The raw tier opens nine questions for 30.31 gigabytes — 3.37 per question. When the downsampled tier is added, the total rises to 103.28 gigabytes, and questions answered rise from nine to **ten**: seventy-three gigabytes, a single question. The marginal cost is 72.96 gigabytes per question, twenty-one times the raw tier. When the summary tier is added, the total becomes 103.35 gigabytes, and questions answered reach eleven. The marginal cost is **0.07 gigabytes per question** — a thousand times cheaper than the downsampled tier. That ratio is this lesson's sharpest number: both tiers open exactly one question, and one is a thousand times more expensive than the other. The reason for the difference is not the tier's length, it is **what it keeps**. The downsampled tier keeps the log line and the fully labeled series for eighty-three days; the single question it opens asks only for the `version` label at an hourly resolution. So almost all of the data kept for a thirty-day version comparison goes unused by that question. The same question could have been answered by the summary tier's own detail. This shows the real decision in tier design: lowering the downsampled tier's resolution and line selection to the minimum the questions require drops the cost of a single question from seventy-three gigabytes to the single-gigabyte range. Shortening the tier's duration, on the other hand, closes Q4 entirely and gains nothing. #### The Questions the Cheapest Tier Loses The summary tier answers three questions by detail for 0.07 gigabytes — 0.02 gigabytes per question, the cheapest number on the whole pipeline. A decision that looked only at this number would reduce everything to the summary. The losses stand in the list by name, and each one's reason is a separate design item. Because resolution rises to 3600 seconds, **Q1** (which service slowed down), **Q11** (why the work-order queue backed up), and partly **Q8** drop — it was measured in the third lesson that a ninety-second degradation in a one-hour window shows up in neither the average nor the percentile. **Q2** drops because the `bucket` label is dropped: without a distribution bucket, the tail cannot be computed. **Q8** and **Q9** drop because the `instance` label is rolled up to the service — which instance restarted and which run was slow can no longer be asked. **Q5**, **Q6**, and **Q10** drop because log lines are not kept at all. All eight of the eight questions close silently. At the summary tier, these queries do not error; they return an empty result or a single aggregate value. This has the same shape as the truncation finding from the fifth lesson: the signal of the loss disappears along with the loss itself. #### Tier Layouts Side by Side Now that the individual cost of each tier is known, the layout itself can be compared: in which layout does the same twelve-question set get answered for how many gigabytes? ```js // policy.mjs — tier layouts are compared: how many gigabytes does the same question set cost under each. import { seriesCount, LOG_LINES, LINE_BYTES, TIERS, QUESTIONS, fits, gb } from "./tiers.mjs"; const resize = (name, start, end) => { // move a tier to a different age range const b = TIERS.find((x) => x[0] === name); return [name, start, end, b[3], b[4], b[5]]; }; const bytes = (b) => seriesCount(b[4]) * Math.floor(86400 / b[3]) * (b[2] - b[1]) * 16 + seriesCount(b[4]) * 120 + (b[2] - b[1]) * b[5].reduce((t, type) => t + LOG_LINES[type] * LINE_BYTES[type], 0); const POLICIES = [ ["raw only, 7 days", [resize("raw", 0, 7)]], ["raw only, 30 days", [resize("raw", 0, 30)]], ["raw only, 395 days", [resize("raw", 0, 395)]], ["three tiers 7 / 90 / 395", [resize("raw", 0, 7), resize("downsampled", 7, 90), resize("summary", 90, 395)]], ["raw 7 + summary 395 (no middle)", [resize("raw", 0, 7), resize("summary", 7, 395)]], ["raw 30 + summary 395", [resize("raw", 0, 30), resize("summary", 30, 395)]], ]; console.log("policy total GB answered GB/question unanswered"); const results = POLICIES.map(([name, bs]) => { const t = bs.reduce((s, b) => s + bytes(b), 0); const y = QUESTIONS.filter((s) => bs.some((b) => fits(b, s))); console.log(`${name.padEnd(33)}${gb(t).padStart(9)}${`${y.length}/12`.padStart(13)}${gb(t / y.length).padStart(13)} ` + QUESTIONS.filter((s) => !y.includes(s)).map(([k]) => k).join(" ")); return [name, t, y.length]; }); const [, threeTierBytes, threeTierAnswered] = results[3], [, noMidBytes, noMidAnswered] = results[4]; console.log(`\nthe middle tier's contribution: ${gb(threeTierBytes - noMidBytes)} GB more, ${threeTierAnswered - noMidAnswered} new question`); console.log(`the cost of keeping everything raw: ${gb(results[2][1] - noMidBytes)} GB more,` + ` ${results[2][2] - noMidAnswered} new question`); ``` ``` policy total GB answered GB/question unanswered raw only, 7 days 30.31 9/12 3.37 Q4 Q7 Q12 raw only, 30 days 129.92 10/12 12.99 Q7 Q12 raw only, 395 days 1710.56 11/12 155.51 Q12 three tiers 7 / 90 / 395 103.35 11/12 9.40 Q12 raw 7 + summary 395 (no middle) 30.41 11/12 2.76 Q12 raw 30 + summary 395 130.00 11/12 11.82 Q12 the middle tier's contribution: 72.94 GB more, 0 new question the cost of keeping everything raw: 1680.16 GB more, 0 new question ``` Four layouts answer the same eleven questions, and there is a fifty-six-fold price difference between them. The cheapest is the layout that **never sets up** the middle tier: seven days of raw, then straight to summary. Total 30.41 gigabytes, 2.76 per question. The middle layer of the three-tier layout holds 72.94 gigabytes extra and opens **zero new questions**. The reason was seen in the fourth section: the downsampled tier's only contribution was Q4, and because Q4 only needs the `version` label at hourly resolution, it is also answered by the summary tier's own detail. The middle tier was built only on the intuition that "seven days is too little, ninety days would be better"; counted against the question set, it has no payoff. The "keep everything raw" layout answers the same eleven questions for 1710.56 gigabytes — 1680 gigabytes extra, zero new questions. Its cost per question is 155.51 gigabytes, fifty-six times the cheapest layout. These two lines pin this course's rule to a number: data that answers no question, no matter how carefully it is collected, is a cost. #### Summary - Four items define a tier, and all four can close off a question: age range, resolution, dropped label, and kept line type. - The raw tier carries 11/12 questions by detail but answers 9/12 because of its age range; keeping the detail is not enough, it has to sit alongside old enough data. - Bytes accumulate on the log side: of the raw tier's 30.31 GB, 26.18 is log, 4.14 is metric; at the downsampled tier, even though request lines are discarded, the log holds 70.51 GB. - Marginal cost reverses: the downsampled tier opens one question (Q4) for 72.96 GB, the summary tier opens one question (Q7) for 0.07 GB — a thousandfold difference, both exactly one question. - The cheapest tier (summary) answers 3/12 questions at 0.02 GB/question and silently loses eight questions by name: Q1, Q11 from resolution; Q2 from the `bucket` label; Q8, Q9 from the `instance` label; Q5, Q6, Q10 from log lines. The queries do not error, they return an empty or a single aggregate value. - In the layout comparison, four different layouts answer the same 11/12: the cheapest is the one that never sets up the middle tier (30.41 GB, 2.76 GB/question). The middle tier holds 72.94 GB extra and opens zero questions; "keep everything raw" holds 1680 GB extra and opens zero questions. #### Next Step This topic took data end to end: how it is collected, which labels it is stored with, which aggregation it is queried through, which layer of the log pipeline loses what, how text is turned into a searchable field, and how much it costs at which tier as it ages. Eleven of the twelve questions became answerable somewhere, and the price of every answer is known. One question remains, and it appeared at no tier, in no label set, on no log line: which reading a bill was generated from. Asking this means asking for a single transaction's **path** from the reading collector to the validator, from there to billing and the work order. A metric cannot carry it, because a transaction ID's cardinality is unbounded; a log line cannot carry it, because every service generates its own identity and the lines cannot be linked to each other. Not a single byte of the hundred gigabytes collected shows that path. The next topic starts from this gap: what does making a request's path across services visible require, and which questions does it open. ## Monitoring and Dashboards Tying the collected data to a question: the collector and storage components of distributed tracing, what a shared telemetry convention buys, the design of a dashboard that answers a question, threshold-based, rate-based, and anomaly-based alert rules, measuring alert fatigue through noise reduction and prioritization, and synthetic monitoring that verifies periodically from the outside. ### Distributed Tracing Infrastructure Source: https://academia.sh/en/courses/devops-observability/distributed-tracing-infrastructure Building the path a request takes across five services as a trace tree: putting eleven operational questions to the metric, log, and trace views separately, measuring the byte cost per span, and counting the slow requests the sampling ratio misses. The metric was collected, the log was stored, and the cost of both was worked out. The latency curve's rise is visible too. What is not visible is the path a single request behind that curve took across five services: which span it waited in, which call was retried, which service was never called at all. None of these three questions has an answer in the metric buckets or the log lines; both count the event, not the request. Distributed tracing has three parts. A span record is the record of one unit of work: its own id, its parent span's id, the service name, the start, the duration. Context propagation carries the request's id from call to call; if it is not carried, the tree splits into five separate pieces. The collector writes span records to the trace store, and the store groups records by trace id: a time series store is indexed by time, a trace store is indexed by id; a metric record is written once a minute, a span record once per span. #### A Request's Tree The model generates one hour of traffic on the regional measurement network; the generator is written in code, it is fiction, not a measurement. ```js // trace/model.mjs — MODEL of one hour of traffic on the regional measurement network (fiction). Each // request passes through five services; each span is one record: id, parent, service, start, duration, // error, attempt. MD1: the call graph is fixed — gateway calls read and billing, read calls validate, // billing calls workorder. MD2: a span's own duration is baseline + noise, error rate per mille. MD3: for // some requests, read answers from cache and validate is never called. MD4: some workorder calls time out // and retry once. MD5: some requests are slow; slowness accumulates in a single span. export const SEED = 20260802, REQUESTS = 24000, MINUTES = 60; export const SERVICES = ["gateway", "read", "validate", "billing", "workorder"]; export const PROFILE = { gateway: [4, 2, 0], read: [18, 6, 2], validate: [26, 9, 3], billing: [31, 11, 4], workorder: [44, 15, 9] }; export const SKIP = 0.11, RETRY = 0.06, SLOW = 0.02, TIMEOUT = 120; export function generate() { let s = SEED; const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); const jitter = (name) => { const [t, g] = PROFILE[name]; return Math.max(1, Math.round(t + (r() * 2 - 1) * g)); }; const traces = []; for (let i = 0; i < REQUESTS; i += 1) { const t0 = Math.floor(r() * MINUTES * 60000), spans = []; let seq = 0; const addSpan = (parent, service, start, duration, error, attempt) => (spans.push({ id: (seq += 1), parent, service, start, duration, error, attempt }), seq); const slowSpan = r() < SLOW ? (r() < 0.6 ? "workorder" : SERVICES[1 + Math.floor(r() * 4)]) : null; const factor = slowSpan ? 8 + Math.floor(r() * 13) : 1; const stretch = (name, d) => (name === slowSpan ? d * factor : d); const skipped = r() < SKIP, verifyDuration = skipped ? 0 : stretch("validate", jitter("validate")); const readOwn = stretch("read", jitter("read")); const readDuration = readOwn + (skipped ? 0 : verifyDuration + 2); const retried = r() < RETRY, workorderDuration = stretch("workorder", jitter("workorder")); const workorderTotal = workorderDuration + (retried ? TIMEOUT + 2 : 0); const billingDuration = stretch("billing", jitter("billing")) + workorderTotal + 2; const billingStart = PROFILE.gateway[0] + readDuration + 2; const root = addSpan(0, "gateway", 0, billingStart + billingDuration, false, 1); const read = addSpan(root, "read", PROFILE.gateway[0], readDuration, false, 1); if (!skipped) addSpan(read, "validate", PROFILE.gateway[0] + readOwn + 1, verifyDuration, false, 1); const billing = addSpan(root, "billing", billingStart, billingDuration, false, 1); let wStart = billingStart + 2; if (retried) { addSpan(billing, "workorder", wStart, TIMEOUT, true, 1); wStart += TIMEOUT + 2; } addSpan(billing, "workorder", wStart, workorderDuration, r() < PROFILE.workorder[2] / 1000, retried ? 2 : 1); traces.push({ traceId: `trace-${(i + 1).toString(36).padStart(4, "0")}`, t0, total: billingStart + billingDuration, spans, skipped, retried, slowSpan }); } return traces; } // A span's own duration: span duration minus the duration of its children. export const ownTime = (trace, a) => a.duration - trace.spans.filter((c) => c.parent === a.id).reduce((x, c) => x + c.duration, 0); if (import.meta.filename === process.argv[1]) { const traces = generate(), spanCount = traces.reduce((x, z) => x + z.spans.length, 0); console.log(`seed ${SEED}, ${REQUESTS} requests, ${spanCount} spans, ${SERVICES.length} services, ${MINUTES} minutes`); console.log(`validation skipped ${traces.filter((z) => z.skipped).length}, retried call ` + `${traces.filter((z) => z.retried).length}, slow request ${traces.filter((z) => z.slowSpan).length}`); const o = traces.find((z) => z.retried && z.slowSpan === "workorder" && !z.skipped); console.log(`\ntrace tree — ${o.traceId} (${o.total} ms)`); const printSpan = (parent, indent) => o.spans.filter((x) => x.parent === parent).forEach((a) => { console.log(`${indent}${a.service.padEnd(14 - indent.length)}${String(a.duration).padStart(6)} ms` + ` own ${String(ownTime(o, a)).padStart(4)} ms attempt ${a.attempt}${a.error ? " timeout" : ""}`); printSpan(a.id, indent + " "); }); printSpan(0, ""); } ``` ``` seed 20260802, 24000 requests, 118794 spans, 5 services, 60 minutes validation skipped 2624, retried call 1418, slow request 477 trace tree — trace-008m (889 ms) gateway 889 ms own 6 ms attempt 1 read 47 ms own 16 ms attempt 1 validate 31 ms own 31 ms attempt 1 billing 836 ms own 44 ms attempt 1 workorder 120 ms own 120 ms attempt 1 timeout workorder 672 ms own 672 ms attempt 2 ``` The indentation shows the bond. The root span took 889 ms, its own work 6 ms. The billing span took 836 ms, its own work 44 ms — the rest passed waiting on the workorder call, and that call was made twice; the first attempt timed out at 120 ms. None of this can be read from a single duration metric: without the parent-child bond, 889 ms is just a number. #### Eleven Questions, Three Views Three views are produced from the same events: minute-by-minute metric buckets, one log line per span, trace trees. The questions are put to all three in the same form; if a view does not carry the required field, the search returns empty and no answer is written. ```js // trace/question.mjs — three views are produced from the same events: minute-by-minute metric // buckets, one log line per span, trace trees. Eleven questions are asked of all three; if a view // does not carry the required field, the search returns empty and no answer is written. import { generate, ownTime, REQUESTS } from "./model.mjs"; const traces = generate(), BUCKETS = [25, 50, 100, 200, 400, 800, 1600, Infinity], THRESHOLD = 400; // MD6 (assumption): the metric bucket and the log line carry no request id; the span carries an // error flag, not the error message. const METRIC = new Map(), LOG = [], MESSAGES = ["timeout", "connection refused", "invalid read"]; for (const z of traces) for (const a of z.spans) { const k = `${a.service}|${Math.floor((z.t0 + a.start) / 60000)}`; let o = METRIC.get(k); if (!o) METRIC.set(k, (o = { n: 0, errors: 0, buckets: new Array(BUCKETS.length).fill(0) })); o.n += 1; o.errors += a.error ? 1 : 0; o.buckets[BUCKETS.findIndex((b) => a.duration <= b)] += 1; LOG.push({ service: a.service, duration: a.duration, error: a.error, message: a.error ? MESSAGES[(a.duration + a.attempt) % 3] : "ok" }); } const p95 = (x) => x.slice().sort((a, b) => a - b)[Math.floor(x.length * 0.95)]; const mostCommon = (m) => [...m].sort((a, b) => b[1] - a[1])[0]; const countBy = (arr, f) => arr.reduce((m, x) => m.set(f(x), (m.get(f(x)) || 0) + 1), new Map()); const root = [...METRIC].filter(([x]) => x.startsWith("gateway|")).map(([, o]) => o); const gatewayLog = LOG.filter((l) => l.service === "gateway"), slowTraces = traces.filter((z) => z.total > THRESHOLD); const slowestSpan = (z) => z.spans.slice().sort((a, b) => ownTime(z, b) - ownTime(z, a))[0]; const avgTotal = (f) => { const g = traces.filter(f); return g.reduce((x, z) => x + z.total, 0) / g.length; }; const percent = (a, b) => `${(a / b * 100).toFixed(a / b < 0.1 ? 2 : 0)}%`; // the reported slow request: the one request whose id we happen to have const TARGET = traces.filter((z) => z.slowSpan && z.retried)[0], ID = TARGET.traceId; const noMetric = () => METRIC.get(ID) ?? null; // buckets are not looked up by id const noLog = () => LOG.find((l) => l.trace === ID) ?? null; // the row carries no trace field const QUESTIONS = [ ["request duration p95", () => { const t = root.reduce((a, o) => a.map((v, i) => v + o.buckets[i]), BUCKETS.map(() => 0)), n = t.reduce((a, b) => a + b, 0); let s = 0; for (let i = 0; ; i += 1) if ((s += t[i]) >= n * 0.95) return `<=${BUCKETS[i]} ms`; }, () => `${p95(gatewayLog.map((l) => l.duration))} ms`, () => `${p95(traces.map((z) => z.total))} ms`], ["error rate", () => percent([...METRIC.values()].reduce((a, o) => a + o.errors, 0), [...METRIC.values()].reduce((a, o) => a + o.n, 0)), () => percent(LOG.filter((l) => l.error).length, LOG.length), () => percent(traces.reduce((x, z) => x + z.spans.filter((a) => a.error).length, 0), LOG.length)], [`requests over ${THRESHOLD} ms`, () => root.reduce((a, o) => a + o.buckets[5] + o.buckets[6] + o.buckets[7], 0), () => gatewayLog.filter((l) => l.duration > THRESHOLD).length, () => slowTraces.length], [`${ID}: which span is slow`, noMetric, noLog, () => slowestSpan(TARGET).service], [`${ID}: slow span's share`, noMetric, noLog, () => percent(ownTime(TARGET, slowestSpan(TARGET)), TARGET.total)], [`${ID}: retried call`, noMetric, noLog, () => { const d = TARGET.spans.find((a) => a.attempt > 1); return d ? `${d.service} x${d.attempt}` : "none"; }], [`${ID}: was validate called`, noMetric, noLog, () => (TARGET.spans.some((a) => a.service === "validate") ? "yes" : "no")], ["most common slow span in slow requests", noMetric, noLog, () => { const [name, n] = mostCommon(countBy(slowTraces, (z) => slowestSpan(z).service)); return `${name} ${percent(n, slowTraces.length)}`; }], ["how much faster is a request with validation skipped", noMetric, noLog, () => `${(avgTotal((z) => !z.skipped) - avgTotal((z) => z.skipped)).toFixed(0)} ms`], ["most common error message", () => null, () => mostCommon(countBy(LOG.filter((l) => l.error), (l) => l.message))[0], () => null], ["the query inside the slow span", () => null, () => null, () => TARGET.spans.find((a) => a.parent === slowestSpan(TARGET).id && a.service === slowestSpan(TARGET).service) ?? null], ]; console.log(`${"question".padEnd(54)}${"metric".padStart(12)}${"log".padStart(22)}${"trace".padStart(15)}`); const n = [0, 0, 0]; for (const [label, ...f] of QUESTIONS) { const y = f.map((fn) => { const v = fn(); return v === null || v === undefined ? "-" : String(v); }); y.forEach((v, i) => { if (v !== "-") n[i] += 1; }); console.log(`${label.padEnd(54)}${y[0].padStart(12)}${y[1].padStart(22)}${y[2].padStart(15)}`); } const notInMetricOrLog = QUESTIONS.filter(([, o, g]) => (o() ?? null) === null && (g() ?? null) === null); console.log(`\n${REQUESTS} requests, ${QUESTIONS.length} questions — metric ${n[0]}, log ${n[1]}, trace ${n[2]}`); console.log(`unanswered by metric and log ${notInMetricOrLog.length} questions — answered by trace ` + `${notInMetricOrLog.filter(([, , , z]) => (z() ?? null) !== null).length}, answered by none ` + `${QUESTIONS.filter(([, ...f]) => f.every((fn) => (fn() ?? null) === null)).length}`); ``` ``` question metric log trace request duration p95 <=400 ms 248 ms 248 ms error rate 1.38% 1.38% 1.38% requests over 400 ms 411 411 411 trace-0024: which span is slow - - read trace-0024: slow span's share - - 45% trace-0024: retried call - - workorder x2 trace-0024: was validate called - - yes most common slow span in slow requests - - workorder 82% how much faster is a request with validation skipped - - 27 ms most common error message - connection refused - the query inside the slow span - - - 24000 requests, 11 questions — metric 3, log 4, trace 9 unanswered by metric and log 7 questions — answered by trace 6, answered by none 1 ``` The first three questions are answered by all three views, but not at the same resolution: the metric view says `<=400 ms` for p95, the real value is 248 ms — the bucket boundary lumps everything between 200 and 400 into one cell. The question is answered, but the answer is not enough to decide on; that is a different defect from being unanswered. The next four questions look at one reported request; the search returns empty in two views, because neither the bucket nor the log line carries a trace id. In request `trace-0024`, the longest own duration is in the read span (45% of the total duration), the workorder call was made twice, validate was called. The set-level question and the single-request question give different answers: in 82% of the 411 requests over 400 ms the slow span was the workorder, which was not the case in the reported request. The last two questions draw the boundary of tracing. Only the log carries the error message; the span holds an error flag, not the text. The query inside the slow span, though, is not answered by any of the three views: instrumentation does not open a span at that depth. Six of the seven questions were answered by the trace; the reason the seventh stays unanswered is not collection but instrumentation depth. #### Bytes per Span ```js // trace/cost.mjs — the byte cost per span and the root sampling decision are scanned. Record size comes // from the real record; no compression. TRACE_ONLY: count of questions answered only by the trace view. import { generate, REQUESTS } from "./model.mjs"; const traces = generate(), THRESHOLD = 400, TRACE_ONLY = 6, perDayMB = (b) => (b * 24 / 1e6).toFixed(1); const spanCount = traces.reduce((x, z) => x + z.spans.length, 0); const traceBytes = traces.reduce((x, z) => x + z.spans.reduce((y, a) => y + JSON.stringify({ trace: z.traceId, ...a }).length, 0), 0); console.log(`${spanCount} spans, ${(traceBytes / spanCount).toFixed(0)} bytes per span, ` + `${(traceBytes / REQUESTS).toFixed(0)} bytes per request (${(spanCount / REQUESTS).toFixed(1)} spans), ${perDayMB(traceBytes)} MB per day`); const hash = (s) => { let h = 7; for (const c of s) h = (h * 31 + c.charCodeAt(0)) >>> 0; return h; }; const slowTraces = traces.filter((z) => z.total > THRESHOLD), reportedId = traces.filter((z) => z.slowSpan && z.retried)[0].traceId; console.log(`\n${"ratio".padEnd(7)}${"recorded traces".padStart(18)}${"MB/day".padStart(8)}` + `${"captured".padStart(10)}${"slow missed".padStart(13)}${"MB/question".padStart(14)} reported request`); for (const n of [1, 2, 5, 20, 100]) { const keep = (z) => hash(z.traceId) % n === 0, kept = traces.filter(keep), bytes = traceBytes / REQUESTS * kept.length; console.log(`${`1/${n}`.padEnd(7)}${String(kept.length).padStart(18)}${perDayMB(bytes).padStart(8)}` + `${String(slowTraces.filter(keep).length).padStart(10)}` + `${String(slowTraces.length - slowTraces.filter(keep).length).padStart(13)}` + `${(+perDayMB(bytes) / TRACE_ONLY).toFixed(1).padStart(14)} ${kept.some((z) => z.traceId === reportedId) ? "recorded" : "not recorded"}`); } ``` ``` 118794 spans, 111 bytes per span, 549 bytes per request (4.9 spans), 316.0 MB per day ratio recorded traces MB/day captured slow missed MB/question reported request 1/1 24000 316.0 411 0 52.7 recorded 1/2 12001 158.0 209 202 26.3 not recorded 1/5 4801 63.2 84 327 10.5 not recorded 1/20 1204 15.9 16 395 2.6 not recorded 1/100 241 3.2 2 409 0.5 not recorded ``` A request costs 4.9 spans and 549 bytes; 111 bytes per span, 316.0 MB per day. Each of the six questions answered only by the trace costs 52.7 MB at this scale. The sampling ratio divides the volume directly: at 1/20 it drops to 15.9 MB, the cost per question to 2.6 MB. The cost sits in the right-hand columns — 395 of the 411 slow requests go unrecorded. The decision is made at the root, before the outcome is known, so no request is kept because it turned out to be slow; whether the reported request happens to be on hand is a matter of the die. The cost of an unanswered question is read from here. When "which span is slow" cannot be answered, the intervention looks at all five services at once; with a trace, there is exactly one span to look at. When the question goes unanswered because of sampling, the cost is not divided evenly: at 1/20, the question stays unanswered for 96% of slow requests. #### Summary - A trace tree is the parent-child bond between spans; without the bond, the total duration is just a number. - Of the eleven questions, the metric view answered 3, the log view 4, the trace view 9; of the 7 questions the metric and log could not answer, 6 were answered by the trace. - One question was answered by none of the three views — the query inside the slow span; what is missing is not collection but instrumentation depth. - The cost of tracing is measured by span count: 111 bytes per span, 316.0 MB per day, 52.7 MB per question answered only by the trace. - Sampling at 1/20, decided at the root, cuts volume to 15.9 MB but leaves 395 of the 411 slow requests unrecorded; which request survives has nothing to do with the question's importance. #### Next Step In this lesson, the span records of five services merged into a single tree, because all of them used the same field names: `service`, `duration`, `attempt`. In a real network this merge does not happen on its own; five teams write their own naming, one keeps duration in milliseconds and another in microseconds, the same concept splits into three names, and two spans placed side by side cannot be compared. The next lesson defines a shared telemetry convention field by field and counts its cost: how many questions become answerable in exchange for how many fields and how many service changes, and how many services stay incompatible. ### Instrumentation Standards Source: https://academia.sh/en/courses/devops-observability/instrumentation-standards The telemetry five services write in their own naming cannot be merged across services: ten questions are answered separately over local schemas and the shared convention, silently wrong answers are counted, and the convention's cost is worked out as fields, service changes, and an incompatible service. In the previous lesson, the span records of five services merged into a single tree. The reason for the merge was written into the code: every service used the same field names, the same unit, and the same trace id. That assumption has no counterpart in a real network. Five teams write five services; each team names its own record by its own habit, writes duration in its own unit, marks error the way it knows. The records are correct one by one, and each team sees the correct numbers on its own dashboard. What breaks is the cross-service question. This lesson has the five services of the regional measurement network (fiction) write in their own local schemas, asks the same ten questions over these schemas first, then defines a shared telemetry convention and asks them again. What a convention is is explained by its shared structure; the schema is defined within the lesson. #### Five Records of the Same Request ```js // convention/record.mjs — MODEL of five services writing the same request in their own local schema. // The real values are generated first (ground truth), then each service writes a record in its own // field names and units. MD7: the local schemas are fiction. MD8: read and billing write the same // "duration" name in different units. MD9: billing carries its own request number, not the id from // the chain. MD10: workorder is a component brought in from outside; its record cannot be changed. export const SEED = 20260803, REQUESTS = 6000; export const SERVICES = ["gateway", "read", "validate", "billing", "workorder"]; export const BASE = { gateway: 6, read: 21, validate: 28, billing: 47, workorder: 19 }; export const FLOW = "billing_query"; export function real() { let s = SEED; const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); const output = []; for (let i = 0; i < REQUESTS; i += 1) { const t0 = Date.UTC(2026, 7, 3) + Math.floor(r() * 3600000), skipped = r() < 0.11, spans = []; for (const name of SERVICES) { if (name === "validate" && skipped) continue; spans.push({ service: name, duration_ms: Math.max(1, Math.round(BASE[name] * (0.6 + r() * 0.9))), status: r() < ({ billing: 0.014, workorder: 0.0005 }[name] ?? 0.004) ? "error" : "ok" }); } output.push({ trace: `trace-${(i + 1).toString(36).padStart(4, "0")}`, t0, skipped, spans }); } return output; } // Local schemas: the same information written with five different names and three different units. export const LOCAL = { gateway: (z, a) => ({ time: new Date(z.t0).toISOString(), service: "gateway", operation: FLOW, duration_ms: a.duration_ms, status: a.status, trace: z.trace }), read: (z, a) => ({ t: Math.floor(z.t0 / 1000), unit: "read", action: "query", duration: a.duration_ms * 1000, result: a.status === "error" ? 1 : 0, tracing: z.trace }), validate: (z, a) => ({ ts: z.t0, component: "validate", operation: "check", latency: a.duration_ms / 1000, error: a.status === "error", trace_id: z.trace }), billing: (z, a) => ({ time: new Date(z.t0).toISOString(), app: "billing", endpoint: "compute", duration: a.duration_ms, status_code: a.status === "error" ? 500 : 200, request_no: `f-${z.trace.slice(6)}` }), workorder: (z, a) => ({ time: z.t0, service: "workorder", procedure: "open_order", operation_duration: a.duration_ms, error_code: a.status === "error" ? 7 : 0 }), }; export const localFlow = (G) => G.flatMap((z) => z.spans.map((a) => LOCAL[a.service](z, a))); if (import.meta.filename === process.argv[1]) { const G = real(), z = G[0]; console.log(`seed ${SEED}, ${REQUESTS} requests, ${SERVICES.length} services, ` + `${G.reduce((x, y) => x + y.spans.length, 0)} records`); console.log(`\nrecords written for one request (${z.trace}) across five services:`); for (const a of z.spans) console.log(` ${JSON.stringify(LOCAL[a.service](z, a))}`); const fields = new Map(); for (const a of z.spans) for (const k of Object.keys(LOCAL[a.service](z, a))) fields.set(k, (fields.get(k) || 0) + 1); console.log(`\n${fields.size} distinct field names in total, common to all five services: ` + `${[...fields].filter(([, n]) => n === 5).map(([k]) => k).join(", ") || "none"}`); } ``` ``` seed 20260803, 6000 requests, 5 services, 29362 records records written for one request (trace-0001) across five services: {"time":"2026-08-03T00:21:34.006Z","service":"gateway","operation":"billing_query","duration_ms":5,"status":"ok","trace":"trace-0001"} {"t":1785716494,"unit":"read","action":"query","duration":30000,"result":0,"tracing":"trace-0001"} {"ts":1785716494006,"component":"validate","operation":"check","latency":0.035,"error":false,"trace_id":"trace-0001"} {"time":"2026-08-03T00:21:34.006Z","app":"billing","endpoint":"compute","duration":54,"status_code":200,"request_no":"f-0001"} {"time":1785716494006,"service":"workorder","procedure":"open_order","operation_duration":25,"error_code":0} 24 distinct field names in total, common to all five services: none ``` In the five records of a single request, three separations show up at once. Field name: the same concept appears under the names `duration_ms`, `duration`, `latency`, `operation_duration` across five services; a total of 24 distinct field names are counted, and there is not one field common to all five services. Unit: duration is written in three different units — milliseconds, microseconds, seconds — and the field name does not say which. Value space: error is marked across the five services as the text `"error"`, the number `1`, the boolean `true`, the code `500`, and the code `7`. The fourth separation is in identity: billing carries its own request number, not the trace id coming from the chain, and workorder carries no id at all. #### The Convention's Shared Structure A telemetry convention is not a tool, it is an agreement: it fixes which field is written under which name, in which unit, in which value space, and how the request's id is carried from call to call. The six-field schema used in this lesson is as follows: | Field | Meaning | Unit and value space | |---|---|---| | `trace` | the request's id, carried across the call chain | text, the same across five services | | `time_ms` | the span's start | epoch milliseconds, integer | | `service` | the service that wrote the record | name from the catalog | | `flow` | the workflow the span belongs to | shared flow name | | `duration_ms` | the span's duration | milliseconds | | `status` | the span's outcome | `ok` or `error` | The alternative to a convention is a mapping table hand-written on the query side: "if you see this field name, count it as duration; if you see this value, count it as an error." The comparison below measures both approaches with the same set of questions. #### Ten Questions, Two Stages ```js // convention/query.mjs — ten questions are asked in two stages: first over the local schemas with a // hand-written mapping table (MD11), then over records moved to the shared convention. Every answer // is compared with the real value: "wrong" if the deviation exceeds 5%, "none" if no field reads. import { real, LOCAL, SERVICES, FLOW, REQUESTS } from "./record.mjs"; const G = real(); // MD11: the mapping table uses the first field it finds; it does not know the value space. const MAPPING = { trace: ["trace", "tracing", "trace_id", "request_no"], time_ms: ["time_ms", "time", "ts", "t"], service: ["service", "unit", "component", "app"], flow: ["flow", "operation", "action", "procedure", "endpoint"], duration_ms: ["duration_ms", "duration", "latency", "operation_duration"], status: ["status", "error", "result", "status_code", "error_code"] }; const lookup = (k, field) => { for (const a of MAPPING[field]) if (a in k) return k[a]; return null; }; const num = (v) => (typeof v === "number" && Number.isFinite(v) ? v : null); const isError = (v) => v === "error" || v === true; // the two forms the alias table's author knows // MD12: the pass to the convention happens where the record is generated; workorder (MD10) stays local. const CONTRACT = ["trace", "time_ms", "service", "flow", "duration_ms", "status"], INCOMPATIBLE = ["workorder"]; const conventional = (z, a) => (INCOMPATIBLE.includes(a.service) ? LOCAL[a.service](z, a) : { trace: z.trace, time_ms: z.t0, service: a.service, flow: FLOW, duration_ms: a.duration_ms, status: a.status }); const records = (f) => G.flatMap((z) => z.spans.map((a) => f(z, a))); const [LOCAL_K, CONV_K] = [records((z, a) => LOCAL[a.service](z, a)), records(conventional)]; const avg = (x) => (x.length ? x.reduce((a, b) => a + b, 0) / x.length : null); const p95 = (x) => (x.length ? x.slice().sort((a, b) => a - b)[Math.floor(x.length * 0.95)] : null); const groupBy = (K, field) => { const m = new Map(); for (const k of K) { const g = lookup(k, field); if (g !== null) m.set(g, [...(m.get(g) ?? []), k]); } return m; }; const durationOf = (k) => num(lookup(k, "duration_ms")) ?? 0; const traceTotal = (K) => [...groupBy(K, "trace").values()].map((v) => v.reduce((x, k) => x + durationOf(k), 0)); const serviceDurations = (K, s) => (groupBy(K, "service").get(s) ?? []).map((k) => num(lookup(k, "duration_ms"))) .filter((v) => v !== null); const slowestService = (K) => SERVICES.map((s) => [s, p95(serviceDurations(K, s)) ?? -1]).sort((a, b) => b[1] - a[1])[0]; const realService = (s) => G.flatMap((z) => z.spans.filter((a) => a.service === s).map((a) => a.duration_ms)); const allSpans = G.flatMap((z) => z.spans), endToEnd = G.map((z) => z.spans.reduce((x, a) => x + a.duration_ms, 0)); const QUESTIONS = [ ["slowest service", "billing", (K) => slowestService(K)[0]], ["slowest service's p95 (ms)", p95(realService("billing")), (K) => slowestService(K)[1]], ["service producing the most errors", "billing", (K) => SERVICES.map((s) => [s, (groupBy(K, "service").get(s) ?? []).filter((k) => isError(lookup(k, "status"))).length]) .sort((a, b) => b[1] - a[1])[0][0]], ["error rate (%)", allSpans.filter((a) => a.status === "error").length / allSpans.length * 100, (K) => K.filter((k) => isError(lookup(k, "status"))).length / K.length * 100], ["requests per hour", REQUESTS, (K) => ((g) => (g.length ? g.length : null))( (groupBy(K, "service").get("gateway") ?? []).filter((k) => num(+lookup(k, "time_ms")) !== null))], ["end-to-end average duration (ms)", avg(endToEnd), (K) => avg(traceTotal(K))], ["records merged per request", avg(G.map((z) => z.spans.length)), (K) => avg([...groupBy(K, "trace").values()].map((v) => v.length))], ["requests with validation skipped", G.filter((z) => z.skipped).length, (K) => [...groupBy(K, "trace").values()].filter((v) => v.every((k) => lookup(k, "service") !== "validate")).length], [`${FLOW} flow's total duration (ms)`, endToEnd.reduce((a, b) => a + b, 0), (K) => (groupBy(K, "flow").get(FLOW) ?? []).reduce((x, k) => x + durationOf(k), 0) || null], ["workorder span's share of end-to-end (%)", avg(realService("workorder")) / avg(endToEnd) * 100, (K) => { const m = groupBy(K, "trace"), i = [...m.values()].flat() .filter((k) => lookup(k, "service") === "workorder" && lookup(k, "trace") !== null); return i.length ? avg(i.map(durationOf)) / avg(traceTotal(K)) * 100 : null; }], ]; const format = (v) => (v === null ? "-" : typeof v === "number" ? v.toFixed(v < 100 ? 2 : 0) : String(v)); const verdict = (g, v) => (v === null ? "none" : typeof g === "number" ? (Math.abs(v - g) / g <= 0.05 ? "correct" : "wrong") : (v === g ? "correct" : "wrong")); console.log(`${"question".padEnd(42)}${"real".padStart(9)}${"local schema".padStart(14)}` + `${"status".padStart(8)}${"convention".padStart(12)}${"status".padStart(9)}`); const tally = { local: {}, conv: {} }; for (const [label, g, f] of QUESTIONS) { const [y, s] = [f(LOCAL_K), f(CONV_K)], [dy, ds] = [verdict(g, y), verdict(g, s)]; tally.local[dy] = (tally.local[dy] || 0) + 1; tally.conv[ds] = (tally.conv[ds] || 0) + 1; console.log(`${label.padEnd(42)}${format(g).padStart(9)}${format(y).padStart(14)}${dy.padStart(8)}` + `${format(s).padStart(12)}${ds.padStart(9)}`); } const summarize = (o) => `correct ${o.correct || 0}, wrong ${o.wrong || 0}, none ${o.none || 0}`; console.log(`\nlocal schema: ${summarize(tally.local)}\nconvention : ${summarize(tally.conv)}`); const fieldChanges = SERVICES.map((s) => [s, INCOMPATIBLE.includes(s) ? 0 : CONTRACT.filter((a) => !(a in LOCAL[s](G[0], G[0].spans[0]))).length]); console.log(`convention ${CONTRACT.length} fields; changed fields: ${fieldChanges.map(([s, n]) => `${s} ${n}`).join(", ")}` + `; total ${fieldChanges.reduce((x, [, n]) => x + n, 0)} field changes, ` + `incompatible ${INCOMPATIBLE.length}/${SERVICES.length} services`); ``` ``` question real local schema status convention status slowest service billing read wrong billing correct slowest service's p95 (ms) 68.00 31000 wrong 68.00 correct service producing the most errors billing validate wrong billing correct error rate (%) 0.56 0.18 wrong 0.54 correct requests per hour 6000 - none 6000 correct end-to-end average duration (ms) 124 11141 wrong 104 wrong records merged per request 4.89 1.95 wrong 3.89 wrong requests with validation skipped 638 6638 wrong 638 correct billing_query flow's total duration (ms) 744212 37943 wrong 625249 wrong workorder span's share of end-to-end (%) 15.99 - none - none local schema: correct 0, wrong 8, none 2 convention : correct 6, wrong 3, none 1 convention 6 fields; changed fields: gateway 2, read 6, validate 6, billing 6, workorder 0; total 20 field changes, incompatible 1/5 services ``` Over the local schemas, none of the ten questions was answered correctly. The real finding is in the distribution: only two stayed unanswered, eight were answered silently wrong. The mapping table mistook the read service's microsecond value for milliseconds and found a p95 of 31,000 instead of 68, showing read instead of billing as the slowest service. The error rate came out as 0.18% instead of 0.56%, because the table only recognizes two of the encodings. The question about which service produces the most errors was answered "validate" instead of "billing". The end-to-end duration appeared as 11,141 ms instead of 124 ms. None of these answers carries a warning; the query runs, produces a number, and is drawn on the dashboard. The two unanswered questions are cheaper for exactly this reason: requests per hour could not be computed at all, because the gateway writes time as text and it could not be converted to a number. The absence of an answer is visible; a wrong answer is not. Once the convention was applied, six questions were answered correctly. All four that remain trace back to a single source: the service that could not be moved to the convention. The end-to-end duration came out as 104 ms instead of 124, records merged per request as 3.89 instead of 4.89, the flow total as 625,249 instead of 744,212 ms — all three are the gap left by one missing span. The workorder span's share of the end-to-end total, though, still goes unanswered: that service's record carries no trace id, so the span cannot be tied to any request. #### The Convention's Cost The convention consists of six fields. Four services moved to these fields, and that move amounts to 20 field changes in total: gateway added two fields, the other three services each renamed or converted the unit of six fields. The fifth service could not be changed because it is a component brought in from outside, and it stayed incompatible. The ratio of these three numbers carries the decision. The gain per field is high: 20 field changes bought a rise from zero correct answers to six, meaning one correct answer cost an average of about 3.3 field changes. Against that, the gain is not linear: 80% of the services conformed to the convention, but only 60% of the questions were answered, because a cross-service question depends on the weakest link. A single incompatible span in a request chain breaks every question that includes that span. The cost of the unanswered question is two-layered here. The first layer is direct: because the workorder span's share is unknown, that service's slowdown does not show up in the end-to-end measurement. The second layer is more expensive: before the convention, eight questions were answered wrong, and decisions were made on those answers — for as long as read is shown as the slowest service, the effort to improve it is spent on the wrong service. #### Summary - When five services write in their own naming, a single request scatters across 24 distinct field names, and no field remains common to all five services. - Over the local schemas, zero of the ten questions were answered correctly; two stayed unanswered, eight were answered silently wrong — a wrong answer is more expensive than no answer, because it is not visible. - The separation shows up in three layers at once: field name, unit (ms, µs, s), and value space (text, number, boolean, code); an identity separation on top of these blocks merging entirely. - The six-field convention raised correct answers to six in exchange for 20 field changes across four services. - 80% of the services conformed, but only 60% of the questions were answered: the single incompatible service breaks all four questions that include its span. #### Next Step Once the convention is in place, questions become answerable, but where the answer is read remains open. A person does not write a query during an incident; they look at the panels in front of them. Which question a panel answers, though, is usually never asked — dashboards fill up over time with panels that get added and that nobody deletes. The next lesson audits a dashboard against a set of questions: how many panels answer a question, how many panels answer none, how many questions remain without a panel, and how many panels it takes to browse through to find an answer to a question. ### Dashboard Design Source: https://academia.sh/en/courses/devops-observability/dashboard-design Auditing a dashboard against a set of questions: how many panels answer a question, how many panels answer none, how many questions remain without a panel, and how reordering brings down the number of panels browsed to find an answer to a question. Once the convention was in place, questions became answerable. How the answer is read is a separate matter: nobody writes a query during an incident, they look at the panels in front of them. The dashboard itself grows over time too — a panel gets added after every incident, no panel gets deleted after any incident, and two years later the result is a page nobody knows the whole of. This lesson audits a dashboard against a set of questions. Three numbers are measured: how many panels answer at least one question, how many panels answer none, how many questions remain without a panel. A fourth number follows from these: how many panels it takes to browse through to find an answer to a question. #### The Question a Panel Answers A panel answers a question only if it meets three conditions at once. The metric must be the same: a latency panel does not answer an error-rate question. The dimension must include the breakdown the question needs: a total error-rate panel cannot answer "which service is broken", because the breakdown is not in the panel's query. The resolution must be as fine as the question needs: a panel drawn with five-minute buckets does not answer a question asking about a one-minute window. These three conditions sit as fields in the model below. The regional measurement network's panel and question list is fiction; the question frequencies are put in to weight how the dashboard gets looked at. ```js // dashboard/dashboard.mjs — MODEL of a dashboard and the question set asked in operations. Panel: [name, metric, // dims, resolution sec, series scanned by the query]. Question: [name, metric, required dim, coarsest resolution // required, how many of forty incidents it is asked in]. MD13: the panel catalog, question set, and question // frequencies are fiction; the dashboard has grown by additions over two years. MD14: a panel answers a question // only if it carries the same metric, the dimension the question needs, and that resolution. MD15: scanning one // panel by eye and moving on takes 8 seconds; writing a manual query for a panelless question takes 4 minutes. export const PANELS = [ ["cluster CPU usage", "cpu", ["node"], 60, 120], ["cluster memory usage", "memory", ["node"], 60, 120], ["node disk usage", "disk", ["node"], 300, 120], ["total requests (daily)", "request", [], 3600, 1], ["request count", "request", ["service"], 60, 5], ["error rate (total)", "error_rate", [], 60, 1], ["version info", "version", [], 3600, 5], ["record count (today)", "record", [], 3600, 1], ["p95 latency (total)", "latency", [], 60, 1], ["p95 latency (service)", "latency", ["service"], 60, 5], ["span duration (service)", "span_duration", ["service"], 60, 5], ["client version distribution", "client", ["version"], 3600, 40], ["node count", "node_count", [], 300, 1], ["queue depth", "queue", ["service"], 60, 5], ["cache hit rate", "cache", ["service"], 60, 5], ["garbage collection duration", "gc", ["node"], 60, 120], ["error rate (service)", "error_rate", ["service"], 60, 5], ["retry count", "retry", [], 60, 1], ["network traffic", "network", ["node"], 60, 240], ["deployment events", "deploy", ["service"], 60, 5], ["nightly batch job duration", "batch", [], 3600, 1], ["hourly read count", "read", [], 3600, 1], ["dependency latency", "span_duration", ["dependency"], 60, 12], ["error rate (region)", "error_rate", ["region"], 300, 30], ["meter type distribution", "meter_type", ["type"], 3600, 8], ["CPU (single node)", "cpu", ["node"], 60, 1], ]; // Question set: what is asked, in order, during an incident (priority order = array order). export const QUESTIONS = [ ["is a member affected", "error_rate", [], 60, 40], ["which service is broken", "error_rate", ["service"], 60, 38], ["which span has the latency", "span_duration", ["service"], 60, 30], ["did traffic change", "request", ["service"], 60, 25], ["is there a new deployment", "deploy", ["service"], 60, 22], ["is the queue backing up", "queue", ["service"], 60, 16], ["did retries increase", "retry", [], 60, 12], ["is the external dependency slow", "span_duration", ["dependency"], 60, 10], ["is the error specific to one region", "error_rate", ["region"], 60, 9], ["was the nightly batch job affected", "batch", [], 3600, 6], ["did the cache hit rate drop", "cache", ["service"], 60, 5], ["which subscriber group is affected", "error_rate", ["subscriber_group"], 60, 4], ]; export const PANEL_SECONDS = 8, QUERY_MINUTES = 4, INCIDENTS = 40; export const answers = (p, q) => p[1] === q[1] && q[2].every((b) => p[2].includes(b)) && p[3] <= q[3]; export const audit = (panels) => { const found = QUESTIONS.map((q) => panels.findIndex((p) => answers(p, q)) + 1); // 0: no panel const used = panels.filter((p) => QUESTIONS.some((q) => answers(p, q))); const weight = QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] : 0), 0); const timeToAnswer = QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] * found[i] : 0), 0) / weight; const minutes = (QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] * found[i] * PANEL_SECONDS : q[4] * QUERY_MINUTES * 60), 0)) / INCIDENTS / 60; return { panels: panels.length, used: used.length, decorative: panels.length - used.length, answered: found.filter((b) => b > 0).length, noPanel: QUESTIONS.filter((_, i) => found[i] === 0).map((q) => q[0]), timeToAnswer, minutes, series: panels.reduce((a, p) => a + p[4], 0), found }; }; if (import.meta.filename === process.argv[1]) { const d = audit(PANELS); console.log(`${d.panels} panels, ${QUESTIONS.length} questions`); console.log(`panels answering a question ${d.used}, answering none ${d.decorative}`); console.log(`questions answered ${d.answered}, questions with no panel ${d.noPanel.length}: ${d.noPanel.join(", ")}`); console.log(`time to answer (panels scanned, weighted by question frequency) ${d.timeToAnswer.toFixed(1)}, ` + `worst ${Math.max(...d.found)}`); console.log(`dashboard minutes per incident ${d.minutes.toFixed(1)}, series scanned on open ${d.series}\n`); console.log(`${"question".padEnd(37)}${"panels scanned".padStart(17)} panel that answered`); QUESTIONS.forEach((q, i) => console.log(`${q[0].padEnd(37)}${(d.found[i] || "-").toString().padStart(17)} ` + `${d.found[i] ? PANELS[d.found[i] - 1][0] : "none"}`)); const decorative = PANELS.filter((p) => !QUESTIONS.some((q) => answers(p, q))); console.log(`\ndecorative panels (${decorative.length}): ${decorative.map((p) => p[0]).join(", ")}`); console.log(`series scanned by decorative panels ${decorative.reduce((a, p) => a + p[4], 0)} ` + `(${(decorative.reduce((a, p) => a + p[4], 0) / d.series * 100).toFixed(0)}% of the total)`); } ``` ``` 26 panels, 12 questions panels answering a question 10, answering none 16 questions answered 10, questions with no panel 2: is the error specific to one region, which subscriber group is affected time to answer (panels scanned, weighted by question frequency) 13.0, worst 23 dashboard minutes per incident 10.1, series scanned on open 859 question panels scanned panel that answered is a member affected 6 error rate (total) which service is broken 17 error rate (service) which span has the latency 11 span duration (service) did traffic change 5 request count is there a new deployment 20 deployment events is the queue backing up 14 queue depth did retries increase 18 retry count is the external dependency slow 23 dependency latency is the error specific to one region - none was the nightly batch job affected 21 nightly batch job duration did the cache hit rate drop 15 cache hit rate which subscriber group is affected - none decorative panels (16): cluster CPU usage, cluster memory usage, node disk usage, total requests (daily), version info, record count (today), p95 latency (total), p95 latency (service), client version distribution, node count, garbage collection duration, network traffic, hourly read count, error rate (region), meter type distribution, CPU (single node) series scanned by decorative panels 814 (95% of the total) ``` Ten of the twenty-six panels answer a question, sixteen answer none. These sixteen panels are not wrong — cluster CPU, node memory, client version distribution all draw correct numbers. But they have no counterpart in this operations question set, so for the person looking during an incident they are just ground to cover. Their cost is not just taking up space either: of the 859 series scanned when the dashboard opens, 814, that is 95%, come from panels that answer no question. Nearly the entire query cost buys nothing. The reason the ratio is this high is readable too: node-broken-down panels scan 120 series each, service-broken-down panels scan 5 each. How fine the breakdown is does not show up in the panel's visual size, it shows up in the query cost. The label "decorative" is not absolute, it is relative to the question set. The cluster CPU and node memory panels are the exact counterpart of capacity questions; the audit here measures not capacity but what gets asked during an incident. That is why these panels are not deleted, they move to a separate dashboard — and that dashboard's own audit is run with its own question set. The same distinction explains why two CPU panels stand side by side on the dashboard: one was added for the cluster, the other for a single node, both answer the same question, and neither has a counterpart on the incident dashboard. Two questions are without a panel. That the region-breakdown question is left without a panel is surprising at first, because there is a region-broken-down error-rate panel on the dashboard — but it is drawn at five-minute resolution and the question asks for a one-minute window. The panel exists, the answer does not. The second panelless question ties to something else entirely: the subscriber-group dimension was never collected, so that question can only be opened by instrumentation, not by dashboard layout. Time to answer in this layout is 13.0 panels. That is, to find an answer to a question, weighted by the question's frequency, an average of thirteen panels are scanned by eye; twenty-three for the worst question. At eight seconds per panel, that comes to 10.1 minutes per incident. #### Reordering the Dashboard The reordering is done in four steps, and the same audit runs again after every step. ```js // dashboard/reorder.mjs — the dashboard is reordered against the question set. Four stages are applied in // sequence and the same audit runs again after each one. MD16: decorative panels are not deleted, they move // from the incident dashboard to a capacity dashboard; their cost there is that dashboard's own series scan. import { PANELS, QUESTIONS, audit, answers, INCIDENTS, QUERY_MINUTES } from "./dashboard.mjs"; const copy = PANELS.map((p) => [...p]); const resolutionFix = copy.map((p) => (p[0] === "error rate (region)" ? [p[0], p[1], p[2], 60, p[4]] : p)); const kept = resolutionFix.filter((p) => QUESTIONS.some((q) => answers(p, q))); // Ordering rule: questions are walked in frequency order; for each question, the first // not-yet-placed panel that answers it is put in the queue; the rest are appended at the end. const ordered = []; for (const q of QUESTIONS) { const p = kept.find((x) => answers(x, q) && !ordered.includes(x)); if (p) ordered.push(p); } for (const p of kept) if (!ordered.includes(p)) ordered.push(p); const STAGES = [["before", copy], ["resolution fix", resolutionFix], ["decorative panels split off", kept], ["ordered by question", ordered]]; console.log(`${"stage".padEnd(29)}${"panels".padStart(8)}${"decorative".padStart(12)}${"answered".padStart(11)}` + `${"no panel".padStart(10)}${"time to answer".padStart(16)}${"min/incident".padStart(14)}${"series".padStart(8)}`); const results = STAGES.map(([name, p]) => { const d = audit(p); console.log(`${name.padEnd(29)}${String(d.panels).padStart(8)}${String(d.decorative).padStart(12)}` + `${String(d.answered).padStart(11)}${String(d.noPanel.length).padStart(10)}` + `${d.timeToAnswer.toFixed(1).padStart(16)}${d.minutes.toFixed(1).padStart(14)}${String(d.series).padStart(8)}`); return d; }); const [first, last] = [results[0], results[3]]; console.log(`\n${"question".padEnd(37)}${"before".padStart(7)}${"after".padStart(7)}`); QUESTIONS.forEach((q, i) => console.log(`${q[0].padEnd(37)}${(first.found[i] || "-").toString().padStart(7)}` + `${(last.found[i] || "-").toString().padStart(7)}`)); console.log(`\ntime to answer went from ${first.timeToAnswer.toFixed(1)} panels to ${last.timeToAnswer.toFixed(1)} panels, ` + `dashboard minutes per incident from ${first.minutes.toFixed(1)} min to ${last.minutes.toFixed(1)} min`); console.log(`gained across forty incidents: ${((first.minutes - last.minutes) * INCIDENTS).toFixed(0)} minutes; ` + `series scanned ${first.series} instead of ${last.series}`); const remaining = QUESTIONS.filter((q) => !ordered.some((p) => answers(p, q))); console.log(`still without a panel: ${remaining.length} questions: ${remaining.map((q) => q[0]).join(", ")} — ` + `${remaining.reduce((a, q) => a + q[4], 0)} incidents, ${remaining.reduce((a, q) => a + q[4] * QUERY_MINUTES, 0)} minutes of manual queries`); ``` ``` stage panels decorative answered no panel time to answer min/incident series before 26 16 10 2 13.0 10.1 859 resolution fix 26 15 11 1 13.5 10.0 859 decorative panels split off 11 0 11 1 4.8 3.8 75 ordered by question 11 0 11 1 4.1 3.3 75 question before after is a member affected 6 1 which service is broken 17 2 which span has the latency 11 3 did traffic change 5 4 is there a new deployment 20 5 is the queue backing up 14 6 did retries increase 18 7 is the external dependency slow 23 8 is the error specific to one region - 9 was the nightly batch job affected 21 10 did the cache hit rate drop 15 11 which subscriber group is affected - - time to answer went from 13.0 panels to 4.1 panels, dashboard minutes per incident from 10.1 min to 3.3 min gained across forty incidents: 273 minutes; series scanned 859 instead of 75 still without a panel: 1 questions: which subscriber group is affected — 4 incidents, 16 minutes of manual queries ``` The first step fixes the resolution of a single panel, and the number of answered questions rises from 10 to 11. What stands out is that time to answer increases at this step: from 13.0 to 13.5. The newly answered question is found in a panel at the twenty-fourth position on the dashboard, and it pulls the average up. Answering one more question does not mean that answer is findable. The real drop is at the second step. Once the fifteen panels answering no question are pulled from the incident dashboard and moved to the capacity dashboard, the panel count drops from 26 to 11, time to answer from 13.5 to 4.8, series scanned from 859 to 75. The third step orders the panels by question frequency and brings time to answer down to 4.1; now every question is found at its own position, and the most frequently asked question is at the first panel. The total gain is dashboard time dropping from 10.1 minutes to 3.3 minutes per incident; 273 minutes across forty incidents. All of that time is time added to the incident's length — the person browsing panels is not doing anything else at that moment. Measuring time to answer by panel count is a simplification: in reality a person does not scan the dashboard from start to end, they jump to where they know. What the simplification holds onto is the ratio: the number of panels someone who does not know, or who just woke up, browses through is directly proportional to the dashboard's layout. It is not the number itself that should be read, but the nearly threefold difference between the two layouts. #### The Question Left Over After the reordering, the one question left without a dashboard is the subscriber-group breakdown, and it is not closed by dashboard layout. It is asked in four of forty incidents, each time a manual query has to be written and it takes four minutes: 16 minutes across forty incidents. That number looks small, but how the question gets answered matters too — the result of a manually written query does not go into the dashboard, and the same query gets written again at the next incident. The three numbers the audit produces are read together. Data that answers no question is cost: 62% of the panels, 95% of the query cost. A question left unanswered is risk: of the two questions, one was closed off by the wrong resolution, the other by an uncollected dimension. Where the answer sits is also cost: the same data, the same panels, gained more than six minutes per incident just by changing the order. #### Summary - A panel answers a question only if it meets three conditions at once: the same metric, the dimension the question needs, the resolution the question needs. - 16 of the 26 panels answered no question; these panels accounted for 814 of the 859 series scanned on dashboard open, that is 95% of the query cost. - Two of the twelve questions were without a panel: one from the wrong resolution (the panel existed, the answer did not), the other from a dimension that was never collected. - The resolution fix raised answered questions to 11 but raised time to answer from 13.0 to 13.5; an answer existing does not mean it is findable. - Splitting off the decorative panels and ordering panels by question brought time to answer down to 4.1 panels, dashboard minutes per incident from 10.1 to 3.3; 273 minutes across forty incidents. - Whether a panel is decorative is not absolute, it is relative to the question set: panels that are split off are not deleted, they move to a separate dashboard with its own question set. #### Next Step A reordered dashboard only works when it is looked at. If nobody is looking at three in the morning, the ordering of eleven panels changes nothing; someone has to be paged. The rule that triggers the page, though, is not written in a single form: it can page when a threshold is crossed, when a rate changes at a certain speed within a certain window, or when a band learned from past behavior is left. Three rule types catch different incidents within the same incident set. The next lesson runs all three over the same known incident sequence and compares them by caught count, missed count, false-alarm count, and detection round. ### Alerting Rules Source: https://academia.sh/en/courses/devops-observability/alerting-rules Running three rule types — threshold, rate, and anomaly — over the same eight known incidents of the same ten days: each type's parameter is swept and the caught, missed, false-alarm, and detection-round counts are taken, and where the three types complement each other is measured through rule-set combinations. A reordered dashboard only works when it is looked at. If nobody is looking at three in the morning, the panel ordering changes nothing; someone has to be paged. Which metric the paging rule should be written from — the component's internal metric or the metric the member sees — was measured in a separate lesson and is not repeated here. The question here is the rule's form: three different rule types can be written on the same metric, and the three do not catch the same incidents. The threshold rule compares the value against a fixed bound. The rate rule looks at how fast the value changes compared with one window earlier. The anomaly rule looks for a move outside a band learned from the past. All three are run over the regional measurement network's (fiction) two ten-day metric series and eight known incidents. #### Ten Days, Eight Incidents ```js // alert/series.mjs — MODEL of two minute-by-minute metric series over ten days and eight known incidents. // MD19: request count carries a daily pattern (low at night, peak in the afternoon); error rate is // patternless. MD20: both series have harmless short spikes (once every five thousand minutes, 2-4 min). // MD21: the eight incidents are known incidents; their windows and target values are written below. export const SEED = 20260804, DAYS = 10, MINUTES = DAYS * 1440; export const HOUR_PATTERN = [40, 32, 28, 26, 30, 48, 78, 118, 150, 168, 176, 172, 165, 170, 178, 174, 160, 138, 120, 104, 92, 78, 62, 50]; // [day, hour, duration min, name, [[series, target value, shape]]] — shape: "spike" or "ramp" export const INCIDENTS = [ [2, 11, 20, "sudden error spike", [["error", 6.0, "spike"]]], [3, 9, 300, "slow error ramp", [["error", 1.4, "ramp"]]], [4, 3, 90, "overnight traffic loss", [["request", 8, "spike"]]], [5, 14, 4, "short deep outage", [["error", 9.0, "spike"]]], [6, 8, 600, "steady drift", [["error", 0.9, "spike"]]], [7, 16, 45, "post-release increase", [["error", 2.6, "spike"]]], [8, 1, 300, "overnight request ramp", [["request", 110, "ramp"]]], [9, 6, 360, "gradual traffic decline", [["request", 0.45, "ramp"]]], ]; export const window = (i) => [INCIDENTS[i][0] * 1440 + INCIDENTS[i][1] * 60, INCIDENTS[i][2]]; export function series() { let s = SEED; const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); const S = { request: new Float64Array(MINUTES), error: new Float64Array(MINUTES) }; const spike = { request: 0, error: 0 }; for (let t = 0; t < MINUTES; t += 1) { const hour = Math.floor(t / 60) % 24, base = HOUR_PATTERN[hour]; S.request[t] = base * (0.94 + r() * 0.12); S.error[t] = 0.2 + (r() * 2 - 1) * 0.08; for (const name of ["request", "error"]) { if (spike[name] > 0) { spike[name] -= 1; S[name][t] *= name === "error" ? 4.2 : 1.35; } else if (r() < 1 / 5000) spike[name] = 2 + Math.floor(r() * 3); } for (let i = 0; i < INCIDENTS.length; i += 1) { const [start, dur] = window(i); if (t < start || t >= start + dur) continue; for (const [name, target, shape] of INCIDENTS[i][4]) { const p = shape === "ramp" ? (t - start + 1) / dur : Math.min(1, (t - start + 1) / 3); const eventBase = name === "request" ? HOUR_PATTERN[hour] : 0.2; const targetValue = name === "request" && target < 1 ? eventBase * target : target; S[name][t] = eventBase + (targetValue - eventBase) * p; } } } return S; } if (import.meta.filename === process.argv[1]) { const S = series(); const avg = (a, b, name) => { let x = 0; for (let t = a; t < b; t += 1) x += S[name][t]; return x / (b - a); }; console.log(`seed ${SEED}, ${DAYS} days, ${MINUTES} minutes, 2 metrics, ${INCIDENTS.length} known incidents`); console.log(`baseline: overnight error ${avg(0, 1440, "error").toFixed(2)}, request ` + `${avg(180, 300, "request").toFixed(0)}/min (03-05), afternoon request ` + `${avg(840, 960, "request").toFixed(0)}/min (14-16)\n`); console.log(`${"incident".padEnd(24)}${"day".padStart(5)}${"duration".padStart(10)}` + `${"peak value in incident".padStart(25)}`); INCIDENTS.forEach(([day, , dur, name, targets], i) => { const [b] = window(i), a = targets[0][0]; let peak = S[a][b]; for (let t = b; t < b + dur; t += 1) peak = a === "error" || targets[0][1] > 100 ? Math.max(peak, S[a][t]) : Math.min(peak, S[a][t]); console.log(`${name.padEnd(24)}${String(day).padStart(5)}${`${dur} min`.padStart(10)}` + `${`${a} ${peak.toFixed(a === "error" ? 2 : 0)}`.padStart(25)}`); }); } ``` ``` seed 20260804, 10 days, 14400 minutes, 2 metrics, 8 known incidents baseline: overnight error 0.20, request 28/min (03-05), afternoon request 176/min (14-16) incident day duration peak value in incident sudden error spike 2 20 min error 6.00 slow error ramp 3 300 min error 1.40 overnight traffic loss 4 90 min request 8 short deep outage 5 4 min error 9.00 steady drift 6 600 min error 0.90 post-release increase 7 45 min error 2.60 overnight request ramp 8 300 min request 110 gradual traffic decline 9 360 min request 71 ``` The incident set was deliberately chosen from different shapes: a twenty-minute sudden spike, a slow ramp spread over five hours, a four-minute deep outage, a steady drift lasting ten hours, and two of them on a seasonal metric — an overnight drop in traffic and an overnight ramp in requests. The last one, the gradual traffic decline, never looks low in absolute terms: it drops relative to the afternoon baseline of 176, but the number never falls below 71. #### Three Sweeps Each type is swept with a single parameter. The selection rule is written in the code: among the parameters that do not exceed a budget of at most twelve false alarms over ten days, the one catching the most incidents is chosen. ```js // alert/rule.mjs — three rule types are run over the same two series of the same ten days. MD22: a rule condition // produces an alert once it holds for K rounds (minutes) in a row, and does not produce again until the condition // drops; evaluation runs one round per minute. MD23: the threshold is written as a multiple of the median; the // rate rule looks at a two-times rise or a halving compared with W minutes earlier; the anomaly rule uses the band // from the previous L days for the same minute-of-day (mean plus four deviations, band floor 15% of the mean). // MD24: there is a budget of at most 12 false alarms over ten days, and for each type the parameter with the most // caught incidents among those not exceeding this budget is chosen. import { series, INCIDENTS, window, MINUTES } from "./series.mjs"; const S = series(), METRICS = ["error", "request"], STREAK = { threshold: 5, rate: 3, anomaly: 10 }, BUDGET = 12; const median = Object.fromEntries(METRICS.map((a) => [a, [...S[a]].sort((x, y) => x - y)[Math.floor(MINUTES / 2)]])); const conditions = { threshold: (k) => (a, t) => (a === "error" ? S[a][t] > k * median[a] : S[a][t] > k * median[a] || S[a][t] < median[a] / k), rate: (w) => (a, t) => { if (t < w) return false; const o = S[a][t] / Math.max(1e-9, S[a][t - w]); return o > 2 || o < 0.5; }, anomaly: (l) => (a, t) => { const g = Math.min(l, Math.floor(t / 1440)); if (g < 1) return false; const past = []; for (let j = 1; j <= g; j += 1) past.push(S[a][t - j * 1440]); const m = past.reduce((x, y) => x + y, 0) / g; const sd = Math.sqrt(past.reduce((x, y) => x + (y - m) ** 2, 0) / g); return Math.abs(S[a][t] - m) > 4 * Math.max(sd, 0.15 * m); }, }; function run(kind, p) { const alerts = []; const condition = conditions[kind](p); for (const a of METRICS) { let streak = 0, active = false; for (let t = 0; t < MINUTES; t += 1) { if (condition(a, t)) { streak += 1; if (streak >= STREAK[kind] && !active) { active = true; alerts.push([t - STREAK[kind] + 1, a]); } } else { streak = 0; active = false; } } } const insideEvent = (t) => INCIDENTS.findIndex((o, i) => t >= window(i)[0] && t < window(i)[0] + o[2]); const caught = new Map(); for (const [t] of alerts) { const i = insideEvent(t); if (i >= 0 && !caught.has(i)) caught.set(i, t - window(i)[0]); } const falseAlarms = alerts.filter(([t]) => insideEvent(t) < 0).length; const rounds = [...caught.values()]; return { p, alerts: alerts.length, caught, missed: INCIDENTS.length - caught.size, falseAlarms, round: rounds.length ? rounds.reduce((a, b) => a + b, 0) / rounds.length : null }; } const SWEEP = { threshold: [2, 3, 5, 8], rate: [10, 30, 60, 180], anomaly: [1, 3, 7] }; const HEADING = { threshold: "threshold (x median)", rate: "rate window (min)", anomaly: "learning window (days)" }; const chosen = {}; for (const kind of Object.keys(SWEEP)) { console.log(`${HEADING[kind].padEnd(25)}${"alerts".padStart(8)}${"caught".padStart(11)}` + `${"missed".padStart(8)}${"false alarms".padStart(14)}${"detection round".padStart(18)}`); const rows = SWEEP[kind].map((p) => run(kind, p)); for (const d of rows) console.log(`${String(d.p).padEnd(25)}${String(d.alerts).padStart(8)}` + `${`${d.caught.size}/${INCIDENTS.length}`.padStart(11)}${String(d.missed).padStart(8)}` + `${String(d.falseAlarms).padStart(14)}${(d.round === null ? "-" : `${d.round.toFixed(0)} rounds`).padStart(18)}`); const fits = rows.filter((d) => d.falseAlarms <= BUDGET); chosen[kind] = (fits.length ? fits : rows).slice() .sort((a, b) => b.caught.size - a.caught.size || a.falseAlarms - b.falseAlarms)[0]; console.log(`chosen: ${chosen[kind].p}\n`); } const KINDS = Object.keys(SWEEP); console.log(`${"incident".padEnd(25)}${KINDS.map((t) => t.padStart(12)).join("")} alone`); const alone = []; INCIDENTS.forEach((o, i) => { const h = KINDS.map((t) => chosen[t].caught.get(i)); const count = h.filter((x) => x !== undefined).length; if (count === 1) alone.push([o[3], KINDS[h.findIndex((x) => x !== undefined)]]); console.log(`${o[3].padEnd(25)}${h.map((x) => (x === undefined ? "-" : `${x} rounds`).padStart(12)).join("")}` + ` ${count === 1 ? KINDS[h.findIndex((x) => x !== undefined)] : count === 0 ? "none" : ""}`); }); const union = new Set(KINDS.flatMap((t) => [...chosen[t].caught.keys()])); console.log(`\nall three together ${union.size}/${INCIDENTS.length} incidents, total false alarms ` + `${KINDS.reduce((a, t) => a + chosen[t].falseAlarms, 0)}, total alerts ${KINDS.reduce((a, t) => a + chosen[t].alerts, 0)}`); console.log(`caught by only one type ${alone.length} incidents: ` + `${alone.map(([a, t]) => `${a} (${t})`).join(", ")}`); const SETS = [["threshold"], ["rate"], ["anomaly"], ["threshold", "anomaly"], ["threshold", "rate"], ["rate", "anomaly"], ["threshold", "rate", "anomaly"]]; console.log(`\n${"rule set".padEnd(24)}${"incidents covered".padStart(19)}${"false alarms".padStart(14)}` + `${"alerts".padStart(8)}`); for (const k of SETS) { const covered = new Set(k.flatMap((t) => [...chosen[t].caught.keys()])); console.log(`${k.join("+").padEnd(24)}${`${covered.size}/${INCIDENTS.length}`.padStart(19)}` + `${String(k.reduce((a, t) => a + chosen[t].falseAlarms, 0)).padStart(14)}` + `${String(k.reduce((a, t) => a + chosen[t].alerts, 0)).padStart(8)}`); } ``` ``` threshold (x median) alerts caught missed false alarms detection round 2 15 4/8 4 11 13 rounds 3 14 5/8 3 9 21 rounds 5 4 4/8 4 0 52 rounds 8 3 3/8 5 0 1 rounds chosen: 3 rate window (min) alerts caught missed false alarms detection round 10 18 5/8 3 13 1 rounds 30 18 5/8 3 13 0 rounds 60 19 6/8 2 12 8 rounds 180 74 8/8 0 57 29 rounds chosen: 60 learning window (days) alerts caught missed false alarms detection round 1 12 6/8 2 6 21 rounds 3 8 6/8 2 0 22 rounds 7 9 6/8 2 0 22 rounds chosen: 3 incident threshold rate anomaly alone sudden error spike 0 rounds 0 rounds 0 rounds slow error ramp 104 rounds 48 rounds 58 rounds overnight traffic loss - 1 rounds 2 rounds short deep outage - 0 rounds - rate steady drift 1 rounds 1 rounds 0 rounds post-release increase 0 rounds 0 rounds 2 rounds overnight request ramp 0 rounds - 68 rounds gradual traffic decline - - - none all three together 7/8 incidents, total false alarms 21, total alerts 41 caught by only one type 1 incidents: short deep outage (rate) rule set incidents covered false alarms alerts threshold 5/8 9 14 rate 6/8 12 19 anomaly 6/8 0 8 threshold+anomaly 6/8 9 22 threshold+rate 7/8 21 33 rate+anomaly 7/8 12 27 threshold+rate+anomaly 7/8 21 41 ``` Each sweep shows its own trade-off. As the threshold tightens, false alarms drop to zero (0 false alarms at five times the median) but caught incidents drop too, and the detection round rises to 52. The eight-times threshold has an average detection round of 1; that number does not come from the rule being fast, it comes from the rule catching only fast incidents — the average is taken only over the incidents it caught. In the rate rule, coverage grows as the window grows: the 180-minute window catches all eight of the eight incidents. Its cost is 57 false alarms, because on the seasonal metric the morning rise also exceeds twice its value from three hours earlier. The window choice here is a compromise between incident shape and the daily pattern. In the anomaly rule, when the learning window is one day, the band rests on a single day, so it stays narrow and produces six false alarms. At three days, false alarms drop to zero and coverage does not fall. Seven days is no better than three; the return on extending the learning window runs out after the third day. #### Complementarity The incident table shows the three types behaving differently on the same incidents. All three catch the sudden spike at round zero. On the slow ramp, threshold fires at round 104, rate at round 48, anomaly at round 58 — the threshold rule arrives toward the end of the incident, because the value only crosses the threshold that late. Only the rate rule catches the four-minute deep outage: the incident goes above the threshold but ends before the five-minute streak condition fills, while the rate rule's three-round condition does fill. The rule-set table carries the real decision. Anomaly alone catches six of the eight incidents with zero false alarms. Adding the threshold rule to this set does not change coverage — all five incidents the threshold catches are already caught by anomaly — but it adds nine false alarms. Adding the rate rule brings coverage from six to seven: the one incident gained is the four-minute deep outage, at a cost of twelve false alarms. Catching one more incident costs twelve false alarms, and that is the subject of the next lesson. The eighth incident is not caught by any of the three types. The gradual traffic decline never falls below the absolute threshold, its hour-over-hour change never reaches double, and it does not deviate enough to cross the learned band's lower bound. The cost of the unanswered question here is a six-hour decline going unnoticed; seeing this incident does not come from changing the rule type but from changing the metric — a ratio normalized against the daily pattern instead of an absolute request count. #### Summary - Three rule types catch different incidents on the same metric: threshold looks at the value, rate at the speed of change, anomaly at a learned band. - The parameter sweep showed a trade-off in every type: as the threshold tightens, false alarms drop to zero but the detection round rises to 52; as the rate window grows to 180 minutes, coverage reaches 8/8 but false alarms jump to 57. - Extending the anomaly rule's learning window from one day to three brought false alarms from 6 to 0; extending it to seven days had no further payoff. - Only the rate rule caught the four-minute deep outage; threshold and anomaly missed it because their streak condition is longer than the incident itself. - Anomaly alone catches 6/8 incidents with zero false alarms; adding threshold brings nine false alarms without changing coverage, adding rate brings twelve false alarms in exchange for one incident. - The eighth incident was not caught by any type: what is missing is not the rule type but the metric itself. #### Next Step When this lesson's three rules were run together, they produced 41 alerts over ten days, and 21 of them matched no incident. The numbers look small on paper; but these alerts do not go to a list, they go to a person. When the same incident trips more than one rule at once, that person is alerted three times for the same thing, closes the fourth alert of the night without reading it, and when a real incident sits among the alerts closed unread, nobody notices. The next lesson applies grouping, suppression, and dependency suppression to the alerts produced over a period and counts alerts per person, the actionable-alert ratio, and the real incidents suppression misses. ### Alert Fatigue Source: https://academia.sh/en/courses/devops-observability/alert-fatigue Applying grouping, silencing, and dependency suppression to a thirty-day alert stream: counting alerts per person, the actionable-alert ratio, and the real incidents closed unread, and sweeping the suppression window to measure the incidents suppression misses. The previous lesson's three rules produced 41 alerts over ten days, and 21 of them matched no incident. A real system does not have three rules; every service has several rules, and all of them route to the same person. This lesson builds a thirty-day alert stream from the ground up, applies three filters in sequence, and measures the same three numbers at every stage: how many notifications reach the person, how many of them turn into work, and how many real incidents get lost before reaching that person. The stream arises from the regional measurement network's (fiction) six services and fourteen rules; a shared data store sits below the services, and some incidents spread upward from there. Alert fatigue is not a state of mind, it is a measurable threshold: as the number of alerts a person receives in twenty-four hours grows, response time lengthens, and past a certain count, alerts get closed unread. The model in this lesson takes that curve as a given assumption and counts its consequences. #### A Thirty-Day Stream ```js // fatigue/stream.mjs — MODEL of a thirty-day alert stream. MD25: six services and the dependency // between them are fiction; there are four services under the data store's edge. MD26: twenty real // incidents are known incidents; 1-3 rules fire in the incident's own service, 1-2 derived rules // fire in each service depending on it. MD27: three of the fourteen rules are noisy and produce // alerts independent of any incident. export const SEED = 20260805, DAYS = 30, ONCALL = 2; export const DEPENDENCY = { gateway: ["read", "billing"], read: ["store"], validate: ["store"], billing: ["workorder", "store"], workorder: ["store"], store: [] }; export const SERVICES = Object.keys(DEPENDENCY); export const RULES = SERVICES.flatMap((s) => [`${s}.error_rate`, `${s}.latency`]) .concat(["store.connection", "store.disk"]); export const NOISY = ["gateway.latency", "store.connection", "workorder.latency"]; // The services standing above a service: when that service breaks, these fire too. export const upstreams = (s) => SERVICES.filter((u) => DEPENDENCY[u].includes(s)); export function stream() { let s = SEED; const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); const minutes = DAYS * 1440, incidents = [], alerts = []; for (let i = 0; i < 20; i += 1) { const t = Math.floor(r() * minutes), service = SERVICES[Math.floor(r() * SERVICES.length)]; incidents.push({ id: i, t, service, duration: 30 + Math.floor(r() * 150) }); } incidents.sort((a, b) => a.t - b.t).forEach((o, i) => { o.id = i; }); for (const o of incidents) { const own = RULES.filter((k) => k.startsWith(`${o.service}.`)); for (const k of own.slice(0, 1 + Math.floor(r() * own.length))) alerts.push({ t: o.t + Math.floor(r() * 8), rule: k, service: o.service, incident: o.id, derived: false }); for (const u of upstreams(o.service)) { // cascade: upstream services fire too const uRules = RULES.filter((k) => k.startsWith(`${u}.`)); for (const k of uRules.slice(0, 1 + Math.floor(r() * 2))) alerts.push({ t: o.t + 2 + Math.floor(r() * 12), rule: k, service: u, incident: o.id, derived: true }); } } for (const k of RULES) { // noise independent of any incident const n = NOISY.includes(k) ? 40 + Math.floor(r() * 30) : 2 + Math.floor(r() * 6); for (let i = 0; i < n; i += 1) alerts.push({ t: Math.floor(r() * minutes), rule: k, service: k.split(".")[0], incident: null, derived: false }); } alerts.sort((a, b) => a.t - b.t); return { incidents, alerts }; } if (import.meta.filename === process.argv[1]) { const { incidents, alerts } = stream(); const real = alerts.filter((u) => u.incident !== null); console.log(`seed ${SEED}, ${DAYS} days, ${RULES.length} rules, ${SERVICES.length} services, ` + `${incidents.length} real incidents, ${alerts.length} raw alerts`); console.log(`incident-linked alerts ${real.length} (${real.filter((u) => u.derived).length} derived), ` + `incident-independent ${alerts.length - real.length}`); console.log(`${(alerts.length / DAYS).toFixed(1)} alerts/day, ${(alerts.length / DAYS / ONCALL).toFixed(1)} per on-call\n`); const count = new Map(); for (const u of alerts) count.set(u.rule, (count.get(u.rule) || 0) + 1); console.log(`${"noisiest rule".padEnd(22)}${"alerts".padStart(8)}${"incident-linked".padStart(17)}`); [...count].sort((a, b) => b[1] - a[1]).slice(0, 5).forEach(([k, n]) => console.log(`${k.padEnd(22)}${String(n).padStart(8)}` + `${String(alerts.filter((u) => u.rule === k && u.incident !== null).length).padStart(17)}`)); } ``` ``` seed 20260805, 30 days, 14 rules, 6 services, 20 real incidents, 268 raw alerts incident-linked alerts 56 (26 derived), incident-independent 212 8.9 alerts/day, 4.5 per on-call noisiest rule alerts incident-linked workorder.latency 70 2 gateway.latency 58 3 store.connection 40 0 gateway.error_rate 16 9 billing.error_rate 13 9 ``` 268 alerts were produced over thirty days; 8.9 per day. Of these, 212 are tied to no incident, and 56 arose from twenty real incidents — 26 of the 56 are derived, that is, an alert born not in the incident's own service but in an upstream service depending on it. Because there are four services below the data store's edge, every incident that touches the store produces alerts from up to four services. The noisiest-rule list shows a second separation: the workorder latency rule produced 70 alerts over thirty days, and only two of them matched an incident. The gateway error-rate rule produced 16 alerts, nine of them incident-linked. What is counted is not the rule count, it is the empty alerts per rule. #### Three Filters The filters are applied in sequence: nearby alerts from the same service are grouped into a single notification, a rule producing too many alerts in a short time is temporarily silenced, and if a service's dependency is already alerting, that service's alert is suppressed. The "actionable notification" in this measurement has a narrow definition: a notification counts as actionable only if it is tied to a real incident and is the first read notification for that incident. A second alert for the same incident does not define work, because the work has already started; an alert with no incident does not define work either. This definition shows a low ratio, but it counts the work the person actually does. ```js // fatigue/filter.mjs — three filters are applied to the raw alert stream in sequence, and the same // measurements are taken again at each stage. MD28: response time base is 4 min; it grows 18% for // every alert the on-call received in the last 24 hours; the ninth and later alerts in the last 24 // hours are closed unread. MD29: grouping window 15 min (same service), silencing threshold 3 // alerts in the last 24 hours (same rule), dependency suppression window 45 min. MD30: on-call is // shared between two people on a fifteen-day rotation. import { stream, DEPENDENCY, DAYS, ONCALL, SEED } from "./stream.mjs"; const { incidents, alerts } = stream(); const BASE = 4, INCREASE = 0.18, UNREAD = 9, WINDOW = { group: 15, silence: 3, dependency: 45 }; const fatigueThreshold = Math.ceil(1 + 1 / INCREASE); // rank at which response time doubles const group = (U) => U.filter((u, i) => !U.some((v, j) => j < i && v.service === u.service && u.t - v.t <= WINDOW.group)); const silence = (U) => U.filter((u) => U.filter((v) => v.rule === u.rule && v.t <= u.t && u.t - v.t < 1440).length <= WINDOW.silence); // Suppression is causal: if a dependency already alerted, the service above it is suppressed. const suppress = (U) => U.filter((u) => !U.some((v) => DEPENDENCY[u.service].includes(v.service) && v.t <= u.t && u.t - v.t <= WINDOW.dependency)); function measure(U) { const seen = new Set(), responseTimes = [], ranks = [], delivered = new Set(U.map((u) => u.incident)); let actioned = 0, unread = 0; // responseTimes: response time of the first read notification of a seen incident for (let i = 0; i < U.length; i += 1) { const u = U[i]; const rank = U.filter((v, j) => j <= i && u.t - v.t < 1440).length; ranks.push(rank); if (rank >= UNREAD) { unread += 1; continue; } // closed unread if (u.incident !== null && !seen.has(u.incident)) { seen.add(u.incident); actioned += 1; responseTimes.push(BASE * (1 + INCREASE * (rank - 1))); } } const avg = (x) => (x.length ? x.reduce((a, b) => a + b, 0) / x.length : 0); const undelivered = incidents.filter((o) => !delivered.has(o.id)).length; return { notifications: U.length, perDay: U.length / DAYS, actioned: actioned / U.length * 100, covered: seen.size, undelivered, unseen: incidents.length - seen.size - undelivered, responseTime: avg(responseTimes), unread, fatigued: ranks.filter((x) => x >= fatigueThreshold).length / U.length * 100 }; } const STAGES = [["raw stream", alerts]]; STAGES.push(["grouping", group(STAGES[0][1])]); STAGES.push(["+ silencing", silence(STAGES[1][1])]); STAGES.push(["+ dependency suppression", suppress(STAGES[2][1])]); console.log(`fatigue threshold: ${fatigueThreshold}. response time after that alert is double the base ` + `(${BASE} min -> ${(BASE * (1 + INCREASE * (fatigueThreshold - 1))).toFixed(1)} min)\n`); console.log(`${"stage".padEnd(26)}${"notifications".padStart(15)}${"day".padStart(6)}${"actioned".padStart(11)}` + `${"covered".padStart(9)}${"undelivered".padStart(13)}${"unread".padStart(9)}` + `${"incident response".padStart(20)}${"above threshold".padStart(17)}`); const results = STAGES.map(([name, U]) => { const d = measure(U); console.log(`${name.padEnd(26)}${String(d.notifications).padStart(15)}${d.perDay.toFixed(1).padStart(6)}` + `${`${d.actioned.toFixed(1)}%`.padStart(11)}${`${d.covered}/${incidents.length}`.padStart(9)}` + `${String(d.undelivered).padStart(13)}${String(d.unseen).padStart(9)}` + `${`${d.responseTime.toFixed(1)} min`.padStart(20)}${`${d.fatigued.toFixed(0)}%`.padStart(17)}`); return d; }); const [first, last] = [results[0], results[3]]; console.log(`\nper on-call day (${DAYS / ONCALL} days each): raw stream ` + `${(first.notifications / ONCALL).toFixed(0)} notifications, after filters ${(last.notifications / ONCALL).toFixed(0)}`); console.log(`alerts closed unread ${first.unread} -> ${last.unread}`); const remaining = incidents.filter((o) => !STAGES[3][1].some((u) => u.incident === o.id)); console.log(`real incidents the filters do not deliver ${remaining.length}: ` + `${remaining.map((o) => `incident ${o.id} (${o.service})`).join(", ") || "none"}`); for (const [name, before, after] of [["grouping", STAGES[0][1], STAGES[1][1]], ["silencing", STAGES[1][1], STAGES[2][1]]]) { const k = before.filter((u) => !after.includes(u)); console.log(`${name} removed ${k.length} notifications, ${k.filter((u) => u.incident !== null).length} incident-linked`); } console.log(`\n${"suppression window".padEnd(21)}${"removed".padStart(9)}${"derived".padStart(9)}` + `${"independent incident".padStart(23)}${"noise".padStart(7)}${"undelivered incident".padStart(23)}`); for (const w of [15, 45, 120, 360]) { WINDOW.dependency = w; const B = suppress(STAGES[2][1]), removed = STAGES[2][1].filter((u) => !B.includes(u)); const undelivered = incidents.filter((o) => !B.some((u) => u.incident === o.id)).length; console.log(`${`${w} min`.padEnd(21)}${String(removed.length).padStart(9)}` + `${String(removed.filter((u) => u.derived).length).padStart(9)}` + `${String(removed.filter((u) => u.incident !== null && !u.derived).length).padStart(23)}` + `${String(removed.filter((u) => u.incident === null).length).padStart(7)}` + `${String(undelivered).padStart(23)}`); } ``` ``` fatigue threshold: 7. response time after that alert is double the base (4 min -> 8.3 min) stage notifications day actioned covered undelivered unread incident response above threshold raw stream 268 8.9 3.0% 8/20 0 12 7.8 min 80% grouping 247 8.2 3.2% 8/20 0 12 7.6 min 77% + silencing 208 6.9 5.8% 12/20 0 8 7.8 min 64% + dependency suppression 191 6.4 6.8% 13/20 0 7 7.8 min 59% per on-call day (15 days each): raw stream 134 notifications, after filters 96 alerts closed unread 152 -> 40 real incidents the filters do not deliver 0: none grouping removed 21 notifications, 18 incident-linked silencing removed 39 notifications, 1 incident-linked suppression window removed derived independent incident noise undelivered incident 15 min 15 14 0 1 0 45 min 17 14 0 3 0 120 min 25 15 3 7 3 360 min 46 16 6 24 6 ``` #### The Number Behind Fatigue The fatigue threshold falls out of the arithmetic: after the seventh alert, response time reaches double the base, from 4 minutes to 8.3 minutes. In the raw stream, 80% of notifications arrive at a rank above this threshold. That is, four-fifths of the alerts reaching the on-call arrive at a moment when that person has already received six alerts in the last twenty-four hours. The third column carries the outcome. In the raw stream, only eight of the twenty real incidents are seen; twelve stay buried among the alerts closed unread. No alert was lost, no rule was written wrong, undelivered incidents are zero — the incidents were only too far back in the queue. The filters bring this number from eight to thirteen. The source of the gain is this: the filters add no new information, they deliver the same information in fewer notifications; as the noise falls away, the real incident's notification moves toward the front of the queue. The actionable-notification ratio rises from 3.0% to 6.8%, that is, it doubles — and still 93% of notifications turn into no work. The contribution of the three filters is not equal. Grouping removed 21 notifications, and 18 of them were incident-linked: several rules of the same service fire at once for the same incident, and grouping collects them into a single notification. The removed count is small, because the noise is not concentrated in one service, it is spread across fourteen rules. Silencing is the opposite: it removed 39 notifications, and only one was incident-linked. This is the step that brings the seen-incident count from eight to twelve — once the daily share of the three noisy rules is cut, the on-call's queue shortens, and the real incidents left behind drop under the threshold. The response time of the seen incidents, though, barely changes: 7.6 minutes at grouping and 7.8 minutes at the other three stages. The reason is that the filters bring down the count, not the rank itself — the newly seen incidents also arrive at high ranks. The gain is not in response time, it is in the number of incidents seen. #### The Cost of Suppression Dependency suppression is the most dangerous filter, because it deletes information. The window sweep shows the cost. At the fifteen-minute window, 14 of the 15 removed notifications are derived, one is noise; no independent incident's own alert is deleted. At forty-five minutes the table is the same: the derived gain stops at 14, because derived alerts are already born within the incident's first forty-five minutes. Widening the window past this point only does harm. At one hundred twenty minutes, three independent incidents' own alerts are suppressed and three real incidents receive no notification at all; at three hundred sixty minutes this count rises to six. The suppression window is chosen against how long the dependency's effect lasts; going longer leaves the upstream service's own incident in the dependency's shadow. The real distinction here is this: grouping and silencing delay or collect a notification, while dependency suppression deletes it. The first two fail visibly, the third fails invisibly — a suppressed alert appears in no list at all. #### Summary - Over the thirty-day stream, 212 of 268 alerts were tied to no incident; a single noisy rule produced 70 alerts, and only two of them matched an incident. - The fatigue threshold is the seventh alert: past that rank, response time rises from 4 minutes to 8.3 minutes, and 80% of notifications in the raw stream arrive above this threshold. - In the raw stream, eight of the twenty real incidents were seen; twelve were delivered yet closed unread — the loss was not in collection, it was in the queue. - The three filters brought notifications from 268 to 191, from 8.9 to 6.4 per day; seen incidents rose from eight to thirteen, the actionable-notification ratio from 3.0% to 6.8%. What really raised the seen-incident count was silencing: only one of the 39 notifications it removed was incident-linked. - The response time of seen incidents stayed in the 7.6-7.8 minute band across all four stages: the filters bring down the count, not the rank, and the gain is in the number of incidents seen. - Dependency suppression's gain saturates at forty-five minutes; at a one-hundred-twenty-minute window three real incidents, and at a three-hundred-sixty-minute window six real incidents, receive no notification at all. #### Next Step Throughout this lesson, every signal was born from real traffic: a request arrived, a rule fired, a person was alerted. There are also cases real traffic never sees. Between three and five in the morning, billing queries almost never arrive; a flow that breaks at that hour produces no alert until nine in the morning, because there is no request to give rise to one. A flow used a few times a year can be broken for months. An external dependency's own failure, in turn, is visible only once someone sends it a request. The next lesson sets up periodic polling from the outside, sweeps the polling frequency to measure detection delay together with polling cost, and names the case where polling passes while the real user flow is broken. ### Synthetic Monitoring Source: https://academia.sh/en/courses/devops-observability/synthetic-monitoring Counting the cases real traffic never sees and meeting them with periodic polling from the outside: measuring ten faults separately with the real-traffic rule and the probe, sweeping the polling frequency together with detection delay and cost, and naming the cases where the probe gives false confidence. Every signal produced up to this point had one thing in common: all of them were born from real traffic. A request arrived, a span record was written, a metric bucket filled, a rule fired. If the first link of this chain breaks, none of the rest works at all — if no request arrives, there is nothing to measure either. This lesson models fourteen days of the regional measurement network's (fiction) traffic across five flows and measures ten faults two separate ways: a rule born from real traffic, and periodic polling from the outside. Three things are counted: the fault real traffic never sees at all, the probe's detection delay together with its cost, and the case where the probe passes while the user is affected. #### Real Traffic's Blind Spots ```js // synthetic/traffic.mjs — MODEL of fourteen days of real traffic and ten faults. MD31: the per-minute request // count of five flows is fiction; the billing query is busy by day and idle at night, the annual closing flow // is called a few times a month. MD32: a fault is one of three shapes — "full" breaks every request, "partial" // breaks only one subscriber group, "data" breaks the path that does not return from cache. export const SEED = 20260806, DAYS = 14, MINUTES = DAYS * 1440; export const HOUR_PATTERN = [2, 1, 1, 1, 1, 4, 14, 32, 46, 52, 55, 53, 48, 51, 54, 50, 44, 34, 26, 18, 12, 8, 5, 3]; // [name, daytime multiplier, runs off-hours, share affected by partial breakage] export const FLOWS = [["billing_query", 1, true, 0.18], ["payment_intake", 0.35, true, 0.22], ["meter_change", 0.012, false, 0.30], ["annual_closing", 0.00006, false, 0.5], ["external_verification", 0.12, true, 0.25]]; // [flow, day, hour, duration min, type, name] export const FAULTS = [ [0, 2, 14, 40, "full", "daytime billing outage"], [0, 4, 3, 120, "full", "overnight billing outage"], [0, 6, 10, 90, "partial", "error in one subscriber group"], [0, 9, 11, 60, "data", "non-cached path broken"], [1, 3, 20, 75, "full", "evening payment outage"], [1, 7, 2, 180, "full", "overnight payment outage"], [2, 5, 22, 600, "full", "weekend meter change"], [3, 8, 4, 1440, "full", "annual closing flow"], [4, 10, 1, 240, "full", "external dependency fault"], [4, 12, 9, 50, "partial", "partial external verification"], ]; export const window = (i) => [FAULTS[i][1] * 1440 + FAULTS[i][2] * 60, FAULTS[i][3]]; export function traffic() { let s = SEED; const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32); const T = FLOWS.map(() => ({ request: new Float64Array(MINUTES), error: new Float64Array(MINUTES) })); for (let t = 0; t < MINUTES; t += 1) { const hour = Math.floor(t / 60) % 24, day = Math.floor(t / 1440) % 7; for (let a = 0; a < FLOWS.length; a += 1) { const [, multiplier, allDay] = FLOWS[a]; const workHours = allDay || (hour >= 8 && hour < 18 && day < 5); const n = HOUR_PATTERN[hour] * multiplier * (workHours ? 1 : 0.02) * (0.85 + r() * 0.3); T[a].request[t] = Math.max(0, Math.floor(n + (r() < n % 1 ? 1 : 0))); let broken = 0; for (let i = 0; i < FAULTS.length; i += 1) { const [flow, , , dur, type] = FAULTS[i], [start] = window(i); if (flow !== a || t < start || t >= start + dur) continue; broken = Math.max(broken, type === "full" ? 1 : type === "partial" ? FLOWS[a][3] : 0.45); } T[a].error[t] = Math.round(T[a].request[t] * broken); } } return T; } if (import.meta.filename === process.argv[1]) { const T = traffic(); const sum = (a, f, b, e) => { let x = 0; for (let t = b; t < e; t += 1) x += T[a][f][t]; return x; }; console.log(`seed ${SEED}, ${DAYS} days, ${FLOWS.length} flows, ${FAULTS.length} faults\n`); console.log(`${"flow".padEnd(24)}${"daily requests".padStart(16)}${"overnight 03-05/min".padStart(22)}` + `${"idle minutes".padStart(15)}`); FLOWS.forEach(([name], a) => { let idle = 0; for (let t = 0; t < MINUTES; t += 1) if (T[a].request[t] === 0) idle += 1; console.log(`${name.padEnd(24)}${(sum(a, "request", 0, MINUTES) / DAYS).toFixed(0).padStart(16)}` + `${((sum(a, "request", 180, 300) + sum(a, "request", 1620, 1740)) / 240).toFixed(2).padStart(22)}` + `${`${(idle / MINUTES * 100).toFixed(0)}%`.padStart(15)}`); }); } ``` ``` seed 20260806, 14 days, 5 flows, 10 faults flow daily requests overnight 03-05/min idle minutes billing_query 36899 1.02 1% payment_intake 12936 0.35 12% meter_change 251 0.00 83% annual_closing 2 0.00 100% external_verification 4438 0.11 24% ``` The table shows three separate blind spots. Idle hour: the billing query gets 36,899 requests a day, but between three and five in the morning it drops to 1.02 requests per minute. Over a five-minute window that comes to five requests, and five requests is not enough to compute an error rate; the rule does not fire, because the denominator itself is not there. Idle flow: 83% of the meter-change flow's minutes, and 100% of the annual-closing flow's minutes, are idle — the annual closing flow gets two requests a day. A flow like this can stay broken for months and no metric shows it. External dependency: the external verification endpoint is visible only when we send it a request; that endpoint's own fault does not show up in our metrics during the hours we do not call it. #### Probe and Traffic Side by Side The probe sends a request to every flow at set intervals and checks the response. Three choices define this request: where it is sent from (from the direction the user comes from, from outside the system), which flow is run (not a single endpoint, but an end-to-end piece of work), and what it leaves behind — when a write-performing flow is probed, a record stays in the system every time, so the probe either picks a read-only path or deletes what it leaves behind itself. The comparison below measures the same ten faults two ways. ```js // synthetic/probe.mjs — the rule born from real traffic and periodic polling from the outside are compared // on the same ten faults. MD33: the real-traffic rule does not fire without seeing at least 20 requests in // a five-minute window; it alerts if the error rate exceeds 5% for three minutes. MD34: the probe sends // one request per flow every P minutes, two consecutive failed probes produce an alert. MD35: the probe // runs with a fixed meter id and a single account; this is why the probe passes on "partial" and "data" // type faults. MD36: every probe to the external verification endpoint is billed. import { traffic, FLOWS, FAULTS, window, MINUTES, DAYS } from "./traffic.mjs"; const T = traffic(), MIN = 20, RATE = 0.05, STREAK = 3; function realDetect(i) { const flow = FAULTS[i][0], dur = FAULTS[i][3], [start] = window(i); let streak = 0; for (let t = start; t < start + dur; t += 1) { let n = 0, h = 0; for (let u = Math.max(0, t - 4); u <= t; u += 1) { n += T[flow].request[u]; h += T[flow].error[u]; } if (n >= MIN && h / n > RATE) { streak += 1; if (streak >= STREAK) return t - start; } else streak = 0; } return null; } // The probe only fails on "full" faults; because it runs with fixed input it never sees the other two types (MD35). const probeFails = (i, t) => { const [, , , dur, type] = FAULTS[i], [start] = window(i); return type === "full" && t >= start && t < start + dur; }; function syntheticDetect(i, P) { const [, , , dur] = FAULTS[i], [start] = window(i); let streak = 0; for (let t = Math.ceil(start / P) * P; t < start + dur; t += P) { if (probeFails(i, t)) { streak += 1; if (streak >= 2) return t - start; } else streak = 0; } return null; } const P0 = 5; console.log(`${"fault".padEnd(31)}${"flow".padEnd(24)}${"type".padStart(9)}` + `${"real traffic".padStart(15)}${"probe (5 min)".padStart(16)}`); const fmt = (x) => (x === null ? "-" : `${x} min`); FAULTS.forEach(([flow, , , , type, name], i) => console.log(`${name.padEnd(31)}${FLOWS[flow][0].padEnd(24)}${type.padStart(9)}` + `${fmt(realDetect(i)).padStart(15)}${fmt(syntheticDetect(i, P0)).padStart(16)}`)); const real = FAULTS.map((_, i) => realDetect(i)), synthetic = FAULTS.map((_, i) => syntheticDetect(i, P0)); const missed = FAULTS.filter((_, i) => real[i] === null); console.log(`\nfaults real traffic never saw ${missed.length}: ` + `${missed.map((a) => a[5]).join(", ")}`); const falseConfidence = FAULTS.filter((_, i) => synthetic[i] === null && FAULTS[i][4] !== "full"); console.log(`faults where the probe passes but the user is affected ${falseConfidence.length}: ` + `${falseConfidence.map((a) => `${a[5]} (${a[4]})`).join(", ")}`); const both = FAULTS.filter((_, i) => real[i] === null && synthetic[i] === null); console.log(`real traffic ${real.filter((x) => x !== null).length}/${FAULTS.length}, probe ` + `${synthetic.filter((x) => x !== null).length}/${FAULTS.length}, both together ` + `${FAULTS.length - both.length}/${FAULTS.length}`); const dailyRequestTotal = T.reduce((x, a) => x + [...a.request].reduce((p, q) => p + q, 0), 0) / DAYS; console.log(`\n${"probe interval".padEnd(17)}${"caught".padStart(11)}${"avg delay".padStart(14)}` + `${"probes/day".padStart(13)}${"billed/day".padStart(13)}${"synthetic share".padStart(18)}` + `${"sparse flow".padStart(15)}`); for (const P of [1, 5, 15, 60]) { const hits = FAULTS.map((_, i) => syntheticDetect(i, P)).filter((x) => x !== null); const dailyProbes = FLOWS.length * (1440 / P), sparse = [...T[3].request].reduce((p, q) => p + q, 0) / DAYS; console.log(`${`${P} min`.padEnd(17)}${`${hits.length}/${FAULTS.length}`.padStart(11)}` + `${`${(hits.reduce((a, b) => a + b, 0) / hits.length).toFixed(1)} min`.padStart(14)}` + `${String(dailyProbes).padStart(13)}${String(1440 / P).padStart(13)}` + `${`${(dailyProbes / (dailyRequestTotal + dailyProbes) * 100).toFixed(2)}%`.padStart(18)}` + `${`${((1440 / P) / (sparse + 1440 / P) * 100).toFixed(1)}%`.padStart(15)}`); } ``` ``` fault flow type real traffic probe (5 min) daytime billing outage billing_query full 2 min 5 min overnight billing outage billing_query full - 5 min error in one subscriber group billing_query partial 3 min - non-cached path broken billing_query data 2 min - evening payment outage payment_intake full 2 min 5 min overnight payment outage payment_intake full - 5 min weekend meter change meter_change full - 5 min annual closing flow annual_closing full - 5 min external dependency fault external_verification full - 5 min partial external verification external_verification partial 2 min - faults real traffic never saw 5: overnight billing outage, overnight payment outage, weekend meter change, annual closing flow, external dependency fault faults where the probe passes but the user is affected 3: error in one subscriber group (partial), non-cached path broken (data), partial external verification (partial) real traffic 5/10, probe 7/10, both together 10/10 probe interval caught avg delay probes/day billed/day synthetic share sparse flow 1 min 7/10 1.0 min 7200 1440 11.66% 99.9% 5 min 7/10 5.0 min 1440 288 2.57% 99.4% 15 min 7/10 15.0 min 480 96 0.87% 98.3% 60 min 6/10 60.0 min 120 24 0.22% 93.6% ``` Real traffic caught five of the ten faults, and it was fast when it did: it saw the daytime billing outage at minute 2, the probe saw the same fault at minute 5. During daytime hours, real traffic beats the probe, because a sample of fifty requests a minute is bigger than any single probe. The five faults it missed fit on one list too: two overnight outages, the weekend meter-change breakage, the annual closing flow, and the external dependency's own fault. What the five have in common is that not enough requests reached that flow during the fault. The probe caught all five of these, because the request the probe sends does not depend on the hour or the season. #### Frequency and Cost The frequency sweep holds two curves in the same table. At a one-minute interval, detection delay drops to 1 minute, but 7,200 probes are sent a day, and 1,440 of them are billed external calls. A heavier cost sits in the two right-hand columns: synthetic requests climb to 11.66% of total traffic, and to 99.9% on the sparse flow. Everything seen on that flow's dashboard is now our own probing; the real usage measure disappears. At a sixty-minute interval, cost drops to 0.22%, but caught faults fall from seven to six: the forty-minute daytime outage closes before it reaches two consecutive probes. The probe interval must be smaller than the shortest fault meant to be seen — the two-consecutive-failed-probes condition doubles the interval. The fifteen-minute interval holds the balance in this sweep: seven faults, 15 minutes of delay, 480 probes a day, 96 billed calls, and 0.87% of total traffic. The choice is a comparison of cost per probe against detection delay, and it can be made separately per flow: a flow left idle at night gets probed often, a flow busy by day gets probed rarely, because real traffic already sees the latter. #### False Confidence There are three faults where the probe passes while the user is affected, and two of them have a clear name. Fixed-input probe: the probe runs with a single meter id and a single account; in a breakage affecting one subscriber group, that account is not affected, so the probe passes. The same shape repeats in the external verification's partial fault. Probe answered from cache: because the probe always requests the same record, the response comes from cache; while the data path behind the cache is broken, the probe stays green. The common root of the two cases is this: the probe measures one path through the system, not the system itself. A green probe dashboard does not say "everything works," it says "the path we tried works." That is why the probe does not replace real traffic; the two together cover ten of ten faults, neither one covers them alone. #### Summary - A rule born from real traffic does not fire where no request arrives: on a flow that drops to one request a minute at night, a five-minute window is not enough to compute an error rate. - Real traffic never saw five of the ten faults; what they all had in common was that not enough requests reached that flow during the fault (two overnight outages, a weekend breakage, a flow used a few times a year, an external dependency's own fault). - The probe caught all five of these too but missed three faults, and it lagged behind real traffic on two faults during daytime traffic; together the two cover ten of ten faults. - The frequency sweep showed two costs together: a one-minute interval brings delay down to 1 minute but costs 7,200 probes a day, 1,440 billed calls, and 11.66% of total traffic; on the sparse flow, 99.9% of the metrics become synthetic. - Because the probe interval alerts on two consecutive failed probes, it must be chosen at less than half the shortest fault meant to be seen; at sixty minutes, the forty-minute outage is missed. - False confidence has two names: a fixed-input probe does not show a breakage affecting one subscriber group, and a probe answered from cache does not show a broken data path. #### Next Step In this topic, the signal passed through three stages: a request's path across services became visible as a trace tree, became mergeable through a shared convention, was tied to a question on a dashboard, was filtered through rules, and was confirmed by polling from the outside. The same thing always sits at the end of the chain: the signal wakes up a person. That person's arrangement, though, was never addressed at all. Who wakes up, how many nights a month they wake up, where they read what to do once they are awake, who they hand the work to in the morning, and whether the item opened after an incident closes so the same incident does not happen again — none of this sits in a metric bucket or on a panel. The next topic continues from where the signal ends and the person begins. ## Incident Management The signal turning into a person: a sustainable on-call rotation, incident classification through severity and escalation, documenting repeated response as a runbook, blameless post-incident review and tracking action items, regulating the pace of change with an error budget, and monitoring the economic side of resource usage. ### On-Call Rotation Source: https://academia.sh/en/courses/devops-observability/on-call-rotation The rotation's own cost: 61 of a year's 136 calls arrive at night, five of eight questions are answered from the schedule while three remain unanswered, the annual per-person load in a three-person team climbs to 17.3 on-call weeks and 118.7 recovery hours, the average band is cleared at eight people while the most loaded person never clears the band under any rotation, and shifting the nightly batch job's hour drops the threshold from eight people to four. The signal was generated, filtered, and confirmed. The metric was collected, the log was parsed, the dashboard was built, the alert rule was tied to a threshold. What stands at the end of the chain is not a gauge, but a person whose phone rings at 02:40. **The on-call rotation** is the schedule that determines in advance which call goes to whom, and at what moment. The M20/K05 `Operations Handover` lesson compared full handover with shared on-call and counted the cost of the **handover**; here the on-call format is a given, and what is measured is **the rotation's own cost**: on-call weeks per person per year, night wake-ups, and the **recovery window**. #### Schedule, Coverage, and the Question Set The example is once more the course's fictional **regional measurement network**: the water-meter reading collector, verifier, billing, and work-order services. Both the network and the rotation are fictional; there is no real schedule or call log. **IM1:** Four services are on a single team's rotation; the nightly batch job starts at 02:00 and runs for two hours. **IM2:** The year is 52 weeks; the rotation hands off in order, a shift lasts 1 or 2 weeks, and the team is the first `n` people of an eight-person pool. **IM3:** A call arriving between 00:00–06:00 wakes the on-call person; one arriving during the day does not. **IM4:** The recovery window for a night wake-up is 4 hours; each additional wake-up in the same shift grows the window by 2 hours, and the accumulation resets at shift handoff. **IM5:** If the on-call person cannot touch the call's service, the first person down the list who covers it is woken as a **backup** (their own 4-hour window), and the first step starts at 35 minutes instead of 10. **IM6:** The band is at most 13 on-call weeks, 12 night wake-ups, and 60 recovery hours per person per year. ```js // oncall/rotation.mjs — the on-call rotation model of the fictional regional measurement network // (model): there is no real schedule, call log, or person; the year is a data structure and randomness comes from our own generator. export const WEEK = 52, DAY = 7, HOUR = 24, YEAR = WEEK * DAY; export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"]; export const NIGHT_END = 6, BATCH_START = 2; // calls between 00-06 wake; the nightly batch starts at 02 export const RECOVERY = 4, ACCUMULATION = 2; // first wake-up is 4 hours, each extra one in the same shift adds +2 hours export const BACKUP_WINDOW = 4; // the backup is woken too: their own 4-hour window export const FIRST_STEP = 10, BACKUP_STEP = 35; // delay before the first step starts (minutes) export const THRESHOLD = { weeks: 13, wakeups: 12, recovery: 60 }; // annual bands per person // daily call probability, and the share of calls falling into the nightly batch job window export const RATE = { "reading-collector": 0.16, verifier: 0.09, billing: 0.06, "work-order": 0.05 }; export const BATCH_SHARE = { "reading-collector": 0.40, verifier: 0.35, billing: 0.30, "work-order": 0.10 }; // Coverage: the pool's eight people and the services each can touch. The first three people cover all four services. export const COVERAGE = ` k1 reading-collector verifier billing k2 verifier work-order k3 reading-collector billing work-order k4 reading-collector verifier k5 billing work-order k6 reading-collector work-order k7 verifier billing k8 reading-collector verifier billing work-order` .trim().split("\n").map((r) => { const [name, ...s] = r.trim().split(/\s+/); return { name, s }; }); // shift-register generator; the seed is visible, the sequence is the same run to run export function generator(seed) { let x = seed >>> 0 || 1; return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0; return x / 4294967296; }; } // batchHour: the call set is identical for the same seed; only the hour of calls tied to the batch shifts. export function calls(seed, batchHour = BATCH_START) { const r = generator(seed), list = []; for (let d = 0; d < YEAR; d++) for (const s of SERVICE) { if (r() >= RATE[s]) continue; const batch = r() < BATCH_SHARE[s]; const hour = batch ? batchHour + Math.floor(r() * 2) : Math.floor(r() * HOUR); list.push({ day: d, hour, service: s, batch, night: hour < NIGHT_END }); } return list; } // rotation: n people, shift length v weeks. In the "unscheduled" counter-run there is no // schedule; night calls fall on the whole team. run(): the annual per-person rotation/wake-up/recovery counter. export function run(callList, n, v, noSchedule = false) { const people = COVERAGE.slice(0, n).map((k) => ({ ...k, weeks: 0, wakeups: 0, backups: 0, recovery: 0, shift: -1, consecutive: 0 })); let offCoverage = 0, delay = 0; for (let h = 0; h < WEEK; h++) people[Math.floor(h / v) % n].weeks++; for (const c of callList) { const shift = Math.floor(c.day / (DAY * v)); const i = shift % n, onCall = people[i]; const inCoverage = noSchedule ? people.some((k) => k.s.includes(c.service)) : onCall.s.includes(c.service); if (!inCoverage) offCoverage++; delay += inCoverage ? FIRST_STEP : BACKUP_STEP; if (!c.night) continue; if (noSchedule) { // no rotation: night calls wake everyone for (const k of people) { k.wakeups++; k.recovery += RECOVERY; } continue; } if (onCall.shift !== shift) { onCall.shift = shift; onCall.consecutive = 0; } onCall.wakeups++; onCall.recovery += RECOVERY + ACCUMULATION * onCall.consecutive++; if (inCoverage) continue; for (let off = 1; off < n; off++) { // off coverage: the backup is woken too const y = people[(i + off) % n]; if (!y.s.includes(c.service)) continue; y.wakeups++; y.backups++; y.recovery += BACKUP_WINDOW; break; } } const max = (f) => Math.max(...people.map(f)); const avg = (f) => people.reduce((a, k) => a + f(k), 0) / n; const withinBand = (h, u, t) => h <= THRESHOLD.weeks && u <= THRESHOLD.wakeups && t <= THRESHOLD.recovery; return { n, v, people, offCoverage, weeks: avg((k) => k.weeks), wakeups: avg((k) => k.wakeups), maxWakeups: max((k) => k.wakeups), recovery: avg((k) => k.recovery), maxRecovery: max((k) => k.recovery), backups: people.reduce((a, k) => a + k.backups, 0), delay: delay / callList.length, avgBand: withinBand(avg((k) => k.weeks), avg((k) => k.wakeups), avg((k) => k.recovery)), personBand: people.every((k) => withinBand(k.weeks, k.wakeups, k.recovery)) }; } ``` ```js // oncall/measure.mjs — the rotation's own cost: a sweep over team size and shift length. import { calls, run, COVERAGE, THRESHOLD, WEEK, BACKUP_WINDOW } from "./rotation.mjs"; const SEED = 20260615; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const f1 = (x) => x.toFixed(1); const yn = (b) => (b ? "yes" : "no"); const C = calls(SEED); const night = C.filter((c) => c.night); console.log(`seed ${SEED}: ${C.length} calls in ${WEEK} weeks, ${night.length} of them at night ` + `(${night.filter((c) => c.batch).length} of them in the nightly batch job window)`); const S = [-8, -13, 13, 16, 18, 10, 13]; console.log("\n1. on-call rotation sweep (annual per person; avg / most loaded person)"); write(S, "people", "shift", "on-call wks", "wake-ups", "recovery hours", "avg band", "person band"); const sweep = []; for (const n of [3, 4, 5, 6, 8]) for (const v of [1, 2]) { const R = run(C, n, v); sweep.push(R); write(S, n, `${v} wk`, f1(R.weeks), `${f1(R.wakeups)} / ${R.maxWakeups}`, `${f1(R.recovery)} / ${f1(R.maxRecovery)}`, yn(R.avgBand), yn(R.personBand)); } const D = run(C, 5, 1, true); write(S, 5, "no rotation", "-", `${f1(D.wakeups)} / ${D.maxWakeups}`, `${f1(D.recovery)} / ${f1(D.maxRecovery)}`, yn(D.avgBand), yn(D.personBand)); console.log(`band: on-call <= ${THRESHOLD.weeks} wks, wake-ups <= ${THRESHOLD.wakeups}, ` + `recovery <= ${THRESHOLD.recovery} hours`); const Q = [-69, 20, 8]; console.log("\n2. the question set the on-call rotation answers"); write(Q, "question", "data", "answer"); const QUESTION = [ ["who is on call at 02:40 at night", "on-call schedule", true], ["when does the rotation hand off", "on-call schedule", true], ["can the on-call person touch this service", "coverage schedule", true], ["if not, who is called, and when", "coverage schedule", true], ["how many times has the on-call person woken in the last thirty days", "wake-up log", true], ["which step does the call close with", "-", false], ["how many hours are unproductive after a wake-up", "-", false], ["can this rotation be sustained for a year", "-", false]]; for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no"); console.log(`${QUESTION.filter((x) => x[2]).length} of ${QUESTION.length} questions are answered ` + `from the schedule, ${QUESTION.filter((x) => !x[2]).length} remain unanswered`); const K = [-8, 14, 18, 22, 16]; console.log("\n3. the cost of an off-coverage call (1-week shift)"); write(K, "people", "off coverage", "backup wake-ups", "backup's share (hrs)", "first step min"); for (const R of sweep.filter((r) => r.v === 1)) write(K, R.n, `${R.offCoverage} (${((R.offCoverage / C.length) * 100).toFixed(0)}%)`, R.backups, f1((R.backups * BACKUP_WINDOW) / R.n), f1(R.delay)); const H = [-8, 10, 13, 14, 17]; console.log("\n4. load distribution in a five-person team (1-wk / 2-wk shift)"); write(H, "person", "coverage", "on-call wks", "wake-ups", "recovery hours"); const [A, B] = [run(C, 5, 1), run(C, 5, 2)]; for (let i = 0; i < 5; i++) write(H, COVERAGE[i].name, COVERAGE[i].s.length, A.people[i].weeks, `${A.people[i].wakeups} / ${B.people[i].wakeups}`, `${f1(A.people[i].recovery)} / ${f1(B.people[i].recovery)}`); const C9 = calls(SEED, 9); // same call set, batch job at 09:00 const T = [-8, 18, 20, 11, 10]; console.log(`\n5. if the nightly batch job runs at 09:00 instead of 02:00 (1-week shift)`); console.log(`night calls ${night.length} -> ${C9.filter((c) => c.night).length}`); write(T, "people", "wake-ups 02 / 09", "recovery 02 / 09", "band 02", "band 09"); for (const n of [3, 4, 5, 6, 8]) { const [a, b] = [run(C, n, 1), run(C9, n, 1)]; write(T, n, `${f1(a.wakeups)} / ${f1(b.wakeups)}`, `${f1(a.recovery)} / ${f1(b.recovery)}`, yn(a.avgBand), yn(b.avgBand)); } ``` ``` seed 20260615: 136 calls in 52 weeks, 61 of them at night (41 of them in the nightly batch job window) 1. on-call rotation sweep (annual per person; avg / most loaded person) people shift on-call wks wake-ups recovery hours avg band person band 3 1 wk 17.3 25.7 / 29 118.7 / 138.0 no no 3 2 wk 17.3 28.7 / 38 154.7 / 180.0 no no 4 1 wk 13.0 19.8 / 25 91.0 / 110.0 no no 4 2 wk 13.0 19.3 / 27 107.0 / 146.0 no no 5 1 wk 10.4 16.2 / 20 74.4 / 96.0 no no 5 2 wk 10.4 16.6 / 22 90.4 / 124.0 no no 6 1 wk 8.7 13.3 / 16 61.3 / 82.0 no no 6 2 wk 8.7 14.5 / 20 78.0 / 96.0 no no 8 1 wk 6.5 10.1 / 14 46.5 / 66.0 yes no 8 2 wk 6.5 11.3 / 16 60.0 / 96.0 yes no 5 no rotation - 61.0 / 61 244.0 / 244.0 no no band: on-call <= 13 wks, wake-ups <= 12, recovery <= 60 hours 2. the question set the on-call rotation answers question data answer who is on call at 02:40 at night on-call schedule yes when does the rotation hand off on-call schedule yes can the on-call person touch this service coverage schedule yes if not, who is called, and when coverage schedule yes how many times has the on-call person woken in the last thirty days wake-up log yes which step does the call close with - no how many hours are unproductive after a wake-up - no can this rotation be sustained for a year - no 5 of 8 questions are answered from the schedule, 3 remain unanswered 3. the cost of an off-coverage call (1-week shift) people off coverage backup wake-ups backup's share (hrs) first step min 3 42 (31%) 16 21.3 17.7 4 42 (31%) 18 18.0 17.7 5 50 (37%) 20 16.0 19.2 6 44 (32%) 19 12.7 18.1 8 45 (33%) 20 10.0 18.3 4. load distribution in a five-person team (1-wk / 2-wk shift) person coverage on-call wks wake-ups recovery hours k1 3 11 20 / 22 96.0 / 124.0 k2 2 11 18 / 14 84.0 / 90.0 k3 3 10 20 / 21 86.0 / 112.0 k4 2 10 14 / 14 66.0 / 68.0 k5 2 10 9 / 12 40.0 / 58.0 5. if the nightly batch job runs at 09:00 instead of 02:00 (1-week shift) night calls 61 -> 20 people wake-ups 02 / 09 recovery 02 / 09 band 02 band 09 3 25.7 / 9.3 118.7 / 38.7 no no 4 19.8 / 6.5 91.0 / 27.0 no yes 5 16.2 / 5.0 74.4 / 20.8 no yes 6 13.3 / 4.7 61.3 / 19.3 no yes 8 10.1 / 3.0 46.5 / 12.5 yes yes ``` The numbers are of the **measurement** class; their inputs are the IM1–IM6 assumptions. The schedule-free counter-run sits in the last row of the first table: with no rotation, every night call wakes the whole team, and wake-ups per person climb to 61 with recovery reaching 244 hours. **The on-call rotation's first job is not to reduce the load, but to dump it on one person** — the same call set gives 16.2 wake-ups per person with five people. #### What Happens As the Team Shrinks On-call weeks fall out directly from team size: 17.3 weeks with three people, that is, a third of the year. Wake-ups and recovery degrade faster than that, because the backup for an off-coverage call also comes from the same small pool — 25.7 wake-ups and 118.7 hours with three people, versus 10.1 and 46.5 with eight. Shift length does not change the total, but it does change the **accumulation**. In a two-week shift, recovery rises from 118.7 to 154.7 hours with three people, and from 61.3 to 78.0 with six; the wake-up count stays nearly constant. The reason is singular: a long shift stacks wake-ups onto a single person, and each extra wake-up opens the window by two more hours. **The same number of wake-ups is cheaper when it is spread out.** #### Is the Threshold in the Average, or in the Person Looking at the average, the rotation clears the band **at eight people**; at six people, recovery at 61.3 sits outside the 60-hour limit and wake-ups at 13.3 sit above 12. Looking at the person, **no rotation clears the band**: even at eight people, the most loaded person carries 14 wake-ups and 66 recovery hours. The fourth table shows why. In the five-person team, `k1` wakes 20 times and `k5` wakes 9 times; recovery spreads between 96.0 and 40.0 hours, even though on-call weeks are 10 and 11. The difference comes from **coverage**: a person who covers three services wakes both on their own shift and as a backup on someone else's. **The average on-call load says nothing about the person carrying the heaviest share of it.** #### The Three Questions Left Unanswered The schedule and the coverage schedule together answer five of the eight questions, and all five take the form of **who** and **when**. The three left unanswered fall into the same gap: **which step** will close the call, how many hours of productivity a wake-up takes away, and whether the rotation can be sustained for a year. When "which step" goes unanswered, the call falls outside coverage: in the third table this share sits between 31% and 37%, and **it does not fall as the team grows**, because coverage is a property of the person, not of the pool. The bill has two line items — starting the first step rises from 10 minutes to 17.7–19.2 minutes, and a **second person** is woken between 16 and 20 times a year; that second wake-up costs 16.0 recovery hours per person with five people, more than a fifth of the 74.4 hours there. The recovery window itself is absent from every operational record: it is an assumption in the model, not something measured in the field, so the rotation's cost never gets logged, and the arrangement becomes visible only once a person leaves. #### Who Produces the Wake-Up The fifth table runs the same call set with a single change: the nightly batch job starts at 09:00 instead of 02:00. The call count does not change, night calls fall from 61 to 20, and the band is cleared not at eight people but at **four**; at five people, wake-ups drop from 16.2 to 5.0 and recovery from 74.4 to 20.8 hours. The three-person team still does not clear the band even after this — 9.3 wake-ups is inside the band, 17.3 on-call weeks is outside it. **The on-call rotation looks like a scheduling question; its bill is written in the job calendar.** #### Summary - 61 of the year's 136 calls arrive at night; without a schedule, the per-person load is 61 wake-ups and 244 recovery hours. With a schedule, this load drops to 17.3 on-call weeks, 25.7 wake-ups, and 118.7 recovery hours with three people, and to 6.5 / 10.1 / 46.5 with eight. - Extending the shift to two weeks does not change the wake-up count, but it grows recovery: 118.7 → 154.7 hours with three people, 61.3 → 78.0 with six. - By the average, the band is cleared at eight people; by the person, it is cleared under no rotation — with five people on the same schedule, 20 and 9 wake-ups, and 96.0 and 40.0 recovery hours, stand side by side. - Five of eight questions are answered, three remain unanswered; the "which step" gap drops 31%–37% of calls outside coverage and produces 16–20 second wake-ups a year. - Moving the nightly batch job to 09:00 drops night calls from 61 to 20 and pulls the threshold from eight people to four. #### Next Step This lesson assumed that every call was worth waking a person for. The next lesson labels a known set of incidents with a **severity** rule and counts misclassification in both directions: the unnecessary wake-up of over-escalation, the delayed response of under-escalation. The **escalation** chain is swept over the same set as well — how many steps, how many minutes per step, and how many incidents go unreached by anyone. ### Incident Classification Source: https://academia.sh/en/courses/devops-observability/incident-classification Severity and escalation counted: sixty incidents labeled by two rules, the threshold rule's 37 correct labels against dropping 4 silent data-loss incidents to d4 and producing 11,040 impact-minutes, the cautious rule cutting that to 180 while raising unnecessary wake-ups from 3 to 7, a trade-off of 678.8 impact-minutes per hour spent awake, and extending the escalation chain from two steps to three dropping unreached incidents from 5 to 2. The on-call rotation answered the "who" and "when" questions, but it assumed that every call was worth waking a person for. The wake-up decision is made before the call, and that decision is made with a label. **Severity** is the label that determines how fast and how hard an incident is met; **escalation** is calling the next step when the response the label demands does not arrive. What this lesson measures is not the label itself, but **the cost of a wrong label in both directions**: over-escalation breaks up the night, under-escalation delays the response. #### The Incident Set, the Actual Severity, and Two Rules The example is once more the course's fictional **regional measurement network**: the water-meter reading collector, verifier, billing, and work-order services, across three environments. Both the network and the incident set are fictional. **IM7:** Sixty incidents are produced; each incident's **actual** severity follows only from the subscriber count affected and any data loss, both known only after it closes — `d1` is heaviest, `d4` is lightest. **IM8:** At call time, only the **visible signature** exists: environment, service, error rate, delay factor, nightly batch job window. **IM9:** `d1` and `d2` page and wake at night too; `d3` waits till morning, `d4` drops onto the work list. **IM10:** The cost of an over-escalation that falls at night is a 4-hour recovery window (the same unit as the on-call lesson); for an under-escalation, if the incident is at night it waits for 06:00, and on top of that a `d4` label adds 240 minutes, a `d3` label adds 60. **IM11:** The cost of the delay is **impact-minutes**: delay minutes × the actual severity's weight (`d1` 8, `d2` 3, `d3` 1, `d4` 0). **IM12:** Escalation is sequential; a step responds with probability 0.70 by day and 0.40 at night, and if no response arrives, the next step is called once the wait time ends. ```js // incident/rule.mjs — the fictional regional measurement network's incident set and // classification model (model): there is no real incident record, call order, or person. The generator is our own, the seed is visible. export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"]; export const SEVERITY = ["d1", "d2", "d3", "d4"]; // d1 is the heaviest export const WEIGHT = { d1: 8, d2: 3, d3: 1, d4: 0 }; // impact-minute multiplier export const WAKES = ["d1", "d2"]; // these wake at night too export const WINDOW = 4; // recovery window for an unnecessary wake-up export const MORNING = 6 * 60; // an incident left till morning is handled at 06:00 export const RESPONSE = { day: 0.70, night: 0.40 }; // the probability a step responds export function generator(seed) { let x = seed >>> 0 || 1; return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0; return x / 4294967296; }; } // Hidden fields (known only after the incident closes): subscribers affected, data loss -> actual severity. // Visible signature (at call time): environment, service, error rate, delay factor, batch job window. export function incidents(seed, count = 60) { const r = generator(seed), list = []; for (let i = 0; i < count; i++) { const service = SERVICE[Math.floor(r() * SERVICE.length)]; const u = r(); const environment = u < 0.72 ? "production" : u < 0.88 ? "staging" : "development"; const minute = Math.floor(r() * 1440); const magnitude = r(); // the size of the impact, a long tail const subscribers = environment === "production" ? Math.floor(magnitude ** 3 * 60000) : 0; const dataLoss = environment === "production" && service === "billing" && r() < 0.45; const silent = dataLoss || (service === "verifier" && environment === "production" && r() < 0.35); const errorRate = +(silent ? 0.01 + r() * 0.05 : environment !== "production" ? 0.35 + r() * 0.6 : 0.04 + magnitude * 0.8).toFixed(2); const delayFactor = +(1 + (silent ? r() : environment !== "production" ? 4 + r() * 5 : magnitude * 8)).toFixed(1); const actual = dataLoss || subscribers >= 20000 ? "d1" : subscribers >= 5000 ? "d2" : subscribers >= 500 ? "d3" : "d4"; list.push({ i, service, environment, minute, night: minute < MORNING, batch: minute >= 120 && minute < 240, subscribers, dataLoss, errorRate, delayFactor, actual }); } return list; } // Two classification rules. Both look only at the visible signature. export const RULE = { // A: threshold-based. Filters out non-production, labels the rest by error rate and delay factor A: (o) => o.environment !== "production" ? "d4" : o.errorRate >= 0.50 && o.delayFactor >= 4 ? "d1" : o.errorRate >= 0.25 || o.delayFactor >= 3 ? "d2" : o.batch ? "d3" : "d4", // B: cautious. Thresholds are low; because billing degrades silently, its floor is pulled to d2 B: (o) => o.environment !== "production" ? "d4" : o.errorRate >= 0.30 ? "d1" : o.errorRate >= 0.10 || o.delayFactor >= 2 ? "d2" : o.service === "billing" ? "d2" : "d3", }; const order = (d) => SEVERITY.indexOf(d); export const direction = (label, actual) => order(label) < order(actual) ? "over" : order(label) > order(actual) ? "under" : "correct"; // The cost of a label. Over-escalation makes an unnecessary night wake-up; under-escalation // delays the response, and the delay is multiplied by the actual severity's weight into impact-minutes. export function cost(o, label) { const calls = WAKES.includes(label); const d = direction(label, o.actual); const wakeup = calls && o.night && d === "over" ? 1 : 0; const delay = !calls && WAKES.includes(o.actual) ? (o.night ? MORNING - o.minute : 0) + (label === "d4" ? 240 : 60) : 0; return { wakeup, window: wakeup * WINDOW, delay, impactMin: delay * WEIGHT[o.actual], d, calls }; } // Escalation chain: number of steps and minutes of wait per step. The dice are rolled in advance, // so the same incident sees the same draws when the chain is lengthened. export function dice(seed, count, most = 6) { const r = generator(seed); return Array.from({ length: count }, () => Array.from({ length: most }, () => r())); } export function escalate(o, d, steps, wait) { const p = o.night ? RESPONSE.night : RESPONSE.day; for (let s = 0; s < steps; s++) if (d[s] < p) return { reached: true, step: s + 1, minute: s * wait, woken: s + 1 }; return { reached: false, step: steps, minute: steps * wait, woken: steps }; } ``` ```js // incident/measure.mjs — the classification's two-way error, its cost, and the escalation chain sweep. import { incidents, RULE, SEVERITY, cost, direction, dice, escalate, WAKES, WINDOW } from "./rule.mjs"; const SEED = 20260921; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const f1 = (x) => x.toFixed(1); const count = (a, f) => a.filter(f).length; const O = incidents(SEED); const Z = dice(SEED + 1, O.length); console.log(`seed ${SEED}: ${O.length} incidents, ${count(O, (o) => o.night)} of them at night, ` + `${count(O, (o) => o.environment !== "production")} of them non-production`); console.log("actual severity distribution: " + SEVERITY.map((d) => `${d} ${count(O, (o) => o.actual === d)}`).join(", ")); const A = [-7, 9, 18, 18, 15, 16]; console.log("\n1. two classification rules on the same incident set"); write(A, "rule", "correct", "over-escalation", "under-escalation", "pages", "actually pages"); for (const [name, k] of Object.entries(RULE)) { const e = O.map((o) => direction(k(o), o.actual)); write(A, name, count(e, (x) => x === "correct"), count(e, (x) => x === "over"), count(e, (x) => x === "under"), count(O, (o) => WAKES.includes(k(o))), count(O, (o) => WAKES.includes(o.actual))); } const B = [-7, 17, 13, 17, 14, 14]; console.log("\n2. the two-way cost of misclassification"); write(B, "rule", "unnec. wake-ups", "hours lost", "del. response", "delay min", "impact-min"); const T2 = {}; for (const [name, k] of Object.entries(RULE)) { const b = O.map((o) => cost(o, k(o))); const late = b.filter((x) => x.delay > 0); T2[name] = { hours: b.reduce((a, x) => a + x.window, 0), impact: b.reduce((a, x) => a + x.impactMin, 0) }; write(B, name, b.reduce((a, x) => a + x.wakeup, 0), T2[name].hours, late.length, late.length ? f1(late.reduce((a, x) => a + x.delay, 0) / late.length) : "-", T2[name].impact); } const dHours = T2.B.hours - T2.A.hours, dImpact = T2.A.impact - T2.B.impact; console.log(`A -> B trade-off: hours lost +${dHours}, impact-minutes -${dImpact} ` + `(${f1(dImpact / dHours)} impact-minutes per hour spent awake)`); const C = [-16, 8, 8, 8, 8]; console.log("\n3. the incidents rule A confuses (label -> actual)"); write(C, "label \\ actual", ...SEVERITY); for (const e of SEVERITY) write(C, e, ...SEVERITY.map((g) => count(O, (o) => RULE.A(o) === e && o.actual === g))); const T = [-9, -9, 13, 16, 21, 14]; console.log("\n4. escalation chain sweep (incidents that rule B pages)"); write(T, "steps", "wait", "avg step", "avg reach min", "unreached incidents", "people woken"); const paging = O.filter((o) => WAKES.includes(RULE.B(o))); for (const steps of [2, 3, 4]) for (const wait of [5, 15]) { const R = paging.map((o) => escalate(o, Z[o.i], steps, wait)); write(T, steps, `${wait} min`, f1(R.reduce((a, x) => a + x.step, 0) / R.length), f1(R.reduce((a, x) => a + x.minute, 0) / R.length), count(R, (x) => !x.reached), R.reduce((a, x) => a + x.woken, 0)); } console.log(`${paging.length} pages, ${count(paging, (o) => o.night)} of them at night; ` + `the unnecessary wake-up window is ${WINDOW} hours`); const Q = [-55, 22, 8]; console.log("\n5. the question set the classification answers"); write(Q, "question", "data", "answer"); const QUESTION = [ ["should this incident wake someone", "visible signature", true], ["at which severity", "visible signature", true], ["who is called if there is no response", "escalation chain", true], ["when is the next step", "escalation chain", true], ["how many subscribers did the incident actually affect", "-", false], ["was the label correct", "-", false], ["what happened if it reached no one", "-", false]]; for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no"); console.log(`${count(QUESTION, (x) => x[2])} of ${QUESTION.length} questions are answered at call ` + `time, ${count(QUESTION, (x) => !x[2])} cannot be answered until the incident closes`); ``` ``` seed 20260921: 60 incidents, 15 of them at night, 17 of them non-production actual severity distribution: d1 12, d2 14, d3 13, d4 21 1. two classification rules on the same incident set rule correct over-escalation under-escalation pages actually pages A 37 16 7 31 26 B 27 28 5 40 26 2. the two-way cost of misclassification rule unnec. wake-ups hours lost del. response delay min impact-min A 3 12 5 306.0 11040 B 7 28 1 60.0 180 A -> B trade-off: hours lost +16, impact-minutes -10860 (678.8 impact-minutes per hour spent awake) 3. the incidents rule A confuses (label -> actual) label \ actual d1 d2 d3 d4 d1 8 6 0 0 d2 0 7 10 0 d3 0 0 1 0 d4 4 1 2 21 4. escalation chain sweep (incidents that rule B pages) steps wait avg step avg reach min unreached incidents people woken 2 5 min 1.4 2.8 5 57 2 15 min 1.4 8.3 5 57 3 5 min 1.6 3.0 2 62 3 15 min 1.6 9.0 2 62 4 5 min 1.6 3.3 2 64 4 15 min 1.6 9.8 2 64 40 pages, 12 of them at night; the unnecessary wake-up window is 4 hours 5. the question set the classification answers question data answer should this incident wake someone visible signature yes at which severity visible signature yes who is called if there is no response escalation chain yes when is the next step escalation chain yes how many subscribers did the incident actually affect - no was the label correct - no what happened if it reached no one - no 4 of 7 questions are answered at call time, 3 cannot be answered until the incident closes ``` The numbers are of the **measurement** class; their inputs are the IM7–IM12 assumptions. Of the sixty incidents, 15 fall at night, 17 are in a non-production environment, and the number that genuinely earns a page (`d1` or `d2`) is 26. The threshold rule A produces 31 pages, the cautious rule B produces 40; both sit above the truth. #### Accuracy Does Not Measure the Cost If the first table compares the two rules by correct-label count, A wins by a wide margin: it labels 37 of 60 incidents correctly, B labels 27. Over-escalation is 16 for A, 28 for B. Anyone looking at this measure picks A. The second table reverses the decision. Of A's 16 over-escalations, only **3** fall at night, meaning a cost of 12 recovery hours; of B's 28, 7 land at night and take 28 hours. Against that, A's 5 delayed responses wait 306 minutes on average and produce **11,040 impact-minutes**; B's single delayed response costs 180 impact-minutes in 60 minutes. The trade-off reads in a single line: moving from A to B buys 10,860 impact-minutes at the cost of 16 hours of sleep, that is, **678.8 impact-minutes per hour spent awake**. **The correct-label ratio is not a quality measure; the two directions' costs are in different units, and they are comparable only when counted separately.** #### The Rule Cannot See Silent Degradation The third table shows where A goes wrong. In the `d4` row there are four true `d1` incidents: the rule threw these four onto the work list. All four sit in the same place — data loss in the `billing` service, error rate between 0.01 and 0.05, delay factor between 1.2 and 1.8. **A silent degradation is indistinguishable from a healthy service in the visible signature**; 10,320 of the 11,040 impact-minutes come from these four incidents. B catches these four, but not with a rule — with an **exception**: it pulls the `billing` service's floor up to `d2`. The exception does not see the silence; it memorizes where the silence is. That is also its price — that service's light incidents start producing pages too, and B's unnecessary wake-ups rise from 3 to 7. The one incident B misses is a silent `d2` in the `verifier` service: **wherever no exception is written, the same gap still stands.** The ten incidents in the `d2` row run the other way: incidents whose actual severity is `d3` were escalated because they showed an error rate above 0.25 in production. The cost of these ten over-escalations is close to zero, because most fall during the day. **The same count of wrong labels costs nothing or four hours, depending on which hour it falls in.** #### How Many Steps the Chain Takes, How Many Minutes per Step The fourth table runs B's 40 pages through the escalation chain; 12 of them are at night, where the response probability is low. In a two-step chain, **5 incidents reach no one**. The third step drops that count to 2, the average step climbs from 1.4 to 1.6, and total people woken rises from 57 to 62: rescuing three of the five incidents costs five extra wake-ups. **The fourth step gains nothing** — unreached incidents stay at 2, and people woken climbs to 64. Wait time is the second axis, and it never changes the unreached count: raising it from 5 minutes to 15 pushes the average reach time from 2.8 to 8.3 minutes, without gaining a single incident in return. **The chain's length determines how many incidents get met; the wait time only determines how late they are met.** #### Summary - 26 of sixty incidents genuinely earn a page; the threshold rule produces 31 and the cautious rule produces 40, and both sit above the truth. - The correct-label count is 37 for A and 27 for B; against that, A produces 11,040 impact-minutes and B produces 180 — the accuracy ratio does not measure the cost of either direction. - The two directions are in different units: A has 3 unnecessary night wake-ups (12 hours) and 5 delayed responses (306 minutes on average); B has 7 wake-ups (28 hours) and 1 delayed response (60 minutes). The price of the switch is 678.8 impact-minutes per hour spent awake. - All four `d1` incidents A misses are silent data loss in the `billing` service: error rate 0.01–0.05, delay factor 1.2–1.8. The visible signature cannot tell silence apart from health. - In escalation, two steps fail to reach anyone for 5 incidents; the third step drops that to 2 and costs five extra wake-ups, and the fourth step only wakes 2 more people. - Raising the wait time from 5 to 15 minutes pushes the average reach from 2.8 to 8.3 minutes, without changing the unreached count at all. #### Next Step Four of seven questions are answered at call time; "how many subscribers did the incident affect", "was the label correct", and "what happened if it reached no one" can only be answered once the incident closes. Once the call reaches the right person at the right severity, one question is left: **which step** will close it. The next lesson measures that step as a document — how many steps, how many decision points, how many can be automated, how far apart the closing time is with and without a runbook, and how many steps go invalid after one period. ### Runbooks Source: https://academia.sh/en/courses/devops-observability/runbooks The runbook itself is measured: four runbooks' 33 steps, 6 decision points, and 14 automatable steps; the same 40 incidents closing in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes with 40 error steps with a written runbook; a runbook aged two periods pushing the close time to only 83.3 minutes while raising error steps to 161; and after four periods, 24 of 33 steps going invalid and producing 384 minutes of delay on first runs. The on-call rotation answered the "who" and "when" questions; classification added "at which severity". Once the right person wakes at the right moment, one question remains in their hands: **which step**. **The runbook** is a recurring response written down step by step. The M20/K05 `Operations Handover` lesson counted the cost of an **unwritten runbook** on the handover; here the runbook is written, and what is measured is **the runbook itself**: how many steps, how many decision points, how many can go to a machine, and how many are still correct some time after being written. #### A Runbook Is Not a Document, It Is a Sequence of Steps The example is once more the course's fictional **regional measurement network**: the water-meter reading collector, verifier, billing, and work-order services. Both the network and the runbooks are fictional; there is no real runbook text or run history. **IM13:** A runbook is written for each of four recurring responses; a step's type is **observe**, **verify**, **intervene**, or **decide**, and a decision point can never be automated. **IM14:** Every step depends on an **anchor** — a gauge name, series name, threshold value, file path, flag name, queue name, ID field, version tag. If the anchor changes, every step that depends on it becomes invalid. **IM15:** In a run with no runbook, finding the step multiplies the time by 2.8, and the per-step error probability is 0.34; with a written runbook it is 0.08. Fixing an erroneous step takes 9 minutes. **IM16:** The probability of picking the wrong branch at a decision point is 0.18, at a cost of 12 minutes. **IM17:** An invalid step produces 16 minutes of delay on the first run; on later runs it is known and worked around in 3 minutes, but it counts as an error step either way. **IM18:** Two anchors change every period; the same runbook set is run over 40 incidents. ```js // runbook/runbook.mjs — the fictional regional measurement network's runbook model (model): // there is no real runbook text, incident record, or run history. The generator is our own, the seed is visible. export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"]; export const TYPE = ["observe", "verify", "intervene", "decide"]; // The system fact a step depends on. If this anchor changes, the step becomes invalid. export const ANCHOR = ["gauge-name", "series-name", "threshold-value", "file-path", "flag-name", "queue-name", "id-field", "version-tag"]; export const DISCOVERY = 2.8; // time multiplier for finding a step with no runbook export const ERROR = { none: 0.34, written: 0.08 }; // per-step error probability export const FIX = 9; // fixing an erroneous step (minutes) export const WRONG_BRANCH = 0.18; // probability of picking the wrong branch at a decision point export const BRANCH_PENALTY = 12; // the cost of the wrong branch (minutes) export const FIRST_PENALTY = 16; // an invalid step, first run: attempted, fails, found by hand export const LATER_PENALTY = 3; // the same step on later runs: known, worked around export const CHANGE = 2; // number of anchors that change per period export function generator(seed) { let x = seed >>> 0 || 1; return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0; return x / 4294967296; }; } const pick = (r, a) => a[Math.floor(r() * a.length)]; // One runbook per service: 6-10 steps. Automatability depends on the step's type; // a decision point can never be automated. export function runbooks(seed) { const r = generator(seed); return SERVICE.map((service) => { const count = 6 + Math.floor(r() * 5); const step = []; for (let i = 0; i < count; i++) { const u = r(); const type = u < 0.22 ? "observe" : u < 0.52 ? "verify" : u < 0.82 ? "intervene" : "decide"; const auto = type === "decide" ? false : r() < (type === "intervene" ? 0.40 : 0.75); step.push({ type, auto, minute: 2 + Math.floor(r() * 11), anchor: pick(r, ANCHOR) }); } return { service, step }; }); } // mode: "none" no runbook, "written" a written runbook, "auto" automatable steps run on a machine. // invalid: invalid anchors. seen: invalid steps already surfaced in an earlier run; // an invalid step produces its full penalty only on the FIRST run, later ones are worked around. export function run(b, mode, r, invalid = new Set(), seen = new Set()) { let minute = 0, errorStep = 0, firstDelay = 0, skipped = 0; for (const [i, a] of b.step.entries()) { const auto = mode === "auto" && a.auto; if (auto) { skipped++; } else { minute += a.minute * (mode === "none" ? DISCOVERY : 1); } if (invalid.has(a.anchor)) { // the runbook has fallen behind the system const key = `${b.service}:${i}`; if (seen.has(key)) { minute += LATER_PENALTY; errorStep++; continue; } seen.add(key); minute += FIRST_PENALTY; firstDelay += FIRST_PENALTY; errorStep++; continue; } if (auto) continue; // a machine-run step never errs if (r() < ERROR[mode === "none" ? "none" : "written"]) { minute += FIX; errorStep++; } if (a.type === "decide" && r() < WRONG_BRANCH) { minute += BRANCH_PENALTY; errorStep++; } } return { minute, errorStep, firstDelay, skipped }; } // Staleness: CHANGE anchors change every period. Every step that depends on a changed anchor is invalid. export function age(seed, period) { const r = generator(seed), g = new Set(); for (let p = 0; p < period; p++) for (let i = 0; i < CHANGE; i++) g.add(pick(r, ANCHOR)); return g; } ``` ```js // runbook/measure.mjs — the runbook itself: steps, decision points, automation, and staleness. import { runbooks, run, age, generator, TYPE, ANCHOR, CHANGE } from "./runbook.mjs"; const SEED = 20261107; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const f1 = (x) => x.toFixed(1); const count = (a, f) => a.filter(f).length; const R = runbooks(SEED); const allSteps = R.flatMap((b) => b.step); const RUNS = 40; // the same runbook set is run over 40 incidents // Runs the same incident set in one mode. Every run starts from the same seed, so the // difference between modes comes from the mode, not from the draws. const attempt = (mode, invalid) => { const r = generator(SEED + 7), seen = new Set(), record = []; for (let i = 0; i < RUNS; i++) record.push(run(R[i % R.length], mode, r, invalid, seen)); const t = (f) => record.reduce((a, x) => a + f(x), 0); return { minute: t((x) => x.minute) / RUNS, error: t((x) => x.errorStep), first: t((x) => x.firstDelay), skipped: t((x) => x.skipped) }; }; const E = [-19, 7, 12, 15, 10, 11]; console.log(`seed ${SEED}: ${R.length} runbooks, ${allSteps.length} steps, ${RUNS} runs`); console.log("\n1. runbook inventory"); write(E, "runbook", "steps", "dec. pts.", "auto steps", "minutes", "anchors"); for (const b of R) write(E, b.service, b.step.length, count(b.step, (a) => a.type === "decide"), count(b.step, (a) => a.auto), b.step.reduce((a, x) => a + x.minute, 0), new Set(b.step.map((a) => a.anchor)).size); write(E, "total", allSteps.length, count(allSteps, (a) => a.type === "decide"), count(allSteps, (a) => a.auto), allSteps.reduce((a, x) => a + x.minute, 0), new Set(allSteps.map((a) => a.anchor)).size); console.log("step type: " + TYPE.map((t) => `${t} ${count(allSteps, (a) => a.type === t)}`).join(", ")); const g2 = age(SEED + 3, 2); // a runbook aged two periods const M = [-22, 17, 14, 16, 16]; console.log("\n2. the same 40 incidents in three run modes (fresh runbook / aged two periods)"); write(M, "mode", "avg close min", "error steps", "auto steps", "vs. no runbook"); const fresh = {}, aged = {}; for (const [name, mode] of [["no runbook", "none"], ["written runbook", "written"], ["automated", "auto"]]) { fresh[mode] = attempt(mode, new Set()); aged[mode] = mode === "none" ? fresh[mode] : attempt(mode, g2); const share = (x) => `${((x.minute / fresh.none.minute) * 100).toFixed(0)}%`; write(M, name, `${f1(fresh[mode].minute)} / ${f1(aged[mode].minute)}`, `${fresh[mode].error} / ${aged[mode].error}`, `${fresh[mode].skipped} / ${aged[mode].skipped}`, `${share(fresh[mode])} / ${share(aged[mode])}`); } const D = [-8, 15, 15, 19, 16, 13]; console.log(`\n3. the runbook's staleness (written runbook, ${CHANGE} changes per period)`); write(D, "period", "invalid anch.", "invalid steps", "1st run delay", "avg close min", "error steps"); for (const period of [0, 1, 2, 3, 4]) { const g = age(SEED + 3, period); const r = attempt("written", g); write(D, period, `${g.size}/${ANCHOR.length}`, `${count(allSteps, (a) => g.has(a.anchor))}/${allSteps.length}`, `${r.first} min`, f1(r.minute), r.error); } const Q = [-47, 20, 8]; console.log("\n4. the question set the runbook answers"); write(Q, "question", "data", "answer"); const QUESTION = [ ["has this incident happened before", "runbook index", true], ["which step does it close with", "runbook text", true], ["how many steps, how many decision points", "runbook text", true], ["which step can go to a machine", "step type", true], ["when was the runbook last run", "run record", true], ["is this step still valid", "-", false], ["which branch is correct at the decision point", "-", false], ["how many steps are now invalid", "-", false]]; for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no"); console.log(`${count(QUESTION, (x) => x[2])} of ${QUESTION.length} questions are answered from ` + `the runbook, ${count(QUESTION, (x) => !x[2])} cannot be answered without running it`); ``` ``` seed 20261107: 4 runbooks, 33 steps, 40 runs 1. runbook inventory runbook steps dec. pts. auto steps minutes anchors reading-collector 6 2 2 47 3 verifier 9 1 5 64 5 billing 10 0 5 86 6 work-order 8 3 2 56 5 total 33 6 14 253 8 step type: observe 7, verify 8, intervene 12, decide 6 2. the same 40 incidents in three run modes (fresh runbook / aged two periods) mode avg close min error steps auto steps vs. no runbook no runbook 203.1 / 203.1 113 / 113 0 / 0 100% / 100% written runbook 72.8 / 83.3 40 / 161 0 / 0 36% / 41% automated 37.5 / 49.8 25 / 154 140 / 140 18% / 24% 3. the runbook's staleness (written runbook, 2 changes per period) period invalid anch. invalid steps 1st run delay avg close min error steps 0 0/8 0/33 0 min 72.8 40 1 2/8 10/33 160 min 80.6 126 2 3/8 14/33 224 min 83.3 161 3 5/8 20/33 320 min 88.1 214 4 6/8 24/33 384 min 91.6 251 4. the question set the runbook answers question data answer has this incident happened before runbook index yes which step does it close with runbook text yes how many steps, how many decision points runbook text yes which step can go to a machine step type yes when was the runbook last run run record yes is this step still valid - no which branch is correct at the decision point - no how many steps are now invalid - no 5 of 8 questions are answered from the runbook, 3 cannot be answered without running it ``` The numbers are of the **measurement** class; their inputs are the IM13–IM18 assumptions. Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes. The inventory alone says two things. `billing` is the longest runbook (10 steps, 86 minutes) but has no decision point at all — it runs straight through. `work-order` is only 8 steps, but carries 3 decision points: a short runbook is not an easy runbook. #### With No Runbook, With a Runbook, With a Machine The left half of the second table closes the same 40 incidents in three modes. In the no-runbook run, average close time is 203.1 minutes with 113 error steps; finding the step multiplies the time, and every step found can go wrong. The written runbook drops close time to 72.8 minutes, that is, **36% of no runbook**, and cuts error steps from 113 to 40. The gain has two sources, and they separate: dropping the discovery multiplier cuts the time, and the error probability falling from 0.34 to 0.08 cuts the mistakes. Automation takes third place. 14 of the 33 steps can go to a machine (42%), and across 40 runs a total of 140 steps run on a machine; close time drops to 37.5 minutes, **18% of no runbook**, and error steps drop to 25. This ceiling is no coincidence: 6 of the remaining 19 steps are **decision points**, and by assumption they cannot be automated. **A runbook's unautomatable core is a countable quantity** — here, six decisions. #### The Runbook's Staleness The third table leaves the runbook where it is and changes the system instead. In one period, two anchors change and **10** of the 33 steps go invalid; after four periods, six of eight anchors have changed and invalid steps reach 24. The invalid step's cost on the first run reads directly: 160 minutes in the first period, 384 in the fourth. The real finding is that the two columns degrade **at different rates**. Average close time rises from 72.8 to 91.6 minutes — only 26% over four periods. Error steps, on the other hand, rise from 40 to 251, a sixfold increase. The reason sits inside the assumption: an invalid step costs 16 minutes on the first run, is worked around in 3 minutes on later runs, but stays an error step every single time. **A stale runbook does not slow down, it becomes unreliable**; a measure that watches only the clock cannot see this. The right half of the second table sharpens this. A runbook aged two periods pushes close time from only 72.8 to 83.3 minutes and is still far faster than no runbook (41% versus 100%). But its error steps are 161 — **more than the no-runbook run's 113 error steps**. A stale runbook speeds the runner up while walking them into the wrong step more often. Automation does not stop this decay. A step handed to a machine still depends on the same anchor: after two periods, the automated mode's error steps rise from 25 to 154, landing almost exactly where the written runbook's 161 sits. Close time rises from 37.5 to 49.8 minutes. **Automation speeds up the step; it does not verify the step's anchor.** #### What the Runbook Does Not Know The fourth table shows that five of eight questions are answered from the runbook: whether the incident has happened before, which step it closes with, how many steps and decision points there are, which step can go to a machine, and when the runbook was last run. The three left unanswered reduce to a single gap — **the runbook does not know its own validity**. "Is this step still valid", "which branch is correct at the decision point", and "how many steps are now invalid" can only be answered once the runbook is run, that is, in the middle of an incident. The delay between 160 and 384 minutes in the third table is the bill for these three questions. #### Summary - Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes; 14 of the steps (42%) can go to a machine. The longest runbook has no decision point at all, and the 8-step runbook has three. - The same 40 incidents close in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes (36%) with 40 error steps with a written runbook. - Automation drops close time to 37.5 minutes (18%) and error steps to 25; what sets the ceiling is the 6 decision points that cannot be automated. - When two anchors change in one period, 10 of the 33 steps go invalid, and after four periods 24 do; the delay invalid steps produce on the first run rises from 160 to 384 minutes. - Staleness degrades time and trust at different rates: close time rises from 72.8 to 91.6 minutes (26%) while error steps rise from 40 to 251 (sixfold). - A runbook aged two periods is still more than twice as fast as no runbook (83.3 versus 203.1 minutes) but errs far more than no runbook (161 versus 113 error steps); automation does not change this (154). #### Next Step A runbook writes **how** an incident closes, not **why** it opened. The third table's invalid steps also do not correct themselves; someone has to see them and update the runbook. The next lesson looks at what happens after the incident closes: a blameless review of the incident, tracking the action items that come out of the review, and an item left open reproducing the same incident. ### Post-Incident Review Source: https://academia.sh/en/courses/devops-observability/post-incident-review Measuring the tracking of the action item: over 18 periods, a no-review run leaves 48 incidents and 90% repeats; when the root cause is written as a person, 84% of 96 items close in 6.9 days, but only 15 are systemic fixes and the repeat rate stays at 84%; when written as a process, 42 of 75 items turn out systemic and incidents drop to 25; when follow-up is dropped, 46 of 61 systemic items never close, the recurring incident rate rises to 84%, and the recurrence outage rises to 1,105 minutes; seven of eight questions are answered with 76,800 bytes, and the eighth is found in no record layer. The previous lesson counted the runbook itself: step count, decision points, and stale steps. The runbook was written, the incident closed, the line reopened. One question was left afterward, never asked: will the same incident happen again, what was done to keep it from happening, and was that work finished. The post-incident analysis itself was measured in M16/K07. What is measured here is not the review's **output**, but the **tracking** of that output. The measure is this: how many action items were opened in a period, how many closed, how long closing took, and **how many times an unclosed item reproduced the same incident**. The blameless review also has a measurable counterpart: how many systemic fixes come out when the root cause is written as a **person**, versus when it is written as a **process**. #### The Question Set and Record Layers The example is once more the fictional **regional measurement network**; the meter-reading collector, verifier, billing, and work-order services. Both the incident history and the review log are fictional. **IM19:** Eighteen periods are run, each period is 30 days; each of five cause classes has a fixed daily incident probability and a fixed outage in minutes. **IM20:** Every incident is reviewed three days later, and the review takes five person-hours. **IM21:** Every review opens three action items; the root cause is written as either person or process, and the writeup determines the item types' shares — for person: reminder 0.55, documentation 0.30, systemic 0.15; for process: systemic 0.55, documentation 0.25, reminder 0.20. **IM22:** The item type determines the closing probability, the closing time, and the person-hours per closed item; the follow-up multiplier scales the closing probability, capped at 0.98. **IM23:** A closed **systemic** item multiplies its class's daily incident probability by 0.70; documentation and reminders never change any probability. **IM24:** *Repeat* means the class has produced an incident before; *recurring incident* means the class produced a new incident while one of its systemic items that will never close sits open. The seed is 20260812, and all dice are rolled up front, so the five runs share the same sequence. Incidents, reviews, and items are a **model** internal to the course; there is no real incident record. ```js // review/model.mjs — the post-incident review model: there is no real incident record; // days, incidents, and action items are data structures. The generator is our own, the seed is visible. export const PERIOD = 30, PERIOD_COUNT = 18, DAY = PERIOD * PERIOD_COUNT; export const SEED = 20260812, ITEM = 3, DELAY = 3, DROP = 0.70; // cause class: name, daily incident probability, outage minutes export const CLASS = [["configuration drift", 0.030, 25], ["capacity", 0.022, 40], ["dependency timeout", 0.026, 18], ["data quality", 0.020, 55], ["batch job window", 0.016, 70]]; // item type: closing probability, closing days, person-hours per closed item export const TYPE = { systemic: [0.55, 25, 12], documentation: [0.80, 8, 3], reminder: [0.95, 2, 0.5] }; // the shares of item types opened when the root cause is written up as person or process export const MIX = { person: [["reminder", 0.55], ["documentation", 0.30], ["systemic", 0.15]], process: [["systemic", 0.55], ["documentation", 0.25], ["reminder", 0.20]] }; export const REVIEW_HOURS = 5; // review person-hours per incident export function generator(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // all dice are rolled up front: runs share the same incident and item sequence const Z = (() => { const z = generator(SEED); return Array.from({ length: DAY * CLASS.length }, () => ({ incident: z(), item: Array.from({ length: ITEM }, () => ({ type: z(), close: z(), duration: z() })) })); })(); const pick = (mix, p) => { let e = 0; for (const [name, y] of mix) { e += y; if (p < e) return name; } }; export function run({ review = true, writeup = "process", followUp = 1.0 } = {}) { const multiplier = CLASS.map(() => 1), openSystemic = CLASS.map(() => 0); // per-class state const stuckSystemic = CLASS.map(() => 0), seen = CLASS.map(() => 0), incidents = [], items = []; for (let g = 0; g < DAY; g++) { for (const m of items) // an item whose closing day has arrived, closes if (m.status === "open" && m.closeDay === g) { m.status = "closed"; if (m.type === "systemic") { multiplier[m.class] *= DROP; openSystemic[m.class]--; } } for (let s = 0; s < CLASS.length; s++) { const z = Z[g * CLASS.length + s]; if (z.incident >= CLASS[s][1] * multiplier[s]) continue; // repeat: the class has produced an incident before. recurring: the class produced a new // incident while one of its systemic items that will never close sits open incidents.push({ g, class: s, outage: CLASS[s][2], repeat: seen[s]++ > 0, recurring: stuckSystemic[s] > 0 }); if (!review) continue; for (const d of z.item) { const type = pick(MIX[writeup], d.type), [closeProb, closeDays] = TYPE[type]; const closes = d.close < Math.min(0.98, closeProb * followUp); const elapsed = Math.round(closeDays * (0.6 + 0.8 * d.duration)); items.push({ class: s, type, opened: g + DELAY, status: "open", closeDay: closes ? g + DELAY + elapsed : null, elapsed: closes ? elapsed : null }); if (type === "systemic") { openSystemic[s]++; if (!closes) stuckSystemic[s]++; } } } } return { incidents, items }; } const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); const pct = (a, b) => (b ? (100 * a / b).toFixed(0) + "%" : "-"); export function measure(settings) { const { incidents, items } = run(settings), M = items.length; const closed = items.filter((m) => m.status === "closed"); const recurring = incidents.filter((o) => o.recurring), repeat = incidents.filter((o) => o.repeat); const minutes = (a) => a.reduce((s, o) => s + o.outage, 0); const systemic = items.filter((m) => m.type === "systemic"); return { incident: incidents.length, outage: minutes(incidents), opened: M, closed: closed.length, repeat: repeat.length, repeatRate: pct(repeat.length, incidents.length), repeatOutage: minutes(repeat), recurring: recurring.length, recurringRate: pct(recurring.length, incidents.length), recurringOutage: minutes(recurring), itemsPerPeriod: (M / PERIOD_COUNT).toFixed(1), closeRate: pct(closed.length, M), closeTime: avg(closed.map((m) => m.elapsed)), systemic: systemic.length, systemicShare: pct(systemic.length, M), systemicClosed: systemic.filter((m) => m.status === "closed").length, hours: incidents.length * (settings.review === false ? 0 : REVIEW_HOURS) + closed.reduce((s, m) => s + TYPE[m.type][2], 0) }; } ``` The measurement file sets up an eight-question set and writes the record fields each question's answer needs. The layers are added on top of each other. ```js // review/measure.mjs — which record layer answers which question, what changes when the // action item is tracked, how many incidents an unclosed item reproduces. import { PERIOD, PERIOD_COUNT, CLASS, SEED, ITEM, measure } from "./model.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); // question set: the record fields needed to answer each question const QUESTION = [ ["when did the incident happen, how long did it last", ["incident"]], ["what was written as the root cause", ["note"]], ["was the root cause written as person or process", ["writeup"]], ["how many items did the review open", ["log"]], ["how many of the items closed, in what time", ["log", "closing"]], ["how many systemic fixes did the person writeup produce", ["writeup", "log"]], ["which incident did an unclosed item reproduce", ["log", "link"]], ["what outage did closing the item prevent", ["counterfactual"]], ]; // record layer: name, the fields it opens, unit bytes, the object the unit is counted on const LAYER = [["incident record", ["incident"], 240, "incident"], ["+ review note", ["note"], 1800, "incident"], ["+ writeup field", ["writeup"], 24, "incident"], ["+ item log", ["log", "closing"], 320, "item"], ["+ item-class link", ["link"], 16, "item"]]; const K = { "no review": { review: false }, "person writeup": { writeup: "person" }, "process writeup": { writeup: "process" }, "process + no follow-up": { writeup: "process", followUp: 0.5 }, "process + weekly follow-up": { writeup: "process", followUp: 1.35 } }; const R = Object.fromEntries(Object.entries(K).map(([a, y]) => [a, measure(y)])); const T = R["process writeup"]; console.log(`${PERIOD_COUNT} periods x ${PERIOD} days, ${CLASS.length} cause classes, ` + `${ITEM} items per review; seed ${SEED}`); console.log("\n1. which record layer answers how many questions (process writeup run)"); const A = [-21, 10, 14, 10, 16]; write(A, "layer", "unit", "cumul. bytes", "answer/8", "bytes/question"); let field = [], bytes = 0; for (const [name, opens, b, unit] of LAYER) { field = field.concat(opens); bytes += b * (unit === "incident" ? T.incident : T.opened); const y = QUESTION.filter(([, g]) => g.every((f) => field.includes(f))).length; write(A, name, unit, bytes, `${y}/8`, Math.round(bytes / y)); } console.log(" unanswered: " + QUESTION.filter(([, g]) => g.includes("counterfactual")).map(([s]) => s) .join("") + " — counterfactual, in no layer"); console.log("\n2. action item tracking and item type"); const B = [-28, 10, 8, 12, 9, 9, 8, 11, 12]; write(B, "run", "incident", "opened", "per period", "closed", "closed", "days", "systemic", "sys. share"); for (const [name, r] of Object.entries(R)) write(B, name, r.incident, r.opened, r.itemsPerPeriod, r.closed, r.closeRate, r.closeTime, `${r.systemic} (${(r.systemic / r.incident).toFixed(2)})`, r.systemicShare); // systemic: count (per incident) console.log("\n3. unclosed items and recurring incidents"); const C = [-28, 9, 8, 11, 8, 15, 17]; write(C, "run", "repeat", "rate", "recurring", "rate", "recur. outage", "unclosed sys."); for (const [name, r] of Object.entries(R)) write(C, name, r.repeat, r.repeatRate, r.recurring, r.recurringRate, r.recurringOutage + " min", `${r.systemic - r.systemicClosed}/${r.systemic}`); console.log("\n4. the cost of the review and the answer to the unanswered question"); const D = [-28, 12, 14, 11, 14]; write(D, "run", "person-hrs", "hrs/incident", "outage", "repeat min"); for (const [name, r] of Object.entries(R)) write(D, name, r.hours.toFixed(0), (r.hours / r.incident).toFixed(1), r.outage + " min", `${r.repeatOutage} (${(100 * r.repeatOutage / r.outage).toFixed(0)}%)`); const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfive runs and the layer sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`); ``` ``` 18 periods x 30 days, 5 cause classes, 3 items per review; seed 20260812 1. which record layer answers how many questions (process writeup run) layer unit cumul. bytes answer/8 bytes/question incident record incident 6000 1/8 6000 + review note incident 51000 2/8 25500 + writeup field incident 51600 3/8 17200 + item log item 75600 6/8 12600 + item-class link item 76800 7/8 10971 unanswered: what outage did closing the item prevent — counterfactual, in no layer 2. action item tracking and item type run incident opened per period closed closed days systemic sys. share no review 48 0 0.0 0 - - 0 (0.00) - person writeup 32 96 5.3 81 84% 6.9 15 (0.47) 16% process writeup 25 75 4.2 52 69% 15.5 42 (1.68) 56% process + no follow-up 38 114 6.3 38 33% 14.0 61 (1.61) 54% process + weekly follow-up 23 69 3.8 60 87% 16.5 40 (1.74) 58% 3. unclosed items and recurring incidents run repeat rate recurring rate recur. outage unclosed sys. no review 43 90% 0 0% 0 min 0/0 person writeup 27 84% 8 25% 335 min 5/15 process writeup 20 80% 14 56% 593 min 16/42 process + no follow-up 33 87% 32 84% 1105 min 46/61 process + weekly follow-up 18 78% 8 35% 433 min 8/40 4. the cost of the review and the answer to the unanswered question run person-hrs hrs/incident outage repeat min no review 0 0.0 1791 min 1583 (88%) person writeup 376 11.7 1173 min 965 (82%) process writeup 480 19.2 966 min 758 (78%) process + no follow-up 404 10.6 1353 min 1145 (85%) process + weekly follow-up 556 24.2 878 min 670 (76%) five runs and the layer sweep within the 2000 ms band: yes ``` The numbers are of the **measurement** class; their inputs are the assumptions above. #### Seven of the Eight Questions The first table turns record layers into questions. With just the incident record, one of eight questions is answered in 6,000 bytes. The review note takes it to 51,000 bytes and the answer count only goes to two — a note is not a field but a text, it is expensive, and it cannot be counted on its own. The writeup field opens the third question at 600 bytes, the item log opens three questions at once at 24,000 bytes; bytes per question fall from 17,200 to 12,600. The last layer is a 16-byte field linking the item to its cause class: at 1,200 bytes, it opens the **which incident did an unclosed item reproduce** question. **The cheapest layer is the last one added, and it answers the most expensive question.** The eighth question is in no layer: what outage did closing this item prevent. Its answer is an unobserved state, and it cannot be answered by widening the record. The closest thing the log gives is "the unclosed item produced this many incidents" — what is counted is what was not prevented, not what was. #### What the Close Rate Does Not Measure At first glance, the second table makes the person writeup look good: 96 items open, 81 close (84%), and closing takes 6.9 days — the fastest of the five runs. In the process writeup, 52 of 75 items close (69%), and closing takes 15.5 days. The last two columns tell the difference. Of the person writeup's 96 items, only **15** are systemic fixes (16%); the rest are reminders and documentation. Of the process writeup's 75 items, **42** are systemic (56%); 0.47 per incident versus 1.68. A reminder closes in two days and changes no probability; a systemic fix closes in twenty-five days and multiplies its class's incident probability by 0.70. **The close rate does not measure what was closed.** The third and fourth tables give the result. In the person writeup, 32 incidents and 1,173 minutes of outage are measured; the repeat rate is 84% — six points below the no-review run's 90%. In the process writeup, incidents drop to 25, outage to 966 minutes, repeats to 80%. This is the measurable counterpart of blame: when the writeup is a person, items open, close fast, and **the system stays the same**. #### The Item Opened and the Item Tracked The third run drops follow-up: the root cause is still written as process, but the closing probability is halved. Opened items do not fall — the opposite, they rise from 75 to 114, because more incidents happen and more reviews take place. Closed items stay at 38 (33%). The critical number is in the systemic items: 61 fixes open, and **46 never close**. The recurring incident rate rises to 84%, recurrence outage rises to 1,105 minutes, and total outage climbs from 966 to 1,353 minutes. Weekly follow-up does the opposite: 60 of 69 items close (87%), 32 of 40 systemic items close, and only 8 stay open. Recurring incidents fall from 14 to 8, total outage falls to 878 minutes. Its cost is in the fourth table: person-hours rise from 480 to 556, to 24.2 person-hours per incident — the most expensive of the five runs. **What is being paid for is not the review, but the tracking of the review.** In the same table, the no-review run looks free: zero person-hours. Its counterpart is 1,791 minutes, and 1,583 minutes of that (88%) comes from repeat incidents. #### Where the Difference Hides The difference hides **between the item opened and the systemic item closed**: in the no-follow-up run, the log shows 114 rows, and 6.3 items per period are reported — the highest output of the five runs. In the same run, 46 of 61 systemic items never close, and 84% of incidents happen while an unclosed item sits open. The reported part is the count of items opened; the unreported part is the item's type and status. In the person writeup, only 5 systemic items go unclosed, and the recurring incident rate looks low at 25%. This is not good news — because only 15 systemic items were opened at all, an incident recurring because of an unclosed item cannot rack up much either. **An item never opened does not show up as an unclosed item either.** The real number there is the repeat rate: 84%. #### Summary - Seven of eight questions are answered by five record layers and 76,800 bytes (10,971 bytes per question); the last field, 16 bytes linking the item to its cause class, opens the most expensive question at 1,200 bytes. - The question left unanswered is "what outage did closing this item prevent"; the counterfactual is in no layer. - When the root cause is written as a person, 84% of 96 items close, and fastest of all (6.9 days), but only 15 are systemic fixes (0.47 per incident) and the repeat rate stays at 84% — close to the no-review run's 90%. When written as a process, 42 of 75 items are systemic (1.68) and incidents drop to 25. - When follow-up is dropped, 114 items open but only 38 close; 46 of 61 systemic items never close, the recurring incident rate rises to 84%, and recurrence outage rises to 1,105 minutes. - Weekly follow-up drops outage to 878 minutes at a cost of 556 person-hours; the no-review run's cost is zero person-hours and 1,791 minutes. #### Next Step Once an item closed, the incident probability dropped, but all of it ran after the incident had already happened: a systemic fix closes in twenty-five days, and the same class kept producing incidents in the meantime. No rule was ever put in place to regulate the pace of change **before** the incident. The next lesson turns the error budget into exactly such a rule: when the budget runs out, new deployment stops — what happens to deployment frequency, what piles up in the queue, how long it takes to recover the budget, and how the rule is loosened when a single major incident stops the line for months. ### Error Budget Management Source: https://academia.sh/en/courses/devops-observability/error-budget-management Turning the error budget into a rule that regulates the pace of change: over 360 days, in a 90-day rolling window, a 129.6-minute budget is exhausted by a single major incident (150 minutes, 116% of the budget), and all four freeze episodes are triggered by a single incident; a full freeze drops deploy frequency from 5.9 to 1.8 while draining a queue of 154 in batches of 40, raising outage from 402 to 901 minutes; a partial freeze holds at 437 minutes and a queue of 3.9; freezing only risky changes gives 480 minutes; and a gradual reopen keeps the same rule while dropping outage to 437 minutes. The previous lesson counted action items and showed that an unclosed one reproduces the same incident. Every fix ran after the incident had already happened; no rule was ever put in place to regulate the pace of change before the incident. The error budget itself was measured in M22/K01: derived from the target, with how much a single major incident takes out of it counted. Here the budget becomes a **rule** — when it runs out, new deployment stops — and the rule itself is measured: deploy frequency, the **piled-up work** in the queue, the budget's **recovery time**, and its **triggering by a single incident**. The rule is then loosened, and each loosening's effect on deploy frequency and incident count is counted. #### Setting Up the Rule The example is once more the fictional **regional measurement network**; both the deployment pipeline and the incident sequence are fictional. **IM25:** 360 days are run, and the budget operates on a **rolling window**: each day, the last 90 days' outage is deducted from it. The window budget is 129.6 minutes, derived in M22/K01 and taken here as an input. **IM26:** One change enters the pipeline daily; the batch floor is five, prep one day, the batch cap forty. **IM27:** A quarter of changes are tagged risky; defect probability is 0.09 for risky ones, 0.014 for the rest. **IM28:** A defective change that ships causes an incident that same day; classes are minor 8 minutes (share 0.58), moderate 35 (0.30), major 150 (0.12). **IM29:** Every change above the batch floor grows the defect probability by a coefficient of 0.04. **IM30:** While the rule is active, *full freeze* ships no batch at all, *partial freeze* extends prep to ten days, *only risky* leaves risky-tagged changes queued, *gradual reopen* drops the batch cap to ten. The seed is 20260819, and every run shares the same change sequence. Days, changes, and incidents are a **model** internal to the course; there is no real deployment pipeline. ```js // budget/rule.mjs — the error budget rule model: there is no real deployment pipeline; days, // changes, and incidents are data structures. The generator is our own, the seed is visible. export const DAY = 360, WINDOW = 90, DAY_MIN = 1440, TARGET = 0.999, SEED = 20260819; export const BUDGET = WINDOW * DAY_MIN * (1 - TARGET); // derived in M22/K01, an input here export const BATCH = 5, BATCH_CAP = 40, PREP = 1, PARTIAL_PREP = 10; export const COUPLING = 0.04; // each extra change in a batch grows the defect probability by this much export const RISK_SHARE = 0.25, DEFECT_RISKY = 0.09, DEFECT_PLAIN = 0.014; export const CLASS = [["minor", 8, 0.58], ["moderate", 35, 0.30], ["major", 150, 0.12]]; export function generator(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // dice are rolled up front: every run shares the same change sequence const Z = (() => { const z = generator(SEED); return Array.from({ length: DAY }, () => ({ risk: z(), defect: z(), class: z() })); })(); export function run({ rule = "none", cap = BATCH_CAP } = {}) { const D = Z.map((z, i) => { // one change enters the pipeline every day const risky = z.risk < RISK_SHARE; let e = 0, minute = 0; for (const [, outage, share] of CLASS) { e += share; if (minute === 0 && z.class < e) minute = outage; } return { i, risky, die: z.defect, base: risky ? DEFECT_RISKY : DEFECT_PLAIN, outage: minute, status: "queued", releaseDay: null }; }); const incidents = [], episodes = []; const S = { release: 0, frozenDays: 0, queueTotal: 0, maxQueue: 0, batchTotal: 0, maxBatch: 0 }; let active = null, start = null, single = false; for (let t = 0; t < DAY; t++) { if (active && active.done === t) { // the batch ships S.release++; S.batchTotal += active.group.length; S.maxBatch = Math.max(S.maxBatch, active.group.length); // the bigger the batch, the more the coupling coefficient grows the defect probability const factor = 1 + COUPLING * (active.group.length - BATCH); for (const x of active.group) { x.status = "done"; x.releaseDay = t; if (x.die < x.base * factor) incidents.push({ day: t, outage: x.outage }); } active = null; } // rolling window: the outage from the last WINDOW days is deducted from the budget const win = incidents.filter((o) => o.day > t - WINDOW); const left = BUDGET - win.reduce((s, o) => s + o.outage, 0); const triggered = rule !== "none" && left < 0; // has the budget run out? if (triggered) { if (start === null) { start = t; single = win.some((o) => o.outage >= BUDGET); } S.frozenDays++; } else if (start !== null) { episodes.push({ length: t - start, single }); start = null; } const queue = D.filter((x) => x.status === "queued" && x.i <= t); S.queueTotal += queue.length; S.maxQueue = Math.max(S.maxQueue, queue.length); if (active) continue; let batch = BATCH, prep = PREP, candidate = queue; if (triggered) { // the ways the rule can be loosened if (rule === "full") candidate = []; else if (rule === "partial") prep = PARTIAL_PREP; else if (rule === "risky") candidate = queue.filter((x) => !x.risky); } if (candidate.length < batch) continue; const group = candidate.slice(0, Math.min(candidate.length, cap)); for (const x of group) x.status = "releasing"; active = { group, done: t + prep }; } if (start !== null) episodes.push({ length: DAY - start, single }); return { D, incidents, episodes, S }; } const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(settings) { const { D, incidents, episodes, S } = run(settings); const L = episodes.map((e) => e.length); return { release: S.release, deploys: (S.release / (DAY / 30)).toFixed(1), incident: incidents.length, outage: incidents.reduce((s, o) => s + o.outage, 0), major: incidents.filter((o) => o.outage === CLASS[2][1]).length, episode: episodes.length, frozenDays: S.frozenDays, longest: L.length ? Math.max(...L) : 0, recovery: avg(L), singleIncident: episodes.filter((e) => e.single).length, avgQueue: (S.queueTotal / DAY).toFixed(1), maxQueue: S.maxQueue, avgBatch: (S.batchTotal / S.release).toFixed(1), maxBatch: S.maxBatch, stuck: D.filter((x) => x.releaseDay === null).length }; } ``` The measurement file first sets up a six-question set — questions genuinely asked in operation while the rule is active — and writes which record answers each question. ```js // budget/measure.mjs — the budget is turned into a rule and run over a period: deploy frequency, // piled-up work, the budget's recovery time, and the rule's early trigger are counted. import { DAY, WINDOW, BUDGET, TARGET, CLASS, SEED, COUPLING, BATCH, measure } from "./rule.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); const K = { "no rule": { rule: "none" }, "full freeze": { rule: "full" }, "partial freeze": { rule: "partial" }, "only risky halts": { rule: "risky" }, "full + gradual reopen": { rule: "full", cap: 10 } }; const R = Object.fromEntries(Object.entries(K).map(([a, y]) => [a, measure(y)])); const T = R["full freeze"]; // question set: the record needed to answer each question, unit bytes, and unit count const QUESTION = [ ["how much budget is left right now", "daily outage record", 48, DAY], ["why did the line freeze", "freeze log", 96, T.episode], ["when will the line reopen", "window drop-off calendar", 32, T.incident], ["how many changes did the freeze pile up", "queue record", 24, DAY], ["which change was marked risky", "risk tag", 8, DAY], ["which incident did the freeze prevent", "counterfactual", 0, 0], ]; console.log(`${DAY} days, rolling window ${WINDOW} days, target ${(100 * TARGET).toFixed(1)}%, ` + `window budget ${BUDGET.toFixed(1)} min; seed ${SEED}`); console.log("incident classes: " + CLASS.map(([a, d, p]) => `${a} ${d} min (share ${p})`).join(", ")); console.log(`major incident / budget = ${(100 * CLASS[2][1] / BUDGET).toFixed(0)}% (a single ` + `incident exhausts the budget); batch floor ${BATCH}, coupling coefficient ${COUPLING}`); console.log("\n1. which question is answered by which record (full freeze run)"); const A = [-41, -26, 10, 8]; write(A, "question", "record needed", "bytes", "answer"); let bytes = 0, answer = 0; for (const [s, k, b, n] of QUESTION) { const needed = b > 0; if (needed) { bytes += b * n; answer++; } write(A, s, k, needed ? b * n : "-", needed ? "yes" : "no"); } console.log(` total ${bytes} bytes, ${answer}/6 questions, ` + `${Math.round(bytes / answer)} bytes per question`); console.log("\n2. how the rule regulates the pace of change"); const B = [-23, 10, 13, 11, 7, 11]; write(B, "run", "releases", "deploys/30d", "incidents", "major", "outage"); for (const [name, r] of Object.entries(R)) write(B, name, r.release, r.deploys, r.incident, r.major, r.outage + " min"); console.log("\n3. freeze episodes and recovering the budget"); const C = [-23, 10, 13, 11, 15, 17]; write(C, "run", "episodes", "frozen days", "longest", "recovery", "single-incident"); for (const [name, r] of Object.entries(R)) write(C, name, r.episode, r.frozenDays, r.longest + " d", r.recovery + " d", `${r.singleIncident}/${r.episode}`); console.log("\n4. piled-up work"); const D = [-23, 13, 15, 12, 16, 11]; write(D, "run", "avg queue", "max queue", "avg batch", "max batch", "stuck"); for (const [name, r] of Object.entries(R)) write(D, name, r.avgQueue, r.maxQueue, r.avgBatch, r.maxBatch, r.stuck); const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfive runs and the question sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`); ``` ``` 360 days, rolling window 90 days, target 99.9%, window budget 129.6 min; seed 20260819 incident classes: minor 8 min (share 0.58), moderate 35 min (share 0.3), major 150 min (share 0.12) major incident / budget = 116% (a single incident exhausts the budget); batch floor 5, coupling coefficient 0.04 1. which question is answered by which record (full freeze run) question record needed bytes answer how much budget is left right now daily outage record 17280 yes why did the line freeze freeze log 384 yes when will the line reopen window drop-off calendar 576 yes how many changes did the freeze pile up queue record 8640 yes which change was marked risky risk tag 2880 yes which incident did the freeze prevent counterfactual - no total 29760 bytes, 5/6 questions, 5952 bytes per question 2. how the rule regulates the pace of change run releases deploys/30d incidents major outage no rule 71 5.9 8 2 402 min full freeze 22 1.8 18 4 901 min partial freeze 58 4.8 9 2 437 min only risky halts 62 5.2 11 2 480 min full + gradual reopen 51 4.3 9 2 437 min 3. freeze episodes and recovering the budget run episodes frozen days longest recovery single-incident no rule 0 0 0 d - d 0/0 full freeze 4 274 90 d 68.5 d 4/4 partial freeze 1 134 134 d 134.0 d 1/1 only risky halts 2 180 90 d 90.0 d 2/2 full + gradual reopen 2 180 90 d 90.0 d 2/2 4. piled-up work run avg queue max queue avg batch max batch stuck no rule 3.0 5 5.0 5 5 full freeze 65.6 154 14.5 40 40 partial freeze 3.9 10 6.1 10 5 only risky halts 9.6 31 5.8 31 3 full + gradual reopen 44.4 154 7.0 10 5 five runs and the question sweep within the 2000 ms band: yes ``` The numbers are of the **measurement** class; their inputs are the assumptions above. #### The Questions the Rule Asks The first table ties the six questions asked while the rule is active to a record. Five are answered, totaling 29,760 bytes; 5,952 bytes per question. The two expensive ones are kept per day: the daily outage record (17,280 bytes) and the queue record (8,640 bytes). The cheap one is **when will the line reopen**: 576 bytes. The rolling window turns this question into an exact date, because each incident's drop-out day is known. Without the calendar, the answer is "we do not know", asked again for every change in the queue. The sixth is the one left unanswered: which incident did the freeze prevent. Here too, no record answers it. While the rule is active, **outage is measured; prevented outage is not**. #### The Budget Runs Out on a Single Incident The third table tells how the rule gets triggered. A major incident is 150 minutes, the window budget is 129.6 minutes: **a single major incident takes out 116% of the budget**. The last column gives the result: in the full freeze, **all four** freeze episodes are triggered by a single incident; the rule responds to a single incident, not to accumulated outage. The recovery time falls out from this. In a rolling window, the budget returns only once the incident drops out of it; a single major incident alone exhausts the budget, so the line stays closed for **ninety days** — the longest freeze is exactly the window's length. In the full freeze, the average episode is 68.5 days, and frozen days are 274: three quarters of the 360 days. In the partial freeze, the single episode lasts 134 days, because the slowed-down line keeps adding new incidents to the window. #### The Rule Increases Outage The row worth reading in the second table is the full freeze: deploy frequency over 30 days drops from 5.9 to 1.8, exactly as intended. But incidents rise from 8 to **18**, outage from 402 to **901 minutes**, major incidents from 2 to 4. The reason is in the fourth table. With no rule, the average queue is 3.0 and the average batch 5.0. In the full freeze, the queue averages 65.6, peaks at 154, and when the line reopens it drains in **batches of 40** — an average batch of 14.5. At a coupling coefficient of 0.04, a batch of 40 carries 2.4 times the floor defect probability. **A freeze does not eliminate change, it piles it up; the piled-up work comes out as a large batch, and a large batch produces more incidents.** New incidents trigger new freezes: episodes climb to 4, and forty changes never ship. #### Three Ways to Loosen the Rule The partial freeze does not stop the line, it slows it down: prep rises to ten days. Deploy frequency stays at 4.8, the average queue is 3.9, and outage stops at 437 minutes — close to the no-rule run's 402 minutes. Its cost is a single 134-day slowdown episode. Freezing only risky changes sits in the middle: deploy frequency 5.2, queue 9.6, outage 480 minutes. It is worse than the partial freeze, because the waiting risky changes eventually ship together in a batch of 31. The tag itself costs 2,880 bytes; what it buys is the queue staying at 31 instead of 154. The third loosening touches not the rule but its **reopening**: the full freeze is kept, but the batch cap is pulled from 40 to 10 as the line reopens. The same rule, the same threshold, the same 90-day longest freeze — but frozen days drop from 274 to 180, episodes from 4 to 2, and the average batch drops from 14.5 to 7.0, incidents from 18 to 9, and outage from 901 to **437 minutes**. Deploy frequency rises from 1.8 to 4.3, and changes that never ship drop from 40 to 5. **The rule's cost is set not by how long the freeze lasts, but by the shape of the exit from it.** #### Where the Difference Hides The difference hides **between how long the freeze lasts and how big the post-freeze batch is**: the full freeze and the gradual reopen share the same rule, the same threshold, and the same 90-day longest freeze, and the max queue is 154 in both. The reported metric is the longest freeze, identical between the two; the unreported part is the batch cap, which alone splits outage between 901 and 437 minutes. The major incident's share is only 0.12, but it takes out 116% of the budget on its own; whether the rule triggers or not depends almost entirely on that single class. The rule reads not accumulated outage, but **the size of one incident**. #### Summary - In the rolling window, the budget is 129.6 minutes and a major incident is 150: a single incident takes out 116% of the budget; in the full freeze, all four episodes are triggered by a single incident, and the longest freeze is 90 days — exactly the window's length. - The full freeze drops deploy frequency from 5.9 to 1.8, but because the queue climbs to 154 and drains in batches of 40, incidents rise from 8 to 18, outage from 402 to 901 minutes, and 40 changes never ship at all. - The partial freeze (prep 1 → 10 days) holds deploy frequency at 4.8, the queue at 3.9, and outage at 437 minutes; its cost is a single 134-day slowdown episode. - Freezing only risky changes holds deploy frequency at 5.2, but outage is 480 minutes, because the waiting risky changes ship together in a single batch of 31. - When the rule is kept but the batch cap is pulled from 40 to 10 at reopening, frozen days drop from 274 to 180, outage from 901 to 437 minutes, and incidents from 18 to 9; deploy frequency rises to 4.3. - Five of six questions are answered with 29,760 bytes (5,952 bytes per question); "which incident did the freeze prevent" is answered by no record. #### Next Step The rule was put in place, the line froze, the queue piled up, and the shape of the exit either doubled the outage or cut it in half. All these numbers are in minutes and changes. The frozen line, the piled-up queue, and the all-night batch job also have a resource counterpart. The next lesson treats capacity as an operating cost — how resource usage turns into a price, which question that price answers, and how much the collected monitoring data costs on its own. ### Capacity and Cost Monitoring Source: https://academia.sh/en/courses/devops-observability/capacity-and-cost-monitoring Tying the operating cost to the question set: 30% of the daily 753 fictional units stays unowned, so only four of eight cost questions are fully answered; monitoring data's own share is 16%; a 90% target utilization saturates on eight of 30 days and produces 360 minutes of outage, while at 75% saturation drops to zero and the avoided minute costs 5.1 units; as scale rises eightfold, infrastructure drops from 507 to 344 units per million requests, but monitoring rises from 100 to 244 and its share climbs to 41%. The previous lesson turned the error budget into a rule: the line froze, the queue piled up, and the shape of the exit either doubled the outage or cut it in half. All of those numbers were denominated in minutes and changes. The question never asked was: what does all of this **cost** to run. The capacity decision itself was measured in M19/K06. Here capacity is treated as an **operating cost**: resource usage turns into a price, that price is applied to a question set, and the collected monitoring data's own cost is counted as a separate line item. In this table, observability is both the one measuring and the one being measured. #### The Cost Question Set The example is once more the fictional **regional measurement network**. The items, prices, and demand are fictional; **the currency is fictional too**, and it is called `u`; it corresponds to no real price list. **IM31:** There are nine cost items; each has a fixed and a variable daily charge, and a scale exponent written for it. The period is 30 days. **IM32:** Some of the items carry an **owner tag**; the shared data store and cross-region transfer are unowned and cannot be apportioned to services. **IM33:** Billing data is delayed by three days; the last three days have no detail at all. **IM34:** Daily demand is a series averaging 1.0 (the generator is our own writing, seed 20260826); capacity is derived from the target utilization rate — the capacity calculation itself was measured in M19/K06, here it is an input. **IM35:** Every day demand exceeds capacity counts as **saturation** and produces 45 minutes of outage. **IM36:** When scale grows `k`-fold, instance count grows `k`-fold and the label cardinality multiplier grows `√k`-fold; the monitoring series is the product of the two, so the scale exponent is 1.5. Items, demand, and the bill are a **model** internal to the course; there is no real bill. ```js // cost/item.mjs — the operating cost model: there is no real bill; items, prices, and demand // are a data structure. The currency is fictional and called `u`. The seed is visible. export const PERIOD = 30, SEED = 20260826, DELAY = 3, SATURATION_MIN = 45; // item: name, fixed u/day, variable u/day, scale exponent, is it owned, is it a monitoring item export const ITEM = [ ["service instances", 40, 134, 1.00, true, false], ["memory and local disk", 0, 86, 1.00, true, false], ["block storage", 12, 58, 1.00, true, false], ["nightly batch job cluster", 55, 20, 0.70, true, false], ["shared data store", 90, 60, 0.80, false, false], ["cross-region transfer", 0, 74, 1.00, false, false], ["time series store", 8, 46, 1.50, true, true], ["log store", 6, 38, 1.50, true, true], ["trace store", 4, 22, 1.50, true, true], ]; export const REQUEST = 1240000; // daily request count at scale 1 export const CAPACITY_UNIT = 278; // daily u for capacity-bound items export const unitCost = (k, x) => x[1] + x[2] * Math.pow(k, x[3]); export const daily = (k, f = () => true) => ITEM.filter(f).reduce((t, x) => t + unitCost(k, x), 0); export function generator(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // daily demand: a series averaging 1.0; the peak-to-floor ratio comes from the assumption export const demand = (() => { const z = generator(SEED); return Array.from({ length: PERIOD }, () => 0.78 + 0.44 * z()); })(); // the cost of a target utilization rate: idle capacity alongside saturation days export function utilization(target) { const avg = demand.reduce((a, b) => a + b, 0) / PERIOD; const capacity = avg / target; const saturation = demand.filter((t) => t > capacity).length; const cost = CAPACITY_UNIT * capacity * PERIOD; return { target, capacity, cost, idle: cost * (1 - target), saturation, outage: saturation * SATURATION_MIN, peak: Math.max(...demand) }; } ``` ```js // cost/measure.mjs — the cost question set, monitoring data's own share, idle resource // alongside saturation, and which line item's unit cost fails to drop as scale grows. import { ITEM, PERIOD, DELAY, SATURATION_MIN, REQUEST, unitCost, daily, utilization } from "./item.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); const G1 = daily(1), owned = daily(1, (x) => x[4]), monitoring = daily(1, (x) => x[5]); const share = (a) => `${(100 * a / G1).toFixed(0)}%`; console.log(`${PERIOD} days, ${ITEM.length} cost items, daily ${G1.toFixed(0)} u ` + `(fictional currency); billing is ${DELAY} days delayed`); // question set: [question, answer, covered cost share, reason unanswered] const QUESTION = [ ["what does the total cost", "yes", 1, "-"], ["what does each service cost", "partial", owned / G1, "unowned resource"], ["where did yesterday's spike come from", "no", 0, "delayed billing data"], ["which resource sits idle", "yes", 1, "-"], ["what does one request cost", "yes", 1, "-"], ["what does one service's request cost", "no", owned / G1, "a shared item cannot be apportioned"], ["what does monitoring data cost", "yes", 1, "-"], ["what happens to price if scale doubles", "partial", 1, "cannot be read from the bill"], ]; console.log("\n1. cost question set"); const A = [-40, 9, 11, 37]; write(A, "question", "answer", "covered", "reason unanswered"); for (const [s, y, p, n] of QUESTION) write(A, s, y, `${(100 * p).toFixed(0)}%`, n); const full = QUESTION.filter((s) => s[1] === "yes").length; console.log(` ${full}/${QUESTION.length} questions are fully answered; ` + `${QUESTION.filter((s) => s[1] === "partial").length} partial, ` + `${QUESTION.filter((s) => s[1] === "no").length} unanswered`); console.log("\n2. item breakdown and monitoring's own share"); const B = [-27, 11, 8, 10, 12]; write(B, "item", "daily u", "share", "owned", "type"); for (const x of ITEM) write(B, x[0], unitCost(1, x).toFixed(0), share(unitCost(1, x)), x[4] ? "owned" : "none", x[5] ? "monitoring" : "infra"); write(B, "total", G1.toFixed(0), "100%", share(owned), share(monitoring)); console.log(` (last row: owned share, monitoring share) monitoring ${monitoring.toFixed(0)} u/day`); console.log("\n3. target utilization rate: idle resource alongside saturation"); const C = [-9, 11, 12, 11, 12, 11, 14]; write(C, "target", "capacity", "period u", "idle u", "saturation", "outage", "idle u/min"); const K = [0.30, 0.45, 0.60, 0.75, 0.90].map(utilization); const baseline = K[K.length - 1]; for (const r of K) { const avoided = baseline.outage - r.outage, diff = r.idle - baseline.idle; write(C, `${(100 * r.target).toFixed(0)}%`, r.capacity.toFixed(2), r.cost.toFixed(0), r.idle.toFixed(0), `${r.saturation}/${PERIOD}`, `${r.outage} min`, avoided > 0 ? (diff / avoided).toFixed(1) : "-"); } console.log(` demand peak ${baseline.peak.toFixed(2)}; a saturated day produces ${SATURATION_MIN} min of outage`); console.log("\n4. unit cost as scale grows"); const D = [-7, 12, 14, 13, 14, 18]; write(D, "scale", "daily u", "u/M requests", "infra/M", "monitoring/M", "monitoring share"); for (const k of [1, 2, 4, 8]) { const t = daily(k), mo = daily(k, (x) => x[5]), M = (REQUEST * k) / 1e6; write(D, `${k}x`, t.toFixed(0), (t / M).toFixed(1), ((t - mo) / M).toFixed(1), (mo / M).toFixed(1), `${(100 * mo / t).toFixed(0)}%`); } const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfour tables and the scale sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`); ``` ``` 30 days, 9 cost items, daily 753 u (fictional currency); billing is 3 days delayed 1. cost question set question answer covered reason unanswered what does the total cost yes 100% - what does each service cost partial 70% unowned resource where did yesterday's spike come from no 0% delayed billing data which resource sits idle yes 100% - what does one request cost yes 100% - what does one service's request cost no 70% a shared item cannot be apportioned what does monitoring data cost yes 100% - what happens to price if scale doubles partial 100% cannot be read from the bill 4/8 questions are fully answered; 2 partial, 2 unanswered 2. item breakdown and monitoring's own share item daily u share owned type service instances 174 23% owned infra memory and local disk 86 11% owned infra block storage 70 9% owned infra nightly batch job cluster 75 10% owned infra shared data store 150 20% none infra cross-region transfer 74 10% none infra time series store 54 7% owned monitoring log store 44 6% owned monitoring trace store 26 3% owned monitoring total 753 100% 70% 16% (last row: owned share, monitoring share) monitoring 124 u/day 3. target utilization rate: idle resource alongside saturation target capacity period u idle u saturation outage idle u/min 30% 3.33 27785 19449 0/30 0 min 51.5 45% 2.22 18523 10188 0/30 0 min 25.7 60% 1.67 13892 5557 0/30 0 min 12.9 75% 1.33 11114 2778 0/30 0 min 5.1 90% 1.11 9262 926 8/30 360 min - demand peak 1.21; a saturated day produces 45 min of outage 4. unit cost as scale grows scale daily u u/M requests infra/M monitoring/M monitoring share 1x 753 607.3 507.3 100.0 16% 2x 1356 546.7 418.5 128.2 23% 4x 2706 545.5 370.9 174.6 32% 8x 5832 587.9 344.3 243.6 41% four tables and the scale sweep within the 2000 ms band: yes ``` The numbers are of the **measurement** class; their inputs are the assumptions above. #### Thirty Percent of the Bill Is Unowned The first table ties eight cost questions to a record and writes what share of the bill each question is answered for: four in full, two partial, two unanswered. The reasons are named separately. **Unowned resource:** the shared data store (20%) and cross-region transfer (10%) carry no owner tag; "what does each service cost" is answered for only **70%** of the bill. **A shared item cannot be apportioned:** the same 30% shuts down "what does one service's request cost" entirely — total unit cost can be calculated, per-service cost cannot. **Delayed billing data:** because the detail comes three days late, "where did yesterday's spike come from" is unanswered with **0%** coverage, and in those three days the same spike is paid for again every day. #### Monitoring Carries Its Own Bill The second table breaks down the items. The daily total is 753 units; **124 units** of that (16%) is monitoring data's own cost — the time series store 54, the log store 44, the trace store 26. All three are on the measuring side, and none of them serves a single request. Measuring the measuring side is the course's own rule applied to itself. Two numbers read alongside this: removing the single label that opened no question recovered 89% of the full series, and panels that answered no question scanned 95% of the query cost. **Unanswered data shows up in this table as a line item.** #### Idle Resource Alongside Saturation The third table puts both costs on the same row. As the target utilization rate rises, capacity shrinks: period cost drops from 27,785 to 9,262 units, and the cost of idle capacity from 19,449 to 926. Against that, at a 90% target, demand exceeds capacity on **eight** of thirty days, and 360 minutes of outage result; demand peak is 1.21, capacity is 1.11. The last column is the decision itself: the per-minute price of avoiding saturation. At a 75% target, saturation is zero, and the extra idle capacity costs **5.1 units per minute avoided**. The same zero saturation costs 12.9 units at a 60% target, 25.7 at 45%, 51.5 at 30%. **Idle resource is insurance, and the price of the same coverage varies tenfold;** pulling the target below 75% does not buy a single additional minute of saturation avoided. One side of this table is unmeasurable: how many units an outage minute corresponds to is nowhere written down. The resource's price is read from the bill; the outage's price is not. #### The Line Item Where Economies of Scale Fail The fourth table scales up eightfold. Infrastructure behaves as expected: per million requests it drops from 507.3 to **344.3 units**, because the fixed floors amortize and the shared data store and the batch job cluster have a scale exponent under one. Monitoring goes the other way. Per million requests it rises from 100.0 to **243.6 units**, and its share climbs from 16% to **41%**. The reason is written plainly in the assumption: as instance count grows `k`-fold, label cardinality grows too, series count is the product of the two, and the scale exponent settles at 1.5. The metric pipeline grows **faster** than the system it is measuring. Total unit cost is the sum of these two curves, and the turning point is in the table: it drops from 607.3 to 546.7, then to 545.5, and climbs back to **587.9** at eight times scale. Economies of scale run out at fourfold growth; past that, growing the system further does not lower unit cost, because the monitoring line item takes the savings back. **The line item where economies of scale fail is observability itself.** #### Summary - Four of eight cost questions are answered in full, two partially, two remain unanswered; three reasons are named — unowned resource (30%), a shared item that cannot be apportioned, and billing data delayed by three days. - 124 of the daily 753 units (16%) is monitoring data's own cost; the measuring side is a separate line item on the bill, and it serves no request at all. - At a 90% target utilization, 8 of 30 days saturate and 360 minutes of outage result; at 75%, saturation drops to zero and the avoided minute costs 5.1 units, while at 30% the same coverage costs 51.5 units. - As scale grows eightfold, infrastructure drops from 507.3 to 344.3 units per million requests, monitoring rises from 100.0 to 243.6 and its share climbs to 41%; total unit cost bottoms out at fourfold scale (545.5) and returns to 587.9 at eightfold. #### Course Wrap-Up | Lesson | Question answered | Question unanswered and why | Cost of the data | |---|---|---|---| | Metric Collection Models | 9/12 (push) | Q12, transaction ID not carried | 1.6 MB/question | | Time Series Data | 11/12 | Q12, not solved by a label | 634.82 GB of unanswered label | | Query and Aggregation | 2/4 measures exact | 729 requests, erased by the summary | 3.6M record scan | | Log Collection | 9/12 (1x load) | 3 questions, the truncated error line | 75 MB/question (2x load) | | Log Parsing | 4/6 correct | 2 questions, an unmaintained rule | 39,957 bytes/correct answer | | Retention and Cost | 11/12 | Q12, in no tier | 2.76 GB/question | | Distributed Tracing Infrastructure | 9/11 (trace) | 1 question, instrumentation depth | 52.7 MB/question | | Instrumentation Standards | 6/10 (convention) | 4 questions, an incompatible service | 20 field changes | | Dashboard Design | 11/12 (after redesign) | 1 question, an uncollected dimension | 814/859 unanswered series | | Alerting Rules | 7/8 incidents (three rules) | 1 incident, the metric itself | 21 false positives | | Alert Fatigue | 13/20 incidents | 7 incidents, the delay down the queue | 191 notifications, 93% empty | | Synthetic Monitoring | 10/10 (both together) | 3 failures, a fixed-input probe | 480 probes/day | | On-Call Rotation | 5/8 | 3 questions, unmeasured recovery | 118.7 recovery hours | | Incident Classification | 4/7 | 3 questions, before the incident closes | 11,040 impact-minutes | | Runbooks | 5/8 | 3 questions, the runbook's validity | 384 minutes of delay | | Post-Incident Review | 7/8 | 1 question, the counterfactual | 10,971 bytes/question | | Error Budget Management | 5/6 | 1 question, the counterfactual | 5,952 bytes/question | | Capacity and Cost Monitoring | 4/8 full | 2 questions, unowned resource | 124 u/day monitoring | The course's rule reads the same way across all eighteen rows: **data is worth as much as the question it answers; data that answers no question is cost, and a question left unanswered is risk.** The first column is never full in any lesson; the fourth column carries a number in every lesson. Four reasons repeat in the third column. **An uncollected dimension** appears in ten rows: the field was never written at all, and neither extending retention nor adding a label opens it — transaction ID, subscriber group, recovery window, unowned resource are all the same gap. **Deleted detail** is in three rows: the data was collected, then lost in some layer — the threshold question the summary erases, the truncated error line, the field an unmaintained rule leaves blank. **A late-arriving answer** is in three rows: the data exists but is not there at the moment it is asked. **The counterfactual** is in two rows, and no record opens it: the outage a closed item prevented, the incident a freeze prevented. The first three can be closed by a design decision; the fourth cannot, and counting what was not prevented takes its place. The system is visible now, and incidents are managed. The next course, **M22/K07 Servers, Networking and Hardening**, goes underneath what is visible: everything these eighteen lessons measured was running on top of a machine and a network, and both are still sitting at their default settings. An open port, an unwritten rule, an unchanged default account — how wide is that surface?