Skip to content
academia.sh
tr
Log in

Course Advanced

Observability and Operations

Published Updated

By the end of this course

Start course

01

The Metric and Log Pipeline

The path the data travels: the scope and latency difference between pull-based and push-based collection, the multiplicative growth of label cardinality in a time series and its retention, queries and aggregation for producing indicators from metrics, log collection through the agent, buffer, and indexing layers, parsing the log into searchable fields, and the balance between level of detail and cost.

  1. 01 Metric Collection Models Pull-based and push-based collection run in the same window and are applied to a set of twelve questions genuinely asked in operations: which model answers how many questions, why short-lived jobs slip through in pull, why resource identity gets confused in push, and how large the discovery delay and the cost per question are.
  2. 02 Time Series Data What question adding a label opens is counted against the question set: for every label, series, bytes, and the question opened are measured together, marginal bytes per question are derived, a label that opens no question is named, and how much the retention window opens each question for is worked out separately.
  3. 03 Query and Aggregation The same question is answered with four aggregations, and the answer is shown to change: the average hides, the percentile shows, the total misleads, and the count and the rate point to different services. The detail pre-aggregation loses irrecoverably and the query cost it buys back are counted in the same table.
  4. 04 Log Collection The log stream is run through the agent, buffer, and indexing layers, and the records lost at every layer are counted: buffer overflow, truncation, and indexing delay. When the load doubles, which layer breaks first is measured, and the question the loss leaves unanswered is written down together with the cost per question.
  5. 05 Log Parsing and Structuring Real log lines are parsed and turned into searchable fields, and two store formats are counted against the question set: how many questions unstructured text answers, how many questions open up once fields are produced. When the format changes, the line the parsing rule silently misses is counted, and the cost of the wrong answer that results is written down.
  6. 06 Retention and Cost Three retention tiers are scanned, and at every tier the number of answerable questions is counted together with the bytes: cost per question is derived tier by tier, one tier's marginal cost differs by a thousandfold, and which questions the cheapest tier loses, and why, is written down by name.

02

Monitoring and Dashboards

Tying the collected data to a question: the collector and storage components of distributed tracing, what a shared telemetry convention buys, the design of a dashboard that answers a question, threshold-based, rate-based, and anomaly-based alert rules, measuring alert fatigue through noise reduction and prioritization, and synthetic monitoring that verifies periodically from the outside.

  1. 01 Distributed Tracing Infrastructure Building the path a request takes across five services as a trace tree: putting eleven operational questions to the metric, log, and trace views separately, measuring the byte cost per span, and counting the slow requests the sampling ratio misses.
  2. 02 Instrumentation Standards The telemetry five services write in their own naming cannot be merged across services: ten questions are answered separately over local schemas and the shared convention, silently wrong answers are counted, and the convention's cost is worked out as fields, service changes, and an incompatible service.
  3. 03 Dashboard Design Auditing a dashboard against a set of questions: how many panels answer a question, how many panels answer none, how many questions remain without a panel, and how reordering brings down the number of panels browsed to find an answer to a question.
  4. 04 Alerting Rules Running three rule types — threshold, rate, and anomaly — over the same eight known incidents of the same ten days: each type's parameter is swept and the caught, missed, false-alarm, and detection-round counts are taken, and where the three types complement each other is measured through rule-set combinations.
  5. 05 Alert Fatigue Applying grouping, silencing, and dependency suppression to a thirty-day alert stream: counting alerts per person, the actionable-alert ratio, and the real incidents closed unread, and sweeping the suppression window to measure the incidents suppression misses.
  6. 06 Synthetic Monitoring Counting the cases real traffic never sees and meeting them with periodic polling from the outside: measuring ten faults separately with the real-traffic rule and the probe, sweeping the polling frequency together with detection delay and cost, and naming the cases where the probe gives false confidence.

03

Incident Management

The signal turning into a person: a sustainable on-call rotation, incident classification through severity and escalation, documenting repeated response as a runbook, blameless post-incident review and tracking action items, regulating the pace of change with an error budget, and monitoring the economic side of resource usage.

  1. 01 On-Call Rotation The rotation's own cost: 61 of a year's 136 calls arrive at night, five of eight questions are answered from the schedule while three remain unanswered, the annual per-person load in a three-person team climbs to 17.3 on-call weeks and 118.7 recovery hours, the average band is cleared at eight people while the most loaded person never clears the band under any rotation, and shifting the nightly batch job's hour drops the threshold from eight people to four.
  2. 02 Incident Classification Severity and escalation counted: sixty incidents labeled by two rules, the threshold rule's 37 correct labels against dropping 4 silent data-loss incidents to d4 and producing 11,040 impact-minutes, the cautious rule cutting that to 180 while raising unnecessary wake-ups from 3 to 7, a trade-off of 678.8 impact-minutes per hour spent awake, and extending the escalation chain from two steps to three dropping unreached incidents from 5 to 2.
  3. 03 Runbooks The runbook itself is measured: four runbooks' 33 steps, 6 decision points, and 14 automatable steps; the same 40 incidents closing in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes with 40 error steps with a written runbook; a runbook aged two periods pushing the close time to only 83.3 minutes while raising error steps to 161; and after four periods, 24 of 33 steps going invalid and producing 384 minutes of delay on first runs.
  4. 04 Post-Incident Review Measuring the tracking of the action item: over 18 periods, a no-review run leaves 48 incidents and 90% repeats; when the root cause is written as a person, 84% of 96 items close in 6.9 days, but only 15 are systemic fixes and the repeat rate stays at 84%; when written as a process, 42 of 75 items turn out systemic and incidents drop to 25; when follow-up is dropped, 46 of 61 systemic items never close, the recurring incident rate rises to 84%, and the recurrence outage rises to 1,105 minutes; seven of eight questions are answered with 76,800 bytes, and the eighth is found in no record layer.
  5. 05 Error Budget Management Turning the error budget into a rule that regulates the pace of change: over 360 days, in a 90-day rolling window, a 129.6-minute budget is exhausted by a single major incident (150 minutes, 116% of the budget), and all four freeze episodes are triggered by a single incident; a full freeze drops deploy frequency from 5.9 to 1.8 while draining a queue of 154 in batches of 40, raising outage from 402 to 901 minutes; a partial freeze holds at 437 minutes and a queue of 3.9; freezing only risky changes gives 480 minutes; and a gradual reopen keeps the same rule while dropping outage to 437 minutes.
  6. 06 Capacity and Cost Monitoring Tying the operating cost to the question set: 30% of the daily 753 fictional units stays unowned, so only four of eight cost questions are fully answered; monitoring data's own share is 16%; a 90% target utilization saturates on eight of 30 days and produces 360 minutes of outage, while at 75% saturation drops to zero and the avoided minute costs 5.1 units; as scale rises eightfold, infrastructure drops from 507 to 344 units per million requests, but monitoring rises from 100 to 244 and its share climbs to 41%.

Final exam

Observability and Operations — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close