Skip to content
academia.sh
tr
Log in

Course Advanced

Observability and Reliability

By the end of this course

Start course

01

Telemetry

Producing the three signals in application code: which question the log, the metric, and the trace each answer, how the field-based log and the correlation id make a request traceable end to end, how the counter, gauge, and histogram correspond to different questions, how the propagation context crosses the process boundary and what trace sampling costs in lost information, a vendor-neutral collection interface, and combining the three signals into a single event.

  1. 01 The Three Signals of Observability Actually producing the log, metric, and trace signals in a split loan system: how many steps each signal takes to answer the same three questions, which question each signal leaves unanswered, and how the three signals grow with request count.
  2. 02 Structured Logging Turning the flat-sentence log into a field-based record and attaching to the request a correlation id that crosses the process boundary: how many parsing and filtering steps the same question takes to answer, how many services a request becomes traceable across, and counting the call sites left without the id carried through them.
  3. 03 Metric Types Measuring the same fact with a counter, a gauge, and a histogram: the question each type answers and forecloses, how many observations the gauge loses between reads, and the histogram's memory cost and percentile error as a function of its bucket count.
  4. 04 Distributed Tracing Hand-writing the trace context and carrying it across the process boundary, and measuring the sampling rate: how many traces break at the call site that does not carry the context, how the sampling rate affects the percentage of problem requests caught, and the trace record's volume per span.
  5. 05 Instrumentation Standards Separating telemetry collection from application code: meeting the same interface with two collector implementations, measuring how many files get touched when the implementation changes in a bound layout versus an interfaced one, and counting how many application files the collector's name appears in.
  6. 06 Correlation Combining the three signals into a single event by correlation id: how many steps and how many signals the same question takes to answer, combined and uncombined, the cost of combining that gets paid once, and the sampling rate's effect on how many events carry all three signals.

02

Service Level Management

The path from telemetry to a decision: distinguishing the concepts of indicator, objective, and agreement, computing the golden signals — latency, traffic, errors, and saturation — in application code, the error budget balancing rate of change against stability, symptom-based alert design cutting down noise, and separating the liveness and readiness endpoints.

  1. 01 Indicator, Objective, and Agreement Turning telemetry into a decision: showing that indicator, objective, and agreement are separate sentences, computing three indicator definitions — request-based, minute-based, and member-based — from the loan system's thirty-day request log, and counting how the same period can be described by twelve different numbers.
  2. 02 Golden Signals Computing latency, traffic, errors, and saturation from the loan system's seven-day records: counting the events each signal alone misses across six event classes, measuring how many minutes earlier saturation triggers than the error signal, and showing the silent loss even the four signals together cannot see.
  3. 03 Error Budget Turning the objective's complement into a spendable number: computing the budget from the loan system's real request records as a count of bad requests, counting how fast four release rates consume that budget, and measuring how many releases the release gate blocks and how much it shrinks the overage once the budget runs out.
  4. 04 Alert Design Showing that an alert rule's source determines the outcome: running twelve cause-based rules and four symptom-based rules across the same seven days, and counting the alerts they produce, how many rules fire together for the same event, and how many real events produce no alert at all.
  5. 05 Health Endpoints Keeping liveness and readiness separate in code: the question each endpoint answers, the restart loop produced when a dependency check lands in the liveness endpoint, and running the same scenario in two modes to count the dropped requests and recovery time each one produces.

03

Resilience

Where patterns that withstand partial failure sit in application code: the duration limit on every outbound call, keeping retry and backoff from turning into a storm, isolating a broken dependency with a circuit breaker, separating resource pools with bulkheads, graceful degradation that keeps the core service running, rate limiting and throttling for fair sharing, classifying the types of partial failure, and post-incident analysis with a blameless root-cause exercise.

  1. 01 Timeouts The duration limit of every outbound call: measuring in source whether the limit sits at the call site or in the client wrapper, counting wrapped and uncovered call sites, the orphaned completion produced when the inner limit exceeds the outer one, and the testability difference between the two placements.
  2. 02 Retry and Backoff Measuring which layer a retry sits in: building the same rule separately into the client wrapper, the middle layer, and the gateway, whether the healthy call site gets retried too, the multiplier produced when three placements overlap, and the repeated side effect counted in deliveries.
  3. 03 Circuit Breaker Measuring where the breaker sits in the code: whether its state is kept per dependency or per call site, the calls reaching the dependency before each placement opens, the same threshold carrying two separate meanings under the two placements, and a rarely called path whose breaker never opens.
  4. 04 Bulkhead Pattern Measuring where the resource pool sits in the code: comparing a single shared pool against a pool per dependency at the same total slot count, a call site that never uses the pool exceeding the limit, and the behavior wrong sizing produces in concurrency and duration.
  5. 05 Graceful Degradation Placing graceful degradation in the code: gathering the core/optional field split into a schema object, comparing the shared wrapper at the call site as 7 and 12 lines, 6 of 14 call sites wrapped and 2 uncovered, and a fallback value that makes itself indistinguishable from a genuine zero.
  6. 06 Rate Limiting and Throttling The rate limiter's placement and key in the code: the same limiter seeing 2 of 8 path-access pairs at the gateway and 6 in the service's middle layer, 2 pairs falling outside every placement, and a single parameter of 20 turning into a cap ranging from 20 to 100,000 depending on the key.
  7. 07 Failure Modes What failure modes correspond to in application code: the same endpoint breaking eight separate ways, producing five different error identities at the call site, the pattern stack seeing five of them and missing three entirely, and the same zero value reaching the call site under six modes.
  8. 08 Post-Incident Analysis Rebuilding an incident from the signals this course produces: which signal answers each of six questions and in how many steps, counting the questions no signal can answer, actions returning to code, and how many of a human-driven analysis's actions make a question answerable in the next incident.

Start typing to search.

↑↓ ↵ Esc navigate · open · close