Skip to content
academia.sh
tr
Log in

Course Advanced

Resilience and Reliability

By the end of this course

Start course

01

Fault Isolation

Patterns that cut a component's failure from spreading through the whole system: recognizing partial failure, slowdown, and network partition as separate failure modes, taking a broken dependency out of the circuit with a circuit breaker, splitting resource pools into bulkheads, the risk of retrying turning into a storm, budgeting the timeout across the chain, reporting excess load back to the source, controlled refusal of service, and degrading while the core function is preserved.

  1. 01 Failure Modes Removing the assumption that the parts work: naming partial failure, slowdown, stale content, and network partition as separate failure modes, telling a mode apart by its signature from the caller's eye, showing where two modes give the same signature, and comparing a component going down against slowing down in terms of blast radius.
  2. 02 Circuit Breaker Taking a broken dependency out of the circuit: the breaker protecting the caller's slot pool shared between services, the threshold being chosen by failure mode, comparing the consecutive-count and window-ratio rules across four modes, counting the calls saved and the neighboring requests rescued while open, and the cost of a false open that cuts off a healthy dependency, measured in rejected valid requests.
  3. 03 Bulkhead Pattern Splitting the resource pool by flow: comparing a single pool, where one slow dependency consumes the whole pool, against a bulkhead layout in the same failure scenario, the bulkhead protecting the neighboring flow without depending on detection at all, its failure-free-day cost measured in idle slots and reduced utilization, and the trade-off bulkhead size draws between the two days.
  4. 04 Retries and Storm Risk Retrying turning into a storm under the call amplification: measuring how far the call rate reaching a broken dependency multiplies when every layer of the chain retries, separating the arithmetic upper bound from the multiplier actually observed under backoff, tying the multiplier to the retry allowance under an outage and to the failure ratio under degradation, and cutting the storm by spreading a retry budget across the chain.
  5. 05 Timeout Budgets How the budget runs out at a slowing dependency: splitting the arm's remaining share across consecutive sub-calls with a greedy, equal, or weighted policy, the greedy split making the healthy step appear at fault, counting the caller's held slot separately from the dependency's abandoned work, and the tight split writing a gain only together with a cancellation signal.
  6. 06 Backpressure The overload signal propagating backward from the service boundary: measuring propagation depth across four settings from zero to the limit, partial propagation only moving the loss one link back, the signal reaching the limit turning an accepted-then-lost write into a never-accepted write, and an unlimited buffer turning loss into staleness.
  7. 07 Throttling and Load Shedding Two separate decisions in controlled service denial: throttling cutting off what exceeds a known rate at the source, load shedding rejecting by shedding-class order according to instantaneous capacity, throttling being unable to see capacity loss while shedding lets idle capacity be used, and counting how much of K01's 513.89 requests/s peak the two mechanisms protect together.
  8. 08 Graceful Degradation Separating the response's core set and answering with missing fields: measuring field skip and stale fallback against K01's 480-byte tracking response, a 15-minute outage zeroing out when an optional dependency goes down, the same pattern only cutting the outage to 14.02 minutes when a core dependency goes down, and isolation's limit written by that difference.

02

Recovery and Continuity

Design after the failure has already happened: choosing between the active–passive and active–active setup by failover cost, the health endpoint feeding automatic removal, converting recovery time and data loss objectives into numbers and comparing them against the outage budget, redundancy at the region and zone level, replicating scale units as stamps, and testing the plan with a drill.

  1. 01 Failover Design The three decisions failover leaves after the setup choice: choosing the consecutive-failed-check threshold that triggers promotion numerically between false failover and detection delay, the second transition window the failback policy produces, counting the writes a replica that responds but cannot do its job silently swallows, and converting all these transitions into the monthly outage budget's failure share.
  2. 02 Health Endpoint Monitoring Designing the content of the health endpoint that feeds failover: measuring the shallow check, readiness check, and deep check in the same failure schedule, how adding a shared dependency to the check evicts every replica from the pool at once, the check interval's trade-off between detection delay and false eviction, and counting the load the check adds to the dependency.
  3. 03 Recovery Objectives Defining the recovery time objective and the data loss objective as two separate thresholds: converting the data loss objective into a number of lost events via the backup interval and replication lag, computing how many failures the recovery time objective fits into K01's monthly failure share, and counting the failure-free-day cost of driving data loss to zero.
  4. 04 Redundancy Zones Measuring where copies are placed: separating the zone and the region as placement units, comparing three placement plans by surviving capacity and lost unpropagated writes under machine, zone, and region loss, computing the capacity that must sit idle to withstand the loss of one unit, and counting the overhead that cross-region replication carries on a failure-free day.
  5. 05 Deployment Stamps and Geo-Replicas Splitting the service into self-contained, replicable units: counting the links that cross the stamp boundary, comparing each component's blast radius across three placements, showing that the scope of the single resource shared across stamps stays independent of stamp count, and converting the load per stamp into K01 numbers.
  6. 06 Disaster Recovery Drill Testing the recovery plan with a controlled failure trial: comparing the plan's step durations against what the drill measures, surfacing a step the plan never wrote down, showing the recovery time objective holding by plan but breaking by drill, and counting the duplicated side effects and orphaned workflows that two delivery policies leave behind for workflows left half-done in the failover window.

03

Distributed Correctness

Getting the right result from work that is retried under failure, left half-finished, or moved in stages: neutralizing a duplicate request without producing side effects, balancing steps that cannot be rolled back with compensating transactions, moving a large payload outside the message, granting limited-time authorization for direct resource access, collecting runtime settings externally, and replacing a legacy system route by route.

  1. 01 Idempotent Operations Neutralizing a duplicate request during recovery: counting the three sources of duplication across a ninety-second failure window, an idempotency key's scope suppressing valid events when it is chosen too narrow, and the record, byte, and write-row cost the key ledger adds at K01 scale.
  2. 02 Compensating Transactions Balancing steps that cannot be rolled back: only two of twenty-four orderings satisfying the dependencies, the case where an unrollbackable step cannot be moved to the end, a failing compensation call leaving 2.979 jobs a day stuck, and the intermediate state window coming from stuck work rather than from rollback.
  3. 03 Claim Check Pattern Moving a large payload outside the message: proof of delivery raising the message to 40,220 bytes, the claim check lowering message bytes by 141.6 times, the queue's backlog dropping from 3016.5 MB to 21.3 MB, the proof never shrinking on the storage side, and the orphaned payload and dangling claim check that splitting a single write into two brings.
  4. 04 Valet Key Pattern Time-limited authorization for direct resource access: taking proof bytes off the service's path drops per-transfer bytes from 80,000 to 512, the held-slot pool saturating at 64 and dropping 28 transfers when the store slows down, the open authorization window growing linearly with key duration, and scope narrowing's payoff being non-linear.
  5. 05 External Configuration Store Centralizing runtime settings: a change propagating to three replicas in 25 seconds, caching cutting store reads 5139x, a wrong value under staged rollout producing 20,556 faulty responses instead of 61,668, and running on the last known good value zeroing out a 40-second equivalent outage when the store goes down.
  6. 06 Strangler Fig Pattern Migrating the old system one route at a time: moving eight routes in two different orders, the volume order carrying traffic 3.89 times faster while growing the dual-write load 1.52 times, the migration facade drawing a share from the timeout budget, and reversibility rising from one round to 3360 seconds the moment dual writes stop.

Start typing to search.

↑↓ ↵ Esc navigate · open · close