Availability Patterns

The three lessons so far carried the same implicit assumption: every replica is up, and the only trouble comes from the link or the latency between them. When a replica goes down entirely, the question changes: a replacement must be found, requests routed to it, and the requests dropped during that transition counted.

How the transition is handled is a choice of pattern, and this lesson measures two in the same outage scenario. The mechanisms underneath were established elsewhere: majority rule, fencing, and the health-check-frequency-versus-false-failover trade-off belong to the Relational Database Administration course; whether the rest of a request survives when one component goes down — fault isolation — was measured in the Architectural Styles course. What gets measured here is how much the pattern costs in requests and writes.

Two Patterns

Active–passive: only one replica accepts writes, the other stands ready. Because the passive replica takes no writes, conflict is impossible. When the active replica goes down, no replica is left to accept writes; writes are dropped until failover completes.

Active–active: every replica accepts writes. When one goes down, the others keep working, and there is no failover window. In exchange, two replicas can write to the same key without knowing about each other; this is a conflicting write and needs a resolution rule. This lesson’s rule is last writer wins: of two values written in the same window, one is chosen and the other shows up nowhere.

In the tracking service, conflict is not contrived: the same shipment can pass through both zones’ scanners at a transfer point, and both zones can write in the same round.

The Setup

The model sets up a two-zone replica set. Scan events originate at a specific zone; zone b34 is unreachable from round 9 through round 12. Latency, detection, and promotion are model parameters in rounds.

// pattern/model.mjs — two-zone failover model. A round is an abstract step; latency, detection,
// and promotion are model parameters in rounds, not measured durations.
export const EVENTS = [            // [round, scan zone, status]
  [1, "b34", "accepted"], [2, "b34", "departed"],
  [3, "b34", "transfer-34"], [3, "b35", "line-35"],
  [4, "b34", "transfer-41"],
  [5, "b34", "out-for-delivery"], [5, "b35", "out-for-delivery-35"],
  [6, "b35", "delivery-attempt"],
  [7, "b34", "address-verification"], [7, "b35", "redelivery"],
  [8, "b34", "line-06"], [9, "b34", "transfer-06"], [10, "b35", "delivered"],
  [11, "b34", "signed"], [12, "b35", "closed"], [13, "b34", "invoice-ready"],
  [14, "b34", "invoice-issued"], [14, "b35", "invoice-approved"],
  [15, "b35", "archived"], [16, "b34", "archive-approved"],
];
export const OUTAGE = { zone: "b34", start: 9, end: 12 };
const LATENCY = 1, DETECTION = 2, PROMOTION = 1;
const other = (z) => (z === "b34" ? "b35" : "b34");
const alive = (zone, round) =>
  zone !== OUTAGE.zone || round < OUTAGE.start || round > OUTAGE.end;

export function run({ pattern, rounds: N }) {
  const s = { accepted: 0, dropped: 0, conflicting: 0, lostConflict: 0, lostUnpropagated: 0, messages: 0 };
  let flight = [], active = "b34", failoverStart = null, failoverRounds = 0;

  for (let round = 1; round <= N; round++) {
    if (round === OUTAGE.start) { // propagations that left the downed zone but have not yet arrived are lost
      s.lostUnpropagated += flight.filter((m) => m.source === OUTAGE.zone).length;
      flight = flight.filter((m) => m.source !== OUTAGE.zone);
    }
    flight = flight.filter((m) => m.arrives !== round); // arriving propagations become visible at the target

    if (pattern === "active-passive" && alive(active, round) === false) {
      if (failoverStart === null) failoverStart = round;
      if (round === failoverStart + DETECTION + PROMOTION) {
        active = other(active);
        failoverRounds = DETECTION + PROMOTION;
      }
    }

    const written = new Set();
    for (const [, scan] of EVENTS.filter(([t]) => t === round)) {
      const target = pattern === "active-passive" ? active
        : (alive(scan, round) ? scan : other(scan));
      if (alive(target, round) === false) { s.dropped += 1; continue; } // no replica available to accept it
      if (target !== scan) s.messages += 1;   // the scan was forwarded to the remote replica
      s.accepted += 1;
      s.messages += 1;                        // the accepting replica propagates to its neighbor
      written.add(target);
      flight.push({ source: target, target: other(target), arrives: round + LATENCY });
    }
    if (written.size === 2) {  // both zones wrote in the same round, each unaware of the other
      s.conflicting += 1;
      s.lostConflict += 1;     // last writer wins: the losing write shows up nowhere
    }
  }
  return { ...s, failoverRounds, active };
}

// pattern/measure.mjs — same outage scenario with two patterns: failover, dropped, conflict, lost
import { run, EVENTS, OUTAGE } from "./model.mjs";

const ROUNDS = 16;
const pad = (x, n) => String(x).padStart(n);
console.log(`${ROUNDS} rounds, ${EVENTS.length} scan events. zone ${OUTAGE.zone} is ` +
  `unreachable in rounds ${OUTAGE.start}-${OUTAGE.end}.`);
console.log();
console.log("pattern         | accept | drop | failover | conflict | lost(conflict) | lost(unprop.) | msgs");
console.log("----------------|--------|------|----------|----------|----------------|----------------|-----");
const results = {};
for (const pattern of ["active-passive", "active-active"]) {
  const r = run({ pattern, rounds: ROUNDS });
  results[pattern] = r;
  console.log(`${pattern.padEnd(15)} | ${pad(`${r.accepted}/${EVENTS.length}`, 6)} | ${pad(r.dropped, 4)} | ` +
    `${pad(`${r.failoverRounds} round`, 8)} | ${pad(r.conflicting, 8)} | ${pad(r.lostConflict, 15)} | ` +
    `${pad(r.lostUnpropagated, 14)} | ${pad(r.messages, 4)}`);
}
console.log();
console.log(`active-passive: active replica after failover = ${results["active-passive"].active}`);
console.log(`active-active: dropped requests during the outage rounds = ${results["active-active"].dropped}`);

node pattern/measure.mjs

16 rounds, 20 scan events. zone b34 is unreachable in rounds 9-12.

pattern         | accept | drop | failover | conflict | lost(conflict) | lost(unprop.) | msgs
----------------|--------|------|----------|----------|----------------|----------------|-----
active-passive  |  17/20 |    3 |  3 round |        0 |               0 |              1 |   24
active-active   |  20/20 |    0 |  0 round |        4 |               4 |              1 |   22

active-passive: active replica after failover = b35
active-active: dropped requests during the outage rounds = 0

Reading the Numbers

The accept column confirms the difference the patterns advertise. Active–passive accepted 17 of 20 scan events; failover took 3 rounds, and the 3 events arriving in those rounds were dropped. Active–active accepted 20/20 and dropped none during the outage, because b34’s scanners were rerouted to b35.

Which three events dropped matters: transfer-06 in round 9, delivered in round 10, signed in round 11. Two were scans at zone b35, which stayed up — dropped anyway. The reason is the definition of a passive replica: it accepts no writes. Being up does not mean a zone’s write gets accepted.

The conflict column shows where that cost moves instead. In active–active, zones wrote the same key without seeing each other in rounds 3, 5, 7, and 14: 4 conflicts. Last writer wins deleted one write per conflict, so 4 lost writes — 0 in active–passive, which instead carries 3 dropped requests. The two patterns pay for the same outage in different places: one as an outage, the other as lost data.

The last two columns show the cost the patterns share. lost(unprop.) is 1 in both: when b34 went down, one propagation had already left it but had not arrived, and it was lost. This comes from asynchronous propagation, not the pattern; with synchronous acknowledgment it would be 0, at the cost of the wait measured in the PACELC lesson. Message counts are close too (24 versus 22); the extra in active–active comes from forwarding a scan to the remote replica. Choosing a pattern is not a decision about saving messages.

Converting Rounds to Requests

A round is an abstract step. Converting the numbers into requests needs two assumptions: how many seconds a round is, and how many failovers happen per year. The volume figures come from the Back-of-the-Envelope Estimation lesson’s assumption table and are recalculated here.

// pattern/budget.mjs — converts the model's round counts into requests using the course's assumption table
const V = { dailyUsers: 2_000_000, queriesPerUser: 6, dailyShipments: 400_000,
  eventsPerShipment: 7, peakFactor: 3 };              // Back-of-the-Envelope Estimation lesson: V1-V4, V8
const YEAR = 365 * 24 * 3600, DAY = 86_400;
const SEC_PER_ROUND = 1;                              // assumption: one round equals one second
const FAILOVERS_PER_YEAR = 6;                         // assumption: how many failovers per year
const DUAL_SCAN = 0.05;                               // assumption: rate of events produced at both zones

const peakWrites = (V.dailyShipments * V.eventsPerShipment / DAY) * V.peakFactor;
const peakTotal = peakWrites + (V.dailyUsers * V.queriesPerUser / DAY) * V.peakFactor;
console.log(`peak writes = ${peakWrites.toFixed(2)} req/s, peak total = ${peakTotal.toFixed(2)} req/s`);
console.log();
for (const failoverRounds of [3, 5]) {
  const secs = failoverRounds * SEC_PER_ROUND;
  const annual = secs * FAILOVERS_PER_YEAR;
  console.log(`active-passive, failover ${failoverRounds} rounds (${secs} sec):`);
  console.log(`  dropped writes in window = ${(secs * peakWrites).toFixed(2)}, dropped total requests = ${(secs * peakTotal).toFixed(2)}`);
  console.log(`  ${FAILOVERS_PER_YEAR} failovers/year -> ${annual} sec, service availability = ` +
    `${(100 * (1 - annual / YEAR)).toFixed(5)}%`);
}
console.log();
for (const rate of [DUAL_SCAN, DUAL_SCAN * 2]) {
  const conflicting = V.dailyShipments * V.eventsPerShipment * rate;
  console.log(`active-active, dual-scan rate ${rate.toFixed(2)}: conflicting writes = ${conflicting}/day, ` +
    `last writer wins -> lost writes = ${conflicting}/day`);
}

peak writes = 97.22 req/s, peak total = 513.89 req/s

active-passive, failover 3 rounds (3 sec):
  dropped writes in window = 291.67, dropped total requests = 1541.67
  6 failovers/year -> 18 sec, service availability = 99.99994%
active-passive, failover 5 rounds (5 sec):
  dropped writes in window = 486.11, dropped total requests = 2569.44
  6 failovers/year -> 30 sec, service availability = 99.99990%

active-active, dual-scan rate 0.05: conflicting writes = 140000/day, last writer wins -> lost writes = 140000/day
active-active, dual-scan rate 0.10: conflicting writes = 280000/day, last writer wins -> lost writes = 280000/day

The active–passive failover window takes up very little of the budget: 6 failovers a year come to 18 seconds and leave service availability at 99.99994%. At 5 rounds, that becomes 30 seconds and 99.99990%. By the budget, this pattern is nearly free. The cost is not spread across the annual percentage but concentrated in a single window: every failover drops 291.67 status events and 1,541.67 requests at the peak, all within three seconds. An outage budget measures an average; it does not measure a loss concentrated in one window.

The active–active number is on an entirely different order. A 5 percent dual-scan rate produces 140,000 conflicting writes a day, and last writer wins turns every one into a lost write; doubling the rate gives 280,000. This loss is not an outage — it does not show up in the budget and does not announce itself as an error — it just leaves some shipments’ status history one step short.

Choosing the Pattern and Key Ownership

The measured numbers decide the choice per flow. For carrier events, active–passive fits: the 291.67 events dropped in a three-second window can be resent by the carrier, while a lost status update cannot be recovered. The same holds for end-of-day billing, where a missing piece of status history produces a wrong invoice. The tracking query, though, is a read, and either pattern can serve it from the passive replica — a read has no failover window.

Active–active’s conflict cost is not unavoidable. Conflict arises from two zones writing the same key; if writes are routed to a single zone by key — adding key ownership to the pattern — conflicting writes drop to 0, and each zone becomes the active replica for its own key set. The system is then active–active as a whole and active–passive for any single key. The 4 conflicts measured are what appears without an ownership rule — not the pattern’s unavoidable cost, but the result of a missing decision.

Summary

• In active–passive, only one replica accepts writes: conflict is 0, but once the active replica goes down, no writes are accepted until failover completes.
• In the same outage scenario, active–passive accepted 17/20 events, failover took 3 rounds, and 3 events were dropped; two of those originated at the zone that stayed up, because a passive replica takes no writes.
• Active–active accepted 20/20 and dropped no requests during the outage; in exchange it produced 4 conflicting writes, and last writer wins deleted 4 of them.
• The loss from asynchronous propagation came out the same in both patterns (1 unpropagated write); this cost comes from the write policy, not the pattern.
• Converted with the assumptions, active–passive failover costs 18 seconds a year (99.99994%) but drops 291.67 status events and 1,541.67 total requests in every window; an outage budget does not measure a loss concentrated in one window.
• Conflict is not the pattern’s unavoidable cost: once key ownership is added, every key has a single active replica and conflicting writes drop to 0.

Course Wrap-Up

The course took on a design problem in two steps. The Approach topic broke a one-line request into requirements: component boundaries, contracts, and quality thresholds were separated, five of eleven questions were counted answered, scope narrowing brought the components from 12 down to 5, and the assumption table gave a peak total of 513.89 req/s at the edge with 976 MB of daily data growth. Its last lesson gave design communication its form: a five-field decision record and a diagram whose edges are labeled from the calculation — the same flow carrying 416.67 req/s on the left edge and 41.67 req/s on the right.

Fundamental Properties turned each property into a number on top of those estimates: latency and throughput were measured as independent, throughput growing close to linearly as concurrency rose while median latency stayed nearly fixed; performance was separated from scalability, and the limits of horizontal and vertical scaling were drawn; service availability became a budget; why it gets spent was named with the CAP theorem; the latency trade-off outside a partition came from PACELC; consistency models were separated by convergence and monotonic reads; failover was measured as two patterns.

The course’s real rule stood beside these numbers: every number carried its class. An assumption is a chosen input with its sensitivity shown — 2,000,000 daily active users is one such number, and doubling it raises the peak total by 1.81 times. A computed value comes from the assumptions by arithmetic, produced with node — peak reads at 416.67 req/s, 712.48 GB of stored data, this lesson’s 1,541.67 dropped requests. A measurement comes from a setup and is flagged for its environment dependence — the throughput upper bound from the concurrency setup, and the last three lessons’ model runs: 6 rejected requests in a twelve-round partition, 8 monotonic-read violations in the three-replica model, 4 conflicting writes here. A number with no class carries no decision; that is why one of four decisions in the first lesson was unauditable.

The question left behind sits one layer down. Every property was defined, thresholds set, and patterns chosen — but the request has not yet entered the system. A tracking query starts with a domain name resolving, may pass through an edge cache, lands on a load balancer, and reaches the application through a gateway. None of that path was designed here — caching was only a multiplier in the calculation, routing a one-line choice in the model. The next course, The Traffic Layer, takes on that path: every stop a request passes through before reaching the application, the decisions made there, and how those decisions change the numbers calculated in this course.