Availability and Consistency Tension

The previous lesson turned service availability into a budget: percentages became annual outage duration, and how that budget gets spent was counted. The open question is why it gets spent. Part of an outage comes from failure; part comes from a deliberate decision. When one replica loses touch with the other, the system must choose between two things: responding, or responding correctly.

This lesson names that choice. It is not an arbitrary preference but a decision every system standing on a network is forced to make, usually without realizing it. Its name is the CAP theorem, the field’s most misread sentence.

Three Letters and the Letter Not Chosen

The theorem names three properties.

Consistency: every read sees the most recently acknowledged write; the system behaves as though it had only one copy. Service availability: every request to a running replica gets a response — not an error, a response. This must not be confused with interface accessibility, covered in the Accessible Component Patterns course and absent from this lesson. Partition tolerance: the system keeps working even when messages between replicas are lost.

The network partition in the third term is not the partitioning covered in the Relational Database Administration course: partitioning is deliberately splitting data into pieces; a network partition is replicas losing sight of each other. Same root, two separate phenomena.

The misreading starts here, because the three letters get presented as three options. They are not. A network partition is not a design choice but a fact: a cable breaks, a switch restarts, a zone cannot hear another for a few seconds. There is no “not choosing” this fact — only what the system does once it occurs. The theorem’s actual statement: while a network partition lasts, consistency and service availability cannot both be provided. With no partition, the theorem is silent.

Replication and replication lag were established in the Relational Database Administration course and are not retold here. What gets measured is what a policy does once the replicas’ link is broken.

The Two-Zone Model

Instead of standing up a real cluster, the behavior is shown with an in-process model: replicas are objects, and whether the link is down is an explicit parameter. Every number below comes from running this model, not a real network; the model reads no clock, so it comes out the same every run.

The shipment tracking service has two zones: carrier status events arrive at zone b34, and tracking queries can arrive at either zone.

// cap/model.mjs — two-zone replica model: replicas are objects, link and messages are open parameters
export function build() {
  const state = { link: true, messages: 0 };
  const replica = { b34: new Map(), b35: new Map() };
  const neighbor = { b34: "b35", b35: "b34" };

  function write(zone, key, value, round) {
    replica[zone].set(key, { value, round });
    if (state.link === false) return { propagated: 0 };
    state.messages += 1;
    replica[neighbor[zone]].set(key, { value, round });
    return { propagated: 1 };
  }

  // "respond": returns the local record without verifying it. "reject": a response is given
  // only when it can be verified with the neighbor; if the link is down the request is rejected.
  function read(zone, key, policy) {
    const local = replica[zone].get(key) ?? null;
    if (policy === "respond") return { response: local, rejected: false };
    if (state.link === false) return { response: null, rejected: true };
    state.messages += 1;
    return { response: local, rejected: false };
  }

  return { state, replica, write, read };
}

The two policies are separated in the code by a single line. The reject policy returns nothing while the link is down; even holding a value, it will not respond, because it cannot verify that value is the latest one. The entire trade-off lives inside that one line.

Same Scenario, Two Policies

The driver script runs a twelve-round scenario. In every round, a carrier status event is written to zone b34 and a tracking query arrives; the query lands on the two zones in turn. The link is down from round 3 through round 8.

// cap/measure.mjs — same scenario with two read policies: answered, rejected, and stale counts
import { build } from "./model.mjs";

const TRACKING = "TR-9042";
const EVENTS = ["accepted", "departed", "transfer-34", "transfer-41", "line-35", "out-for-delivery",
  "delivery-attempt", "address-verification", "redelivery", "delivered", "signed", "closed"];
const PARTITION = [3, 8]; // link is down between these rounds (both ends included)

function run(policy) {
  const m = build();
  const s = { requests: 0, responses: 0, rejected: 0, stale: 0, trace: [] };
  for (let round = 1; round <= EVENTS.length; round++) {
    m.state.link = round < PARTITION[0] || round > PARTITION[1];
    m.write("b34", TRACKING, EVENTS[round - 1], round); // the carrier event always arrives at b34
    const actual = m.replica.b34.get(TRACKING);
    const zone = round % 2 === 1 ? "b35" : "b34"; // the tracking query alternates between the two zones
    const r = m.read(zone, TRACKING, policy);
    s.requests += 1;
    if (r.rejected) { s.rejected += 1; s.trace.push(`${zone}:rejected`); continue; }
    s.responses += 1;
    const stale = r.response.round < actual.round;
    if (stale) s.stale += 1;
    s.trace.push(`${zone}:${r.response.value}${stale ? "~" : ""}`);
  }
  return { ...s, messages: m.state.messages };
}

const windowOf = (a) => a.slice(PARTITION[0] - 1, PARTITION[1]);
console.log(`scenario: ${EVENTS.length} rounds, link down in rounds ${PARTITION[0]}-${PARTITION[1]}`);
console.log("policy      | resp. | rej | stale | msgs. | partition window");
console.log("------------|-------|-----|-------|-------|------------------");
const results = {};
for (const p of ["respond", "reject"]) {
  const s = run(p);
  results[p] = s;
  console.log(`${p.padEnd(11)} | ${String(`${s.responses}/${s.requests}`).padStart(5)} | ` +
    `${String(s.rejected).padStart(3)} | ${String(s.stale).padStart(5)} | ` +
    `${String(s.messages).padStart(5)} | ${windowOf(s.trace).join(" ")}`);
}
const pw = PARTITION[1] - PARTITION[0] + 1;
const count = (p, k) => windowOf(results[p].trace).filter(k).length;
console.log(`in the partition window (${pw} rounds): respond stale = ` +
  `${count("respond", (x) => x.endsWith("~"))}/${pw}, reject rejected = ` +
  `${count("reject", (x) => x.endsWith("rejected"))}/${pw}`);
console.log(`outside the partition (${EVENTS.length - pw} rounds): both policies gave the same response`);

node cap/measure.mjs

scenario: 12 rounds, link down in rounds 3-8
policy      | resp. | rej | stale | msgs. | partition window
------------|-------|-----|-------|-------|------------------
respond     | 12/12 |   0 |     3 |     6 | b35:departed~ b34:transfer-41 b35:departed~ b34:out-for-delivery b35:departed~ b34:address-verification
reject      |  6/12 |   6 |     0 |    12 | b35:rejected b34:rejected b35:rejected b34:rejected b35:rejected b34:rejected
in the partition window (6 rounds): respond stale = 3/6, reject rejected = 6/6
outside the partition (6 rounds): both policies gave the same response

Reading the Numbers

The last line shows where the theorem stays silent: in the six rounds outside the partition, the two policies cannot be told apart. The trade-off exists only inside the window, and there it is sharp.

The respond policy answered 12/12 requests and rejected none; in exchange it produced 3 stale responses. The ~ mark in the trace flags a stale response: zone b35 kept returning departed throughout the partition, while the shipment had already moved to out-for-delivery and address-verification. The query got a response, but the response was not telling the truth.

The reject policy answered 6/12 requests, turning back all six inside the partition window and producing not one stale response. One detail is easy to miss: it also rejected requests arriving at zone b34 — even though b34 held the newest value every round. The replica cannot know its value is the newest; it cannot learn, without asking its neighbor, whether the neighbor holds a write it knows nothing about. The guarantee requires not returning the correct value but being able to verify it; cut the verification channel and the guarantee is cut too.

The message column counts a third cost. Respond spent 6 messages: the six writes propagated only while the link was up. Reject spent 12 — the same six propagations plus one verification message per answered read. This has nothing to do with the partition; it would be there without one too. That difference is the next lesson’s subject.

Where the Budget Goes

The rates the model produces convert into an annual budget. The inputs are assumptions: how many times a year a network partition occurs and how long it lasts, and how many tracking queries arrive per second during it. The second run shows the assumption’s sensitivity: what happens if the partition duration doubles.

// cap/budget.mjs — converts the model's rates into an annual budget; the inputs are assumptions
const YEAR = 365 * 24 * 3600;
const ASSUMPTION = { partition: 12, duration: 40, query: 30 }; // times/year, seconds, tracking queries/second
const MODEL = { rejectRate: 6 / 6, staleRate: 3 / 6 };         // rates from the measurement

const availabilityPercent = (s) => (100 * (1 - s / YEAR)).toFixed(4);
for (const factor of [1, 2]) {
  const secs = ASSUMPTION.partition * ASSUMPTION.duration * factor;
  const queries = secs * ASSUMPTION.query;
  console.log(`partition ${secs} sec/year (${ASSUMPTION.partition} times x ${ASSUMPTION.duration * factor} sec)` +
    ` -> ${queries} tracking queries in the window`);
  console.log(`  reject   : rejected = ${Math.round(queries * MODEL.rejectRate)} requests,` +
    ` service availability = ${availabilityPercent(secs * MODEL.rejectRate)}%, stale = 0`);
  console.log(`  respond  : rejected = 0 requests,` +
    ` service availability = ${availabilityPercent(0)}%, stale = ` +
    `${Math.round(queries * MODEL.staleRate)} responses`);
}

partition 480 sec/year (12 times x 40 sec) -> 14400 tracking queries in the window
  reject   : rejected = 14400 requests, service availability = 99.9985%, stale = 0
  respond  : rejected = 0 requests, service availability = 100.0000%, stale = 7200 responses
partition 960 sec/year (12 times x 80 sec) -> 28800 tracking queries in the window
  reject   : rejected = 28800 requests, service availability = 99.9970%, stale = 0
  respond  : rejected = 0 requests, service availability = 100.0000%, stale = 14400 responses

The two policies pay for the same fact in two different currencies. Reject spent 480 seconds of budget and cut service availability to 99.9985%, producing no wrong information in exchange. Respond spent nothing from the budget but told 7,200 tracking queries a stale status. When the assumption doubled, both numbers grew linearly: 99.9970% and 14,400.

The real conclusion: the two costs are not comparable. An outage ends; an incorrectly reported delivery status has already interfered with whoever looked at it and made a decision. So the decision rests on which error the flow can tolerate, not the size of the numbers. A tracking query showing a status a few seconds stale is tolerable; end-of-day billing invoicing against a stale status is not. Two flows in the same system, two separate policies.

The Theorem’s Misreadings

Four misreadings are common; the measurement refutes all four.

“Pick two of three.” The theorem offers no menu. Partition tolerance is not a chosen property but the consequence of standing on a network. The choice is between two letters, and it applies only while a partition lasts.

“Our system is CA.” For a system distributed over a network this is not a class but a design that never considered a partition. When a partition arrives the system makes a choice regardless; if the code never wrote that choice down, the library’s default behavior makes it instead. The measurement’s one-line difference shows exactly where that default is hiding.

“The choice applies to the whole system.” Both runs in the measurement used the same model; only the read policy differed. A policy can be given per request — the tracking query can run under respond while the billing read runs under reject.

“Choosing consistency just makes the system slower — that is all.” The message column shows this is half true: verification does spend messages. But what a partition spends is not time, it is the response itself — 6 requests were rejected. Two separate costs, handled separately.

Summary

• The CAP theorem offers no three-way choice: a network partition is a fact, and the choice between consistency and service availability applies only while it lasts.
• In the same twelve-round scenario, respond answered 12/12 requests and produced 3 stale responses; reject answered 6/12, rejected 6 requests, and produced 0 stale responses.
• In the six rounds outside the partition, the two policies could not be told apart; the trade-off exists only inside the window.
• Reject also rejected requests at the zone holding the newest value: the guarantee requires not the correct value but the ability to verify it.
• Under the assumption of 12 partitions a year lasting 40 seconds each, reject rejected 14,400 requests and cut service availability to 99.9985%; respond spent nothing from the budget but produced 7,200 stale responses. Doubling the assumption doubled both numbers.
• The two costs are not in the same unit; the decision is made per request, based on which error the flow can tolerate.

Next Step

One column in the measurement was left unexplained. Reject spent 12 messages, respond spent 6, and that difference had nothing to do with the partition: verification messages were sent while the link was up too. So the choice does not end even without a network partition. There remains a decision between having every write acknowledged by both replicas and writing to one while waiting for it to propagate to the other; the cost is not a rejected request but a number of rounds waited. The next lesson names this region where CAP stays silent, and measures three things there: the acknowledgment-wait step, the staleness window seen on reads, and the number of messages per write.