Stateful Applications

A persistent volume claim makes a pod’s data outlive the pod. That is enough for the bond between a single claim and a single volume. The measurement network’s verifier, though, is not one replica: it runs five, each replica has its own 40-Gi share of data, and the replicas cannot serve that data interchangeably. A replica owns a range of meters; that range’s data sits in its own volume.

A replica set builds the exact opposite assumption. For it, replicas are interchangeable; their names are not persistent across a run, because nothing is supposed to be tied to the name. The moment a volume tied to a name appears, this assumption breaks: if a re-created replica’s name has changed, no information is left to say which volume it should bind to.

This lesson names the devolved decision as follows: which replica owns which data, and where it binds when re-created. The system that makes this decision needs to know three items per replica — sequence number, volume binding, readiness. Three things are counted: how many orphaned volumes the two objects leave behind over the same sequence of events, what round cost ordered startup pays, and what breaks when order is not followed.

The Same Sequence of Events, Two Objects

The two objects pass through the same six events: setup, node failure, re-creation, scale-down, full restart, scale-up. The comparison only becomes meaningful at the end of this sequence; on any single event the two look the same.

• CO37 — Replicas, volumes, cluster membership, and shard ownership are a model built in node; no real cluster runs.
• CO38 — The measurement network is fictional. The verifier runs five replicas, each has its own 40-Gi volume, and they are placed on three nodes in sequence.
• CO39 — In a stateful set the replica name is derived from the sequence number, and the volume claim is tied to the name; in a replica set the name is regenerated on every creation, and every new name spawns a new volume claim.
• CO40 — On scale-down a stateful set removes the highest sequence number; in a replica set the departing replica is chosen by the generator. The removed replica’s volume is not deleted, it is left orphaned.
• CO41 — Readiness times are given by the generator, between 1 and 3 rounds. In an ordered startup each replica waits for the previous one’s readiness; shutdown runs in reverse order, one round per replica.
• CO42 — Membership model: sequence zero is the seed; a replica that opens while the seed is not up sets up its own membership, and there is no merging afterward. Shard ownership is distributed within each membership by shard mod member count; shard p’s data physically sits in the volume of sequence p mod 5. Seed 20260807.

// measure/identity.mjs — a stateful set and an unordered replica set in the same sequence of events.
// MODEL: replicas, volumes, membership, and shard ownership are a model built in node; no real
// cluster runs. The numbers below are the model's numbers.
const N = 5, NODES = ["n1", "n2", "n3"], SHARDS = 12, SEED = 20260807;
let s = SEED % 2147483647;
const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
const suffix = () => Array.from({ length: 4 }, () => "abcdefghijklmnopqrstuvwxyz"[Math.floor(random() * 26)]).join("");

function run(kind) {
  const volumes = new Map();                    // volume name -> owner replica name (null if empty)
  let replicas = [], log = [];
  const create = (seq) => {
    const name = kind === "ordered" ? `rep-${seq}` : `rep-${suffix()}`;
    const v = `data-${name}`, existed = volumes.has(v);
    volumes.set(v, name);
    return { seq, name, vol: v, existed, node: NODES[seq % NODES.length] };
  };
  const remove = (k) => { volumes.set(k.vol, null); replicas = replicas.filter((x) => x !== k); };
  const event = (label, fresh) => {
    const orphaned = [...volumes.values()].filter((v) => v === null).length;
    log.push([label, fresh.length, fresh.filter((k) => k.existed).length, replicas.length, orphaned]);
  };
  let y = Array.from({ length: N }, (_, i) => create(i)); replicas = [...y]; event("initial setup", y);
  for (const k of replicas.filter((k) => k.node === "n2")) remove(k); event("node n2 failure", []);
  const missing = [0, 1, 2, 3, 4].filter((i) => !replicas.some((k) => k.seq === i));
  y = missing.map((i) => create(i)); replicas.push(...y); event("re-creation", y);
  for (let i = 0; i < 2; i += 1) {             // scale 5 -> 3
    const candidate = kind === "ordered" ? replicas.reduce((a, b) => (b.seq > a.seq ? b : a))
      : replicas[Math.floor(random() * replicas.length)];
    remove(candidate);
  }
  const remainingSeq = replicas.map((k) => k.seq).sort();
  event(`scale 5->3 (leaving ${[0, 1, 2, 3, 4].filter((i) => !remainingSeq.includes(i)).join(",")})`, []);
  const old = [...replicas]; for (const k of old) remove(k);
  y = old.map((k) => create(k.seq)); replicas = [...y]; event("full restart", y);
  const empty = [0, 1, 2, 3, 4].filter((i) => !replicas.some((k) => k.seq === i)).slice(0, 2);
  y = empty.map((i) => create(i)); replicas.push(...y); event("scale 3->5", y);
  return { log, orphaned: [...volumes.values()].filter((v) => v === null).length, volumes: volumes.size };
}

console.log(`${"event".padEnd(26)}${"object".padStart(13)}${"created".padStart(11)}${"returned to old volume".padStart(24)}${"open replicas".padStart(15)}${"orphaned volumes".padStart(18)}`);
for (const kind of ["ordered", "unordered"]) {
  const r = run(kind);
  for (const [e, created, returned, open, orphaned] of r.log)
    console.log(`${e.padEnd(26)}${(kind === "ordered" ? "stateful set" : "replica set").padStart(13)}` +
      `${String(created).padStart(11)}${String(returned).padStart(24)}${String(open).padStart(15)}${String(orphaned).padStart(18)}`);
  console.log(`  -> ${kind}: ${r.volumes} volumes created in total, ${r.orphaned} orphaned (${r.orphaned * 40} Gi unclaimed)`);
}

// --- round cost of ordered startup and shutdown ---
console.log(`\n${"replicas".padEnd(9)}${"readiness rounds".padStart(20)}${"ordered startup".padStart(17)}${"parallel".padStart(10)}${"ordered shutdown".padStart(18)}${"parallel".padStart(10)}${"difference".padStart(12)}`);
s = SEED % 2147483647;
for (const n of [3, 5, 9]) {
  const ready = Array.from({ length: n }, () => 1 + Math.floor(random() * 3));
  const seqSum = ready.reduce((a, b) => a + b, 0), parallelMax = Math.max(...ready);
  console.log(`${String(n).padEnd(9)}${ready.join(",").padStart(20)}${String(seqSum).padStart(17)}${String(parallelMax).padStart(10)}` +
    `${String(n).padStart(18)}${String(1).padStart(10)}${String(seqSum + n - parallelMax - 1).padStart(12)}`);
}

// --- when order is not followed: cluster membership and data ownership ---
function membership(order) {
  const groups = [];
  for (const i of order) {
    const seeded = groups.find((g) => g.includes(0));
    if (i === 0 || !seeded) groups.push([i]); else seeded.push(i);
  }
  const claims = Array.from({ length: SHARDS }, () => []);
  for (const g of groups) for (let p = 0; p < SHARDS; p += 1) claims[p].push(g[p % g.length]);
  return {
    membership: groups.length,
    dual: claims.filter((a) => a.length > 1).length,
    wrong: claims.reduce((a, l, p) => a + l.filter((k) => k !== p % N).length, 0),
    noData: claims.filter((a, p) => !a.includes(p % N)).length,
  };
}
console.log(`\n${"startup order".padEnd(18)}${"membership".padStart(11)}${"dual-claimed".padStart(15)}${"wrong claims".padStart(14)}${"served without data".padStart(21)}`);
const orders = [[0, 1, 2, 3, 4]];
for (let i = 0; i < 3; i += 1) {
  const p = [0, 1, 2, 3, 4];
  for (let j = p.length - 1; j > 0; j -= 1) { const k = Math.floor(random() * (j + 1)); [p[j], p[k]] = [p[k], p[j]]; }
  orders.push(p);
}
for (const order of orders) {
  const r = membership(order);
  console.log(`${order.join(",").padEnd(18)}${String(r.membership).padStart(11)}${String(r.dual).padStart(15)}` +
    `${String(r.wrong).padStart(14)}${String(r.noData).padStart(21)}`);
}
console.log(`${SHARDS} shards, ${N} replicas, seed ${SEED}; information items: 3 per replica (sequence, volume binding, readiness) = ${N * 3}`);

event                            object    created  returned to old volume  open replicas  orphaned volumes
initial setup              stateful set          5                       0              5                 0
node n2 failure            stateful set          0                       0              3                 2
re-creation                stateful set          2                       2              5                 0
scale 5->3 (leaving 3,4)   stateful set          0                       0              3                 2
full restart               stateful set          3                       3              3                 2
scale 3->5                 stateful set          2                       2              5                 0
  -> ordered: 5 volumes created in total, 0 orphaned (0 Gi unclaimed)
initial setup               replica set          5                       0              5                 0
node n2 failure             replica set          0                       0              3                 2
re-creation                 replica set          2                       0              5                 2
scale 5->3 (leaving 0,1)    replica set          0                       0              3                 4
full restart                replica set          3                       0              3                 7
scale 3->5                  replica set          2                       0              5                 7
  -> unordered: 12 volumes created in total, 7 orphaned (280 Gi unclaimed)

replicas     readiness rounds  ordered startup  parallel  ordered shutdown  parallel  difference
3                       2,1,1                4         2                 3         1           4
5                   1,2,3,3,1               10         3                 5         1          11
9           1,1,1,1,2,3,1,1,1               12         3                 9         1          17

startup order      membership   dual-claimed  wrong claims  served without data
0,1,2,3,4                   1              0             0                    0
2,1,0,3,4                   3             12            30                    6
1,4,3,0,2                   4             12            38                    2
0,1,2,4,3                   1              0             4                    4
12 shards, 5 replicas, seed 20260807; information items: 3 per replica (sequence, volume binding, readiness) = 15

Is the Name Persistent, Does the Volume Come Back

The first two rows of the first table are the same for both objects: five replicas were set up, n2 went down, two replicas were lost, two volumes were left orphaned. The split starts on the re-creation row. Because the stateful set sees two missing sequence numbers, it regenerates the same two names; since the name is the same, the volume claim is the same too, and both replicas return to their old volume, orphan count drops to zero. The replica set generates two new names; the new name spawns a new claim, the count of replicas returning to their old volume is zero, and the two orphaned volumes stay right where they are.

The identity of the departing replicas on scale-down is also different. In the stateful set the departing replicas are 3 and 4 — the two highest sequences, and this rule does not change from run to run. In the replica set the departing replicas turned out to be 0 and 1; the selection comes from whatever order the generator gives, and a different run would send a different pair. The difference is not just which replica leaves: in the stateful set, which data the scale-down sets aside is known in advance, and when scale-up comes the same two replicas return to the same two volumes. In the table, this is the number 2 in the last row.

The last two rows of the sequence give the summary. Across the stateful set’s run 5 volumes were created and 0 ended up orphaned. The replica set created 12 volumes over the same events and 7 were left orphaned — 280 Gi of data sits in volumes no replica is bound to. This data is not lost; what is lost is the name that would reach it. When identity is not persistent, persistent storage itself becomes unreachable: persistence and identity are not problems that can be solved separately.

The seven orphaned volumes carry a second cost too. There is no lifecycle that deletes them — when a replica leaves, the volume stays, because the data in it may be valuable. The decision falls to a human, and that human has to know which night, which meter range, a name like data-rep-qxlm belongs to. Because the name is not persistent, this information sits nowhere; the only option left is to open the volume and look inside. In the stateful set this question is never asked, because the name data-rep-3 already carries the answer.

The Round Cost of Ordered Startup

The second table gives the price paid in exchange for a stable identity. A stateful set opens replicas one at a time; each replica waits for the previous one to pass its readiness check. Shutdown runs in reverse order, again one at a time.

For five replicas, readiness times are 1, 2, 3, 3, 1 rounds. Ordered startup takes their sum, 10 rounds; if all of them opened at once, the slowest replica’s time, 3 rounds, would be enough. Shutdown is 5 rounds ordered, 1 round parallel. The total gap between the two ends is 11 rounds. For nine replicas, startup rises to 12 rounds and shutdown to 9, a gap of 17 rounds. As scale grows the gap grows with it: parallel cost stays tied to the slowest replica while ordered cost grows linearly with replica count.

This cost is an information cost. For the system to enforce the order, it has to answer the question “is the previous replica really ready” at every step, and that answer comes from a check. If the check is relaxed, rounds are gained, but the order guarantee is lost too — the next table counts what that means.

There is also a case where this cost buys nothing, and it needs to be named. Order is paid for because there is a formation dependency among replicas: one is the seed, the others join it. In an application where replicas know nothing about each other, no such dependency exists; in that case the 11 rounds paid for five replicas buy nothing at all, they only stretch out the release window. In the measurement network the distinction is clear: the verifier’s replicas form a shared membership, the reading collector’s replicas do not — one needs a stateful set, the other does not.

When Order Breaks: Membership and Ownership

The third table separates two distinct breakdowns. In the correct order — 0,1,2,3,4 — a single membership forms, there are no dual-claimed shards, no wrong claims, no shards served without data.

In the 2,1,0,3,4 order the seed opens third. The two replicas that open before it, finding no seed, set up their own memberships: three separate memberships. Because each membership distributes the 12 shards according to its own member count, all 12 shards end up dual-claimed, and 30 claims fall to the wrong replica. Six shards are served by replicas that do not hold their data. In the 1,4,3,0,2 order the split rises to four, and wrong claims climb to 38.

The last row is the sneakiest case and deserves to be read on its own. In the 0,1,2,4,3 order the seed opens first, and membership does not split — a single membership, zero dual claims. Despite this, four shards fall to a replica that does not hold their data, because ownership is distributed by join order, and 4 and 3 have swapped places. So an order violation produces two different failures: the visible one is membership splitting, the invisible one is ownership drifting. In the second case the cluster looks healthy, queries are answered, only the answer to four shards comes from the wrong replica.

The devolution’s cost adds up in one line here: the decision rests on three items per replica — sequence number, volume binding, readiness; 15 items for five replicas. If all of these items are produced correctly and on time, orphaned volumes are 0, membership is 1, wrong claims are 0. If the sequence-number item is missing, 7 orphaned volumes result; if the readiness item is missing, split membership and drifting ownership appear.

Summary

• Over the same six events, the stateful set creates 5 volumes and leaves 0 orphaned; the replica set creates 12 volumes and leaves 7 orphaned (280 Gi of unclaimed data).
• Name persistence is a precondition for volume binding: on re-creation, 2 of 2 replicas returned to their old volume in the stateful set, and 0 did in the replica set.
• On scale-down the stateful set removes the highest sequence (3 and 4), and on scale-up the same two replicas return to the same volumes; in the replica set the departing pair (0 and 1) depends on the run, and there is no return.
• The cost of ordered startup is in rounds: 10 rounds against 3 for five replicas, 12 against 3 for nine; combined with shutdown, the total gap is 11 and 17 rounds.
• An order violation produces two separate failures: when the seed opens late, membership splits into 3–4 and all 12 shards end up dual-claimed; when the seed opens early but the join order is out of place, membership stays single but 4 shards fall to a replica without their data.
• The decision rests on three items per replica (sequence, volume binding, readiness); 15 items for five replicas.

Next Step

The difference between a stateful set and a replica set was measured, and both share one common assumption: pods do not finish. The replica count is brought to the desired value and held there; if a pod terminates, this is a failure, and it is restarted. The measurement network’s nightly batch job does not fit this assumption. That job starts, verifies the day’s readings, produces the billing entry, and finishes — finishing is the expected behavior, not a failure. Who decides it is finished, and by looking at what; how many times is it retried if it is not finished; if it has to start at the same time every night, who guards that time? The next lesson builds one-shot and recurring tasks with two separate objects and counts: how many attempts a completion guarantee costs, what result the parallelism and completion-count settings give, what happens on a missed run, and how three policies diverge from each other when a new run arrives before the previous one has finished.