Storage and Volumes

Settings moved outside the pod: configuration and secrets stand as separate objects, and the pod reads them at run time. The same cannot be said for data. Every byte the pod writes is still tied to the pod’s lifetime; when the pod is re-created it starts with a fresh writable layer, and yesterday’s working state is not there.

On a single machine, the solution to this problem was already measured — moving data out of the writable layer, into a named volume or a directory referenced by an absolute path — and what was counted there was the bytes remaining after the container was deleted. In the cluster, that number is taken as solved; the question asked is a different one. If a pod is to find the same bytes no matter which node it opens on, the storage holding those bytes has to be reachable from that node. This is a decision the application cannot make on its own: the application knows what it wants, not what the cluster offers.

The devolution here happens through a declaration split in two. The application writes a persistent volume claim; the cluster declares a volume pool; what matches the two is a storage driver interface. This lesson counts three things: how many requests go unmet and why, how many options the access mode eliminates from placement, and how many bindings turn out wrong once the inventory declaration goes stale.

Request, Supply, and Matching

Request and supply have to use the same vocabulary. The application declares three items: how much space, which access mode, which performance class. The cluster declares four items per volume: size, class, the modes it supports, and topology. The matching rule is nothing more than three comparisons; what is hard is not the comparison, it is both sides keeping their declarations current.

• CO31 — The cluster, volume pool, node topology, and inventory declaration are a model built in node; no real cluster runs. The numbers below are the model’s numbers.
• CO32 — The measurement network is fictional. Ten persistent volume claims come from the network’s services and its nightly batch job; the names are drawn from that fiction.
• CO33 — The performance classes are fast, standard, archive, and ultra-fast; the access modes are single-writer, multi-reader, multi-writer.
• CO34 — The cluster is six nodes and two regions. A volume is either bound to a single node or open to an entire region; there is no third topology.
• CO35 — In the staleness sweep the pool has 24 volumes, the run is 10 rounds, and four requests arrive per round. Each round two volumes are bound by another claimant outside the cluster, and one volume is freed.
• CO36 — Randomness is produced by a linear congruential generator, seed 20260801. Three inventory ages run over the same pool and the same request sequence.

// measure/claim-supply.mjs — matching a persistent volume claim against cluster supply and the placement constraint.
// MODEL: the cluster, volume pool, node topology, and inventory declaration are a model built in
// node; no real cluster runs. The numbers below are the model's numbers.
const NODES = [["n1", "region-a"], ["n2", "region-a"], ["n3", "region-a"],
  ["n4", "region-b"], ["n5", "region-b"], ["n6", "region-b"]];

// request: name, size(Gi), access mode, perf class  -> the 3 items the application declares
const REQUESTS = [
  ["collector-state", 20, "single-writer", "fast"], ["verifier-cache", 5, "single-writer", "standard"],
  ["billing-data", 200, "single-writer", "fast"], ["work-order-data", 50, "single-writer", "standard"],
  ["tariff-table", 2, "multi-reader", "standard"], ["nightly-output", 100, "multi-writer", "standard"],
  ["archive-2025", 500, "multi-reader", "archive"], ["raw-measurement", 300, "single-writer", "fast"],
  ["report-share", 40, "multi-writer", "fast"], ["closing-ledger", 30, "single-writer", "ultra-fast"]];

// supply: name, size(Gi), class, supported modes, region, bound node (null if region-wide)
const SUPPLY = [
  ["v01", 30, "fast", ["single-writer"], "region-a", "n1"], ["v02", 8, "standard", ["single-writer"], "region-a", "n2"],
  ["v03", 250, "fast", ["single-writer"], "region-b", "n4"], ["v04", 60, "standard", ["single-writer"], "region-b", "n5"],
  ["v05", 4, "standard", ["single-writer", "multi-reader"], "region-a", null],
  ["v06", 120, "standard", ["single-writer"], "region-a", null], ["v07", 400, "archive", ["single-writer", "multi-reader"], "region-b", null],
  ["v08", 20, "fast", ["single-writer"], "region-a", "n3"], ["v09", 512, "standard", ["single-writer", "multi-reader"], "region-b", null],
  ["v10", 50, "fast", ["single-writer", "multi-reader", "multi-writer"], "region-a", null]];

const fits = (t, b) => b[1] >= t[1] && b[2] === t[3] && b[3].includes(t[2]);
function match(rule) {                                        // rule: first-fit | best-fit
  const free = new Set(SUPPLY.map((b) => b[0])), result = [];
  for (const t of REQUESTS) {
    const candidates = SUPPLY.filter((b) => free.has(b[0]) && fits(t, b));
    if (candidates.length) {
      const s = rule === "first-fit" ? candidates[0] : candidates.reduce((a, b) => (b[1] < a[1] ? b : a));
      free.delete(s[0]); result.push([t, s, null]); continue;
    }
    const hasClass = SUPPLY.some((b) => free.has(b[0]) && b[2] === t[3]);
    const hasMode = SUPPLY.some((b) => free.has(b[0]) && b[2] === t[3] && b[3].includes(t[2]));
    result.push([t, null, !hasClass ? "no class" : !hasMode ? "no mode" : "size too small"]);
  }
  return result;
}

const A = match("best-fit"), reason = {};
console.log(`${"request".padEnd(22)}${"Gi".padStart(5)}${"mode".padStart(15)}${"class".padStart(12)}${"volume".padStart(8)}${"result".padStart(16)}`);
for (const [t, b, n] of A) {
  if (n) reason[n] = (reason[n] || 0) + 1;
  console.log(`${t[0].padEnd(22)}${String(t[1]).padStart(5)}${t[2].padStart(15)}${t[3].padStart(12)}` +
    `${(b ? b[0] : "-").padStart(8)}${(b ? "bound" : n).padStart(16)}`);
}
const bound = A.filter(([, b]) => b);
console.log(`${REQUESTS.length} requests, ${bound.length} bound, ${REQUESTS.length - bound.length} unmet: ` +
  Object.entries(reason).map(([k, v]) => `${k} ${v}`).join(", "));
for (const k of ["first-fit", "best-fit"]) {
  const r = match(k).filter(([, b]) => b);
  console.log(`rule ${k.padEnd(11)} bound ${r.length}  overallocated ${r.reduce((a, [t, b]) => a + b[1] - t[1], 0)} Gi`);
}

const eligible = (b) => NODES.filter(([d, bo]) => (b[5] ? d === b[5] : bo === b[4])).map(([d]) => d);
console.log(`\n${"request".padEnd(22)}${"mode".padStart(15)}${"volume topology".padStart(20)}${"eligible nodes".padStart(15)}${"excluded".padStart(9)}`);
let eligibleTotal = 0;
for (const [t, b] of bound) {
  const u = eligible(b); eligibleTotal += u.length;
  console.log(`${t[0].padEnd(22)}${t[2].padStart(15)}${(b[5] ? `node ${b[5]}` : `region ${b[4]}`).padStart(20)}` +
    `${u.join(",").padStart(15)}${String(NODES.length - u.length).padStart(9)}`);
}
console.log(`${bound.length * NODES.length} placement options with no constraint, ${eligibleTotal} remained; excluded ${bound.length * NODES.length - eligibleTotal}`);
for (const k of ["single-writer", "multi-reader", "multi-writer"]) {
  const g = bound.filter(([t]) => t[2] === k); if (!g.length) continue;
  console.log(`mode ${k.padEnd(14)} requests ${g.length}  avg eligible nodes per request ` +
    `${(g.reduce((a, [, b]) => a + eligible(b).length, 0) / g.length).toFixed(2)}`);
}

// --- inventory staleness: the decision is made with a stale declaration ---
const SEED = 20260801;
let s = SEED % 2147483647;
const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
const POOL = 24, ROUNDS = 10, ROUND_REQUESTS = 4, EXTERNAL = 2;
const pool = Array.from({ length: POOL }, (_, i) => [`h${String(i + 1).padStart(2, "0")}`,
  [10, 40, 120, 400][Math.floor(random() * 4)], ["fast", "standard", "archive"][Math.floor(random() * 3)]]);
const incoming = [];
for (let t = 0; t < ROUNDS; t += 1)
  for (let i = 0; i < ROUND_REQUESTS; i += 1)
    incoming.push([t, [10, 40, 120][Math.floor(random() * 3)], ["fast", "standard", "archive"][Math.floor(random() * 3)]]);

function run(age) {
  s = SEED % 2147483647; for (let i = 0; i < 3 * POOL + 2 * incoming.length; i += 1) random();
  const occupied = new Set(), history = [], queue = [];
  let correct = 0, wrong = 0, undecided = 0, waitTotal = 0;
  for (let t = 0; t < ROUNDS; t += 1) {
    history.push(new Set(occupied));
    const inventory = age === 0 ? occupied : history[Math.max(0, t - age)];
    queue.push(...incoming.filter((g) => g[0] === t));
    for (const request of queue.splice(0, queue.length)) {
      const candidate = pool.find((b) => !inventory.has(b[0]) && b[1] >= request[1] && b[2] === request[2]);
      if (!candidate) { undecided += 1; queue.push(request); continue; }
      if (occupied.has(candidate[0])) { wrong += 1; queue.push(request); continue; }
      occupied.add(candidate[0]); correct += 1; waitTotal += t - request[0];
    }
    for (let i = 0; i < EXTERNAL; i += 1) {                   // another claimant binds a volume from the pool
      const free = pool.filter((b) => !occupied.has(b[0]));
      if (free.length) occupied.add(free[Math.floor(random() * free.length)][0]);
    }
    const d = [...occupied]; if (d.length) occupied.delete(d[Math.floor(random() * d.length)]);
  }
  return { correct, wrong, undecided, waitTotal, pending: queue.length };
}

console.log(`\n${"inventory age"}${"correct".padStart(8)}${"wrong bind".padStart(16)}${"undecided".padStart(16)}` +
  `${"pending".padStart(10)}${"avg wait rounds".padStart(19)}${"declarations per round".padStart(23)}`);
for (const age of [0, 1, 3]) {
  const r = run(age);
  console.log(`${String(age).padEnd(13)}${String(r.correct).padStart(8)}${String(r.wrong).padStart(16)}` +
    `${String(r.undecided).padStart(16)}${String(r.pending).padStart(10)}${(r.waitTotal / Math.max(1, r.correct)).toFixed(2).padStart(19)}` +
    `${String(Math.round(POOL * 3 / (age + 1))).padStart(23)}`);
}
console.log(`information items: ${REQUESTS.length} requests x 3 + ${SUPPLY.length} volumes x 4 + ${NODES.length} nodes x 1 = ` +
  `${REQUESTS.length * 3 + SUPPLY.length * 4 + NODES.length} items (seed ${SEED})`);

request                  Gi           mode       class  volume          result
collector-state          20  single-writer        fast     v08           bound
verifier-cache            5  single-writer    standard     v02           bound
billing-data            200  single-writer        fast     v03           bound
work-order-data          50  single-writer    standard     v04           bound
tariff-table              2   multi-reader    standard     v05           bound
nightly-output          100   multi-writer    standard       -         no mode
archive-2025            500   multi-reader     archive       -  size too small
raw-measurement         300  single-writer        fast       -  size too small
report-share             40   multi-writer        fast     v10           bound
closing-ledger           30  single-writer  ultra-fast       -        no class
10 requests, 6 bound, 4 unmet: no mode 1, size too small 2, no class 1
rule first-fit   bound 6  overallocated 85 Gi
rule best-fit    bound 6  overallocated 75 Gi

request                          mode     volume topology eligible nodes excluded
collector-state         single-writer             node n3             n3        5
verifier-cache          single-writer             node n2             n2        5
billing-data            single-writer             node n4             n4        5
work-order-data         single-writer             node n5             n5        5
tariff-table             multi-reader     region region-a       n1,n2,n3        3
report-share             multi-writer     region region-a       n1,n2,n3        3
36 placement options with no constraint, 10 remained; excluded 26
mode single-writer  requests 4  avg eligible nodes per request 1.00
mode multi-reader   requests 1  avg eligible nodes per request 3.00
mode multi-writer   requests 1  avg eligible nodes per request 3.00

inventory age correct      wrong bind       undecided   pending    avg wait rounds declarations per round
0                  21               0              80        19               0.33                     72
1                  15              86              44        25               1.93                     36
3                  10             134              33        30               3.20                     18
information items: 10 requests x 3 + 10 volumes x 4 + 6 nodes x 1 = 76 items (seed 20260801)

The Unmet Request: Four Rejections, Three Reasons

Six of ten requests were bound, four went unmet, and the rejection reasons are not all in the same class. One request said no mode: a multi-writer volume is requested for the nightly batch job’s output, the pool has a volume of that size and class, but none of them supports that mode. Two requests said size too small: in the archive class there is only a 400-Gi volume, and 500 Gi was requested; in the fast class the largest volume left is 50 Gi, and 300 Gi was requested. One request said no class: not a single volume declares a class called ultra-fast.

These three reasons do not call for the same work, and separating them is the first payoff of devolving. A size rejection is a capacity decision: a larger volume gets added to the pool. A mode rejection is an interface decision; no amount of growing fixes it, because offering multi-writer mode is a property of the storage itself. A class rejection can be a vocabulary error: the application has declared a class name the cluster does not recognize. Once the decision is devolved, the rejection reason has to be produced by the machine, or the request just says “not bound.”

The pair of second-block rows gives the cost of the selection rule. Choosing the first fitting volume versus choosing the smallest one that fits does not change the count met, six in both; what changes is waste: eighty-five Gi against seventy-five Gi overallocated. A rule that consumes large volumes early ends up unable to meet large requests that arrive later. The selection rule does not change today’s decision, it changes tomorrow’s met rate, and the system does not know what requests tomorrow will bring.

Access Mode: Nodes Eliminated From Placement

The second table shows that the storage decision is really a placement decision. For the six bound requests, there were 36 placement options in the six-node cluster; mode and topology together eliminated 26 of them, leaving 10 options.

The elimination splits sharply by mode. The four requests in single-writer mode have volumes bound to a specific node; a single eligible node is left for each, and five are eliminated. The two requests in multi-reader and multi-writer mode have volumes open to the whole region; three nodes are eligible for them, and three are eliminated. The average eligible node count is 1.00 in single-writer mode and 3.00 in the others.

The consequence is that placement’s first question is no longer “which node has room.” The moment a pod requests a volume in single-writer mode, the node is already chosen; the only question left is whether that node has enough processor and memory, and if not, placement stays unsolvable with no other node to fall back to. The devolved decision ties two systems together: the storage match narrows the scheduler’s selection space, and the scheduler has to know about that narrowing. If it does not, it puts the pod on an ineligible node, the volume does not bind, and the pod does not open.

In the measurement network the distinction turns into a readable rule: the tariff table is declared in multi-reader mode because every verifier replica reads it, and three nodes stay eligible; the work order data is held by a single writer, so single-writer mode nails it to one node.

The Inventory’s Age and the Wrong Binding

The third table gives the devolution’s real cost. The matching decision rests on a declared inventory, and past a certain moment that inventory no longer tells the truth.

At zero inventory age — that is, when the pool’s actual occupancy is read at the moment of decision — there is no wrong binding: 21 requests bound correctly, average wait 0.33 rounds. When the age rises to one round, correct bindings fall to 15 and 86 wrong bindings appear: the system picks a volume that looks free in the inventory, discovers at binding time that the volume is occupied, and the request returns to the queue and retries next round. At an age of three rounds, correct bindings fall to 10 and wrong bindings rise to 134; average wait climbs from 0.33 to 3.20 rounds, and the number of requests still pending at the end of the run rises from 19 to 30.

The undecided column moves in the opposite direction, and this is deceptive at first glance: it falls from 80 to 33. The number is not improving — because the pool looks emptier than it actually is in the stale inventory, the system stops saying “no suitable volume” and starts picking the wrong volume instead. In other words, delayed information turns one class of decision into another: it produces a wrong “found it” in place of an honest “I do not know.” The second is more expensive, because a binding attempt is made, it fails, and a round is spent.

The last column is the other end of the cost. Keeping the age at zero requires 72 declaration items per round — re-reading three items for each of the pool’s 24 volumes every round. At an age of three this number falls to 18, declaration load drops to a quarter, and in exchange wrong bindings rise from zero to 134. For this lesson, the devolution’s price can be stated in one line: if the 76 information items the matching decision rests on (10 requests × 3, 10 volumes × 4, 6 nodes × 1) are not produced continuously and on time, the result degrades even though the decision mechanism itself has not changed. What the system does not know is who bound the volume and when; because it does not know, what it does is attempt to bind an occupied volume.

Summary

• Request and supply are separate declarations: the application declares three items, the cluster declares four per volume; the storage driver interface does the matching.
• Six of ten requests were met, four were not, and the reasons fall into separate classes: size 2, mode 1, class 1. Size is a capacity decision, mode is an interface decision, class is a vocabulary error.
• The selection rule does not change the number of requests met (6 in both), it changes how much space is overallocated: first-fit 85 Gi, best-fit 75 Gi.
• Access mode is a placement constraint: 26 of 36 options were eliminated. In single-writer mode an average of 1.00 eligible node per request remains, in the multi-node modes 3.00.
• As inventory age rises 0 → 1 → 3 rounds, correct bindings go 21 → 15 → 10, wrong bindings 0 → 86 → 134, average wait 0.33 → 1.93 → 3.20 rounds. Declaration load falls to 72 → 36 → 18 items per round.
• Delayed information turns “failure to decide” (80 → 33) into a “wrong decision”: cheap not-knowing is replaced by expensive wrong binding.

Next Step

Persistent data found its place in the cluster: a request is declared, supply is matched, and the pod opens on the node where the volume is reachable. Everything measured was between a single request and a single volume. Yet the verifier feeding the nightly batch job is not one replica; it runs three, and each replica has its own data. A replica set treats replicas as interchangeable and does not keep their names permanent either — so when a replica restarts, what will tell it which volume to bind to, and when the scale is turned down, whose data is left orphaned? The next lesson compares an object that gives out a stable identity against an unordered replica set in the same sequence of events, and counts: how many replicas returned to the same volume on restart, which replica left on scale-down, what round cost a stable startup pays, and how cluster membership and data ownership break when order is not followed.