File Storage

The previous lesson ended at a door: a block volume is bound to a single instance, and only one of twelve collector instances can write at once. In the fictional measurement network this is not a detail but a direct obstacle — field teams drop work-order attachments into the same files, invoice generation reads from the same directory, and all of it runs on separate instances. Object storage accepts multiple writers but offers no lock or partial update; a block volume offers both but allows only a single writer.

A third type fills this gap: the shared file system. It connects over the network, multiple instances mount it at once, and it offers a full file system interface — directory, rename, partial write, and byte-range lock. What is measured is how much of this is real gain: the same data can be kept on all three types, and the difference among them only shows up under multi-writer access.

• ST25 — Fictional workload: twelve field instances, 340 appends a second, each append 2 KiB, 4,800 active work-order files. Which file a write lands on is set by the generator in the model, seed 71549.
• ST26 — Latencies are a model: an object storage request 25 ms, a block volume 0.6 ms, a shared file system 2.5 ms; the network round trip from a remote instance to the single writer 0.4 ms.
• ST27 — On object storage the read-modify-write window is 50 ms; a second write landing on the same file within that window overwrites the first.

// file/model.mjs — three storage types, a multi-writer workload, and fees are a MODEL. Fees are a
// fictional unit; types are called by access form, not product name.
export const INSTANCES = 12, WRITES_PER_SEC = 340, APPEND_KIB = 2, FILE_KIB = 11, ACTIVE = 4800;
export const MONTH_SEC = 2_592_000;                // a 30-day month, in seconds
export const TYPE = {
  "object storage": { writers: INSTANCES, lock: "none", latencyMs: 25, storage: 0.023, request: 0.0050 / 1000, throughput: 0 },
  "block volume": { writers: 1, lock: "single writer", latencyMs: 0.6, storage: 0.080, request: 0, throughput: 0 },
  "shared file": { writers: INSTANCES, lock: "byte range", latencyMs: 2.5, storage: 0.300, request: 0, throughput: 0.60 },
};
export const NETWORK_RTT_MS = 0.4;                 // the round trip from a remote instance to the single writer
export const COLLISION_MS = 50;                    // object storage's read-modify-write window
export const gib = (bytes) => bytes / 1024 ** 3;

// Which file a write lands on is a MODEL: the generator is here, the seed is visible. The distribution is skewed.
export const SEED = 71549;
export function* writes(count) {
  let x = SEED >>> 0;
  const u = () => (x = (Math.imul(x, 1664525) + 1013904223) >>> 0) / 4294967296;
  for (let i = 0; i < count; i++)
    yield { t: (i / WRITES_PER_SEC) * 1000, file: Math.floor(ACTIVE * Math.pow(u(), 3)) };
}
// Single-server queue: mean wait with a fixed service time (utilization p, service S).
export const wait = (p, S) => (p >= 1 ? Infinity : (p / (2 * (1 - p))) * S);

Same Data, Three Types, Multiple Writers

// file/multi-writer.mjs — the same data on three types: how many writers, how the lock works, latency and fee
import { INSTANCES, WRITES_PER_SEC, APPEND_KIB, FILE_KIB, ACTIVE, MONTH_SEC, TYPE, NETWORK_RTT_MS, COLLISION_MS, gib, SEED, writes, wait }
  from "./model.mjs";

const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const s = (x, n) => String(x).padStart(n);
const DURATION_SEC = 60, COUNT = WRITES_PER_SEC * DURATION_SEC;

// 1) Object storage: no lock. A second write landing on the same file within COLLISION_MS overwrites the first.
const lastWrite = new Map();
let lost = 0, hottest = 0;
const counts = new Map();
for (const w of writes(COUNT)) {
  const previous = lastWrite.get(w.file);
  if (previous !== undefined && w.t - previous < COLLISION_MS) lost++;
  lastWrite.set(w.file, w.t);
  counts.set(w.file, (counts.get(w.file) ?? 0) + 1);
}
for (const n of counts.values()) hottest = Math.max(hottest, n);
const hottestRate = hottest / DURATION_SEC;

// 2) Block volume: single writer. The others send over the network to that instance, forming a queue there.
const S_BLOCK = TYPE["block volume"].latencyMs, pBlock = WRITES_PER_SEC * S_BLOCK / 1000;
const blockLatency = NETWORK_RTT_MS + S_BLOCK + wait(pBlock, S_BLOCK);
// 3) Shared file system: a byte-range lock per file; the queue forms only within the same file.
const S_SHARED = TYPE["shared file"].latencyMs, pHottest = hottestRate * S_SHARED / 1000;
const sharedLatency = S_SHARED + wait(pHottest, S_SHARED);

const dataGib = gib(ACTIVE * FILE_KIB * 1024);
const monthlyWrites = WRITES_PER_SEC * MONTH_SEC;
const FEE = {
  "object storage": TYPE["object storage"].storage * dataGib + monthlyWrites * TYPE["object storage"].request,
  "block volume": TYPE["block volume"].storage * Math.ceil(WRITES_PER_SEC / 3 / 4) * 4,   // iops budget demands size
  "shared file": TYPE["shared file"].storage * dataGib +
    TYPE["shared file"].throughput * (WRITES_PER_SEC * APPEND_KIB / 1024),
};
console.log(`${INSTANCES} field instances, ${WRITES_PER_SEC} appends/s (${APPEND_KIB} KiB), ${ACTIVE} active work-order files ` +
  `(${FILE_KIB} KiB). Distribution skewed, generator in the model, seed ${SEED}. ${DURATION_SEC} s run.`);
console.log(`the hottest file took ${hottest} writes in ${DURATION_SEC} s: ${g(hottestRate, 0, 1)} writes/s.`);
console.log("\ntype             | writers at once | lock           | latency (ms) | lost update  | monthly fee");
console.log("-----------------|-----------------|----------------|--------------|--------------|------------");
const LATENCY = { "object storage": TYPE["object storage"].latencyMs, "block volume": blockLatency, "shared file": sharedLatency };
const LOST = { "object storage": lost, "block volume": 0, "shared file": 0 };
for (const [name, t] of Object.entries(TYPE))
  console.log(`${name.padEnd(16)} | ${s(t.writers, 15)} | ${t.lock.padEnd(14)} | ${g(LATENCY[name], 12, 2)} | ` +
    `${s(LOST[name], 12)} | ${g(FEE[name], 11)}`);
console.log(`on object storage, ${lost} of the ${COUNT} writes (${g(100 * lost / COUNT, 0, 2)}%) were lost silently: ` +
  `no lock, last writer wins.`);
console.log(`on the block volume, 11 instances cannot write directly; the single writer's utilization is ${g(100 * pBlock, 0, 1)}%, ` +
  `wait ${g(wait(pBlock, S_BLOCK), 0, 3)} ms. If that instance drops, ${INSTANCES}/${INSTANCES} writers stop.`);
console.log(`on the shared file system, the hottest file's utilization is ${g(100 * pHottest, 0, 2)}%, ` +
  `lock wait ${g(wait(pHottest, S_SHARED), 0, 4)} ms; the lock is per file, not per bucket.`);

// The case it gains nothing: single writer, large sequential read. Same data, same job, fee on three types.
const INVOICE = { dataGib: 400, mibS: 180 };
const COST = {
  "object storage": TYPE["object storage"].storage * INVOICE.dataGib,
  "block volume": 0.045 * 452,                        // throughput-optimized volume, computed in the previous lesson
  "shared file": TYPE["shared file"].storage * INVOICE.dataGib + TYPE["shared file"].throughput * INVOICE.mibS,
};
console.log(`\nthe case it gains nothing — invoice generation: single writer, ${INVOICE.mibS} MiB/s sequential read, ${INVOICE.dataGib} GiB:`);
for (const [name, u] of Object.entries(COST))
  console.log(`  ${name.padEnd(16)} ${g(u, 8)} units/month${name === "shared file" ?
    ` — ${g(u / Math.min(...Object.values(COST)), 0, 1)}x the cheapest, the only thing it gains is an unused lock` : ""}`);

12 field instances, 340 appends/s (2 KiB), 4800 active work-order files (11 KiB). Distribution skewed, generator in the model, seed 71549. 60 s run.
the hottest file took 1168 writes in 60 s: 19.5 writes/s.

type             | writers at once | lock           | latency (ms) | lost update  | monthly fee
-----------------|-----------------|----------------|--------------|--------------|------------
object storage   |              12 | none           |        25.00 |         1057 |     4406.40
block volume     |               1 | single writer  |         1.08 |            0 |        9.28
shared file      |              12 | byte range     |         2.56 |            0 |        0.41
on object storage, 1057 of the 20400 writes (5.18%) were lost silently: no lock, last writer wins.
on the block volume, 11 instances cannot write directly; the single writer's utilization is 20.4%, wait 0.077 ms. If that instance drops, 12/12 writers stop.
on the shared file system, the hottest file's utilization is 4.87%, lock wait 0.0639 ms; the lock is per file, not per bucket.

the case it gains nothing — invoice generation: single writer, 180 MiB/s sequential read, 400 GiB:
  object storage       9.20 units/month
  block volume        20.34 units/month
  shared file        228.00 units/month — 24.8x the cheapest, the only thing it gains is an unused lock

The table separates the three types on a single screen. Object storage accepts twelve writers but has no lock: 1,057 of 20,400 writes in sixty seconds, 5.18%, are lost silently. No error is returned, no trace is left in a log, the field team’s note is nowhere to be found. Its monthly fee is also the highest — 4,406.40 units, almost all from write requests: small, frequent writes are object storage’s most expensive use.

A block volume gives lock and partial write, its latency is the lowest (1.08 ms), its fee 9.28 units — but its writer count is one. Eleven instances cannot write directly, they send their writes to the single instance. That instance’s utilization is 20.4%, comfortable today; the real item is not in performance but in structure: when that instance goes down, all twelve writers stop. The single-writer funnel is not a performance constraint, it is an availability item.

The shared file system gives both at once: twelve writers, a byte-range lock, zero lost updates. The lock is per file, not per bucket — even the hottest file only takes 19.5 writes a second, its utilization is 4.87%, and the mean lock wait is 0.064 ms. Its latency is more than double a block volume’s (2.56 ms against 1.08 ms), because every operation goes through the network.

The last block writes the type’s limit by name. Invoice generation is single-writer and does a large sequential read; it needs no lock, no directory tree. On the shared file system it costs 228.00 units a month — 24.8 times the cheapest option. A shared file system is bought for multi-writer access; on a single-writer large sequential read, the only thing it gains is an unused lock.

The Access Protocol Is an Unrecoverable Item

• ST28 — The fictional application’s file access points and how many sites each appears at are written into the model.
• ST29 — A shared file system is bound to a region; copying data elsewhere incurs an egress fee.
• ST30 — Fees are a fictional unit; a shared file system bills used size and throughput, a block volume bills allocated size.

// file/protocol.mjs — the access protocol is an unrecoverable item: call sites are counted (MODEL)
import { ACTIVE, FILE_KIB, gib } from "./model.mjs";

// The fictional application's file access points: call, how many sites it appears at, object-model equivalent.
const POINT = [
  ["read the whole file", 9, "yes", "full object get"],
  ["read a range of the file", 6, "yes", "range request"],
  ["write the whole file", 5, "yes", "full object put"],
  ["write into the middle of the file", 7, "no", "no partial update, the object is rewritten from scratch"],
  ["append to the file", 8, "no", "no append, the object is rewritten from scratch"],
  ["truncate the file", 2, "no", "no equivalent"],
  ["list the directory", 4, "partial", "prefix listing, paginated and billed"],
  ["read the directory's size", 2, "no", "every key is scanned"],
  ["rename", 3, "no", "copy and delete"],
  ["acquire a byte-range lock", 5, "no", "no lock"],
  ["force the change to disk", 3, "no", "a write is already all-or-nothing"],
];
const g = (x, n, d = 2) => x.toFixed(d).padStart(n);
const s = (x, n) => String(x).padStart(n);
const total = POINT.reduce((a, n) => a + n[1], 0);
const direct = POINT.filter((n) => n[2] === "yes").reduce((a, n) => a + n[1], 0);
const partial = POINT.filter((n) => n[2] === "partial").reduce((a, n) => a + n[1], 0);

console.log("call                               | sites | object model | equivalent");
console.log("-----------------------------------|-------|--------------|--------------------------------------");
for (const [name, sites, status, equivalent] of POINT)
  console.log(`${name.padEnd(34)} | ${s(sites, 5)} | ${status.padEnd(12)} | ${equivalent}`);
console.log(`\n${total} call sites in total: ${direct} carry over directly, ${partial} carry over by changing ` +
  `behavior, ${total - direct - partial} have to be rewritten.`);
console.log(`the share that must be rewritten is ${g(100 * (total - direct - partial) / total, 0, 1)}%; this number ` +
  `is independent of data size — even at ${gib(ACTIVE * FILE_KIB * 1024).toFixed(3)} GiB the same ${total - direct - partial} points are rewritten.`);

// Egress and transfer: the shared file system is bound to a region.
const EGRESS = { "in-region": 0, "cross-region": 0.020, "to the internet": 0.085 };
const dataGib = gib(ACTIVE * FILE_KIB * 1024), ARCHIVE_GIB = 640;
console.log(`\negress: if a ${ARCHIVE_GIB} GiB field archive is read off the shared file system and copied elsewhere`);
for (const [where, rate] of Object.entries(EGRESS))
  console.log(`  ${where.padEnd(16)} ${g(ARCHIVE_GIB * rate, 7)} units`);
console.log(`the active work-order data is ${dataGib.toFixed(3)} GiB; the egress fee is measured not by the data's ` +
  `size but by where it moves to.`);

call                               | sites | object model | equivalent
-----------------------------------|-------|--------------|--------------------------------------
read the whole file                |     9 | yes          | full object get
read a range of the file           |     6 | yes          | range request
write the whole file               |     5 | yes          | full object put
write into the middle of the file  |     7 | no           | no partial update, the object is rewritten from scratch
append to the file                 |     8 | no           | no append, the object is rewritten from scratch
truncate the file                  |     2 | no           | no equivalent
list the directory                 |     4 | partial      | prefix listing, paginated and billed
read the directory's size          |     2 | no           | every key is scanned
rename                             |     3 | no           | copy and delete
acquire a byte-range lock          |     5 | no           | no lock
force the change to disk           |     3 | no           | a write is already all-or-nothing

54 call sites in total: 20 carry over directly, 4 carry over by changing behavior, 30 have to be rewritten.
the share that must be rewritten is 55.6%; this number is independent of data size — even at 0.050 GiB the same 30 points are rewritten.

egress: if a 640 GiB field archive is read off the shared file system and copied elsewhere
  in-region           0.00 units
  cross-region       12.80 units
  to the internet    54.40 units
the active work-order data is 0.050 GiB; the egress fee is measured not by the data's size but by where it moves to.

When a shared file system is chosen, the delegated-decision list is short: the provider takes over the metadata server, replication, and growth. What stays holds one more thing, and it does not show up in the billing column — the access protocol. The application is written against file paths, a directory tree, and locks. This is not a configuration, it is a shape of code.

The table gives its measure. Of fifty-four access points, twenty carry over directly to the object model, four carry over by changing behavior, thirty must be rewritten: 55.6%. This number is independent of data size — even at 0.050 GiB of work-order data the same thirty points are rewritten. Key layout was an unrecoverable item because the data had to be moved; the access protocol is an unrecoverable item because the application has to be moved, and that move’s cost does not shrink with the data.

The egress side closes out the third item. A shared file system is bound to a single region; reading a 640 GiB field archive within the region incurs no egress fee, moving it cross-region costs 12.80 units, downloading it to the internet costs 54.40 units. The same rule has held since this course’s first lesson: data is free coming in, billed going out, and its fee is measured by where it goes.

Summary

• The three types diverge under multi-writer access: object storage takes twelve writers but has no lock, a block volume gives the lock but takes only one writer, a shared file system gives both at once.
• Lockless access costs silently: 1,057 of 20,400 writes in sixty seconds, 5.18%, are lost with no error returned. Under the same workload, object storage is the most expensive option at 4,406.40 units.
• The single-writer funnel is not a performance constraint but an availability item: utilization sits comfortably at 20.4%, but when that instance goes down, all twelve writers stop.
• The lock is per file: the hottest file takes 19.5 writes a second, its utilization is 4.87%, lock wait is 0.064 ms. Latency is more than double a block volume’s (2.56 ms against 1.08 ms), because every operation goes through the network.
• The case where it gains nothing is named: single-writer large sequential reads cost 228.00 units, 24.8 times the cheapest option. The unrecoverable item is the access protocol — thirty of fifty-four call sites are rewritten regardless of data size.

Next Step

Five lessons leave one gap. The fictional measurement network’s field photos are written once and almost never read; in the distribution measured in the second lesson, 97% of this family was never opened all year. For such data the frequent class is an obvious waste, but what the archive tier is has not yet been measured: its storage fee is nearly nothing, and in exchange the data cannot be read right away. The next lesson counts this trade-off — retrieval speed options are scanned, time and fee are worked out together for each, the hours an audit request waits are computed, and what the minimum retention period makes an early-deleted object pay is shown.