Object Storage Service

The previous topic defended five cases from the write side, and all of them shared one thing in common: the data was small and meaningful on its own, per record. The design was driven not by volume but by the record’s correctness, order, and delivery guarantee.

This topic removes that assumption. Here a single record holds megabytes, and the storage line item overshadows the others. The first case is an object storage service: a client writes a binary body under a key, then reads it back by the same key. Two decisions need measuring — the extra bytes a multipart upload sends, and how many copies of the data are kept. This lesson’s durability is about the data not being lost; the Resilience and Reliability course’s resilience was about the service staying up — a store can stay up and still lose data.

Constraints

Functional requirement: writing, reading, deleting, and prefix-listing objects by key; sending large objects part by part, merging at the end, resuming an unfinished upload.

Non-functional requirement, in numbers: the extra bytes sent per object — resends plus per-part overhead — do not exceed 2 percent of the object size; objects lost per year stay under 500; stored bytes do not exceed 1.5 times the raw data; a small object read is served by a single request.

Scope reduction: access control and signature verification (the Backend Development curriculum’s Authentication and Authorization course), versioning, lifecycle, cross-region replication, and content search are not designed.

Assumptions

Code	Assumption	Value	Rationale
ND1	daily uploaded objects	1,200,000	the daily total across clients
ND2	small object share	0.92	small attachments dominate by count
ND3	small object average size	180 KB	documents and small images
ND4	large object average size	90 MB	recordings and backup files
ND5	reads per object over its lifetime	4	an object is read a handful of times
ND6	peak multiplier	3	peak hour’s ratio to the daily average
ND7	bytes sent between interruptions	250 MB	clients are on mobile networks
ND8	per-part overhead	8 KB	header, signature, part-ledger entry
ND9	annual node loss probability	0.01	from hardware and operations
ND10	retention period	365 days	the contract runs on an annual cycle
ND11	object metadata record	400 bytes	key, size, checksum, placement
ND12	slowest uploading client	1 Mbit/s	the lower bound for the valet key’s lifetime

Scale

// object-scale.mjs — the scale computation from the ND table; all of it is arithmetic
const ND = { objects: 1_200_000, smallShare: 0.92, smallBytes: 180e3, largeBytes: 90e6, reads: 4,
  peak: 3, interruptionBytes: 250e6, partOverhead: 8192, retentionDays: 365, metadataBytes: 400,
  partSize: 2e6, slowClientMbit: 1 };
const DAY = 86_400;
const avgBytes = ND.smallShare * ND.smallBytes + (1 - ND.smallShare) * ND.largeBytes;
const dailyBytes = ND.objects * avgBytes, objectsPerSec = ND.objects / DAY;
const partsPerSec = objectsPerSec * ((1 - ND.smallShare) * Math.ceil(ND.largeBytes / ND.partSize) + ND.smallShare);

const r = {
  "large objects' byte share": ((1 - ND.smallShare) * ND.largeBytes) / avgBytes,
  "daily uploaded TB": dailyBytes / 1e12,
  "raw stored PB": (dailyBytes * ND.retentionDays) / 1e15,
  "peak upload objects/s": objectsPerSec * ND.peak,
  "peak part requests/s": partsPerSec * ND.peak,
  "part-split request multiplier": partsPerSec / objectsPerSec,
  "peak upload Mbit/s": (dailyBytes / DAY) * ND.peak * 8 / 1e6,
  "peak read Mbit/s": (dailyBytes / DAY) * ND.reads * ND.peak * 8 / 1e6,
  "metadata stored GB": (ND.objects * ND.retentionDays * ND.metadataBytes) / 1e9,
  "object/metadata volume ratio": avgBytes / ND.metadataBytes,
};
for (const [name, d] of Object.entries(r))
  console.log(name.padEnd(30) + (Number.isInteger(d) ? String(d) : d.toFixed(2)).padStart(10));

const best = (k) => Math.sqrt(2 * ND.partOverhead * ND.interruptionBytes * k);   // lowest extra bytes
console.log(`\nbest part size = sqrt(2 * overhead * interruption interval) = ${(best(1) / 1e6).toFixed(2)} MB` +
  `; ND7 x2 -> ${(best(2) / 1e6).toFixed(2)} MB, ND7 x0.5 -> ${(best(0.5) / 1e6).toFixed(2)} MB`);
console.log(`at the slowest client a 2 MB part takes ${(ND.partSize * 8) / (ND.slowClientMbit * 1e6)} s -> ` +
  `valet key lifetime 60 s (x${(60 / ((ND.partSize * 8) / (ND.slowClientMbit * 1e6))).toFixed(2)} margin)`);

large objects' byte share           0.98
daily uploaded TB                   8.84
raw stored PB                       3.23
peak upload objects/s              41.67
peak part requests/s              188.33
part-split request multiplier       4.52
peak upload Mbit/s               2455.20
peak read Mbit/s                 9820.80
metadata stored GB                175.20
object/metadata volume ratio    18414.00

best part size = sqrt(2 * overhead * interruption interval) = 2.02 MB; ND7 x2 -> 2.86 MB, ND7 x0.5 -> 1.43 MB
at the slowest client a 2 MB part takes 16 s -> valet key lifetime 60 s (x3.75 margin)

These numbers are in the calculation class, and four of them drive the design. Eight percent of objects carry 98 percent of the bytes, so the two size classes are designed separately. Metadata volume is 1/18,414th of object volume: 175.20 GB against 3.23 PB. Splitting into parts multiplies the request count by 4.52: 41.67 object uploads become 188.33 part requests at peak. Peak read throughput is 9820.80 Mbit/s — the dominant line item is not requests, it is bandwidth and storage.

Part Size

When an interruption cuts a stream, the bytes sent up to that point are wasted. As parts shrink, wasted bytes drop, but the overhead (ND8) is paid on every part. The sum of the two curves is measured.

// part-size.mjs — in-process model: multipart upload of a 90 MB object. No network, no file;
// interruptions are generated at exponential intervals over bytes sent. The seed is visible, the generator is hand-written.
const OBJECT = 90e6, INTERRUPTION = 250e6, OVERHEAD = 8192, RUNS = 5000, SEED = 20260730, L = 1 / INTERRUPTION;
let state = SEED;                                    // linear congruential generator
const rand = () => { state = (state * 1103515245 + 12345) % 2147483648; return state / 2147483648; };
const interval = () => -Math.log(1 - rand()) / L;    // bytes sent before an interruption

function upload(size) {
  let sent = 0;
  for (let remaining = OBJECT; remaining > 0; remaining -= size) {
    const p = Math.min(size, remaining);
    for (;;) {
      sent += OVERHEAD;                              // every attempt is a request round
      const g = interval();
      if (g >= p) { sent += p; break; }
      sent += g;                                      // interrupted: this part is resent from the start
    }
  }
  return sent;
}

function expected(size) {                             // closed form over the same partitioning
  let t = 0;
  for (let remaining = OBJECT; remaining > 0; remaining -= size) {
    const p = Math.min(size, remaining);
    t += (Math.exp(L * p) - 1) / L + OVERHEAD * Math.exp(L * p);
  }
  return t;
}

console.log(`model: ${RUNS} uploads, object ${OBJECT / 1e6} MB, interruption interval ${INTERRUPTION / 1e6} MB, ` +
  `per-part overhead ${OVERHEAD} bytes, seed ${SEED}\n\n` + "part".padStart(14) + "part count".padStart(14) +
  "sent MB".padStart(15) + "measured ratio".padStart(16) + "expected ratio".padStart(16));
for (const b of [256e3, 1e6, 2e6, 4e6, 16e6, OBJECT]) {
  let t = 0;
  for (let i = 0; i < RUNS; i += 1) t += upload(b);
  const avg = t / RUNS;
  console.log((b === OBJECT ? "single stream" : `${(b / 1e6).toFixed(2)} MB`).padStart(14) +
    String(Math.ceil(OBJECT / b)).padStart(14) + (avg / 1e6).toFixed(2).padStart(15) +
    (avg / OBJECT).toFixed(4).padStart(16) + (expected(b) / OBJECT).toFixed(4).padStart(16));
}

model: 5000 uploads, object 90 MB, interruption interval 250 MB, per-part overhead 8192 bytes, seed 20260730

          part    part count        sent MB  measured ratio  expected ratio
       0.26 MB           352          92.94          1.0326          1.0326
       1.00 MB            90          90.91          1.0101          1.0102
       2.00 MB            45          90.68          1.0075          1.0081
       4.00 MB            23          90.90          1.0100          1.0101
      16.00 MB             6          93.11          1.0346          1.0319
 single stream             1         108.14          1.2015          1.2038

The measured ratio is a valid measurement for this run; the expected ratio is a run-independent calculation; the two agree on three of four digits. The 2 percent constraint cuts off both ends of the curve: at 256 KB the extra bytes run 3.26 percent, and the cause is not resending but the overhead of 352 parts; at 16 MB it is 3.46 percent, and the cause is resending. Only the range from 1 MB to 4 MB meets the constraint, and the best point sits at 2 MB, the same place the closed form gives. In a single stream, an average of 108.14 MB goes out for a 90 MB object; this ratio worsens as the object grows, because the expected bytes grow exponentially with object size.

Sensitivity. If ND7 doubles, the best part size shifts to 2.86 MB; if it halves, to 1.43 MB; a fourfold change in ND7 shifts the part size by a factor of two. The decision is weakly tied to ND7 — 2 MB is defensible across a wide range.

Design

The upload path relies on the Valet Key Pattern (the Resilience and Reliability course’s Distributed Correctness topic). The service does not carry bytes; it returns a time-limited key that authorizes the client to write a single part. The key’s scope is an upload ID paired with a part number, and its lifetime is 60 seconds — at the slowest client a 2 MB part takes 16 seconds, leaving a margin of 3.75x. This choice keeps 2455.20 Mbit/s of upload traffic outside the service nodes. A part write is idempotent (same topic, Idempotent Operations); the uniqueness key is the same binary data, so a client can send without being certain which part actually landed.

The metadata store is a key-value store (the Scaling the Data Layer course’s Data Distribution topic, Store Types) and is sharded by the hash of the object key (same topic, Sharding). Hashing is chosen over range partitioning: keys collected under a shared prefix would overheat a single shard under range partitioning. The Scheduler–Agent–Supervisor pattern collects the parts of unfinished uploads (the Application Layer and Service Interaction course’s Queues and Workflows topic); its window is 7 days.

Deliberately unused pattern: caching. Cache-aside from the Scaling the Data Layer course’s Cache Architecture topic has no place here: an object is read only 4 times over its lifetime (ND5), and the cost of a layer holding a fraction of the daily 8.84 TB is not worth four reads. The second is session stickiness (the Traffic Layer course’s Load Balancing topic): part writes are stateless.

Eliminated Alternative: Three Copies of Every Object

The simplest design keeps three copies of every object. The alternative is erasure coding: the object is split into k data parts, m parity parts are computed, and any k of the k+m parts return the object.

// durability.mjs — the storage, loss, and request cost of replica count versus erasure coding.
// Losses are assumed independent (ND9); this is a model assumption, not a measurement.
let P = 0.01;                                               // ND9
const N = 1_200_000 * 365, SMALL = 180e3, OVERHEAD = 8192;  // ND1xND10, ND3, ND8
const PEAK = (1_200_000 * 4 / 86_400) * 3 * 0.92;           // object-scale.mjs: small object reads/s
const binom = (n, i) => { let t = 1; for (let j = 0; j < i; j += 1) t = (t * (n - j)) / (j + 1); return t; };
const loss = (k, m) => {
  if (m === 0) return P ** k;                               // all k copies are lost
  let t = 0;                                                 // m+1 or more of the k+m parts are lost
  for (let i = m + 1; i <= k + m; i += 1) t += binom(k + m, i) * P ** i * (1 - P) ** (k + m - i);
  return t;
};

console.log(`ND9 = ${P}, stored objects ${N.toLocaleString("en-US")}\n\n` + "setup".padEnd(16) +
  "storage factor".padStart(16) + "lost objects/year".padStart(19) + "read requests".padStart(15) +
  "overhead at 180 KB".padStart(20));
for (const [name, k, m] of [["3 copies", 3, 0], ["erasure (6,3)", 6, 3], ["erasure (10,4)", 10, 4],
  ["erasure (12,4)", 12, 4]])
  console.log(name.padEnd(16) + (m === 0 ? k : (k + m) / k).toFixed(2).padStart(16) +
    (loss(k, m) * N).toFixed(2).padStart(19) + String(m === 0 ? 1 : k).padStart(15) +
    `${((((m === 0 ? k : k + m) * OVERHEAD) / SMALL) * 100).toFixed(1)}%`.padStart(20));

const RAW = 3.23, LARGE_SHARE = 0.9775;                      // object-scale.mjs
const mix = LARGE_SHARE * 1.4 + (1 - LARGE_SHARE) * 3;       // large erasure (10,4), small 3 copies
// GB-month: under linear growth the year average is half the year-end figure, x12 months -> x6
console.log(`\nyear-end storage: all erasure (10,4) ${(RAW * 1.4).toFixed(2)} PB, mixed ` +
  `(factor ${mix.toFixed(3)}) ${(RAW * mix).toFixed(2)} PB, all 3 copies ${(RAW * 3).toFixed(2)} PB`);
console.log(`the mix's excess is ${(RAW * (mix - 1.4) * 6e6).toFixed(0)} GB-month; for small objects, peak ` +
  `part requests/s is ${PEAK.toFixed(2)} for 3 copies, ${(PEAK * 10).toFixed(2)} for erasure (10,4)\n`);
for (const p of [0.005, 0.01, 0.02, 0.04]) {
  P = p;
  console.log(`ND9=${p} -> 3 copies ${loss(3, 0).toExponential(2)}, erasure (10,4) ` +
    `${loss(10, 4).toExponential(2)}, ratio ${(loss(3, 0) / loss(10, 4)).toFixed(2)}`);
}

ND9 = 0.01, stored objects 438,000,000

setup             storage factor  lost objects/year  read requests  overhead at 180 KB
3 copies                    3.00             438.00              1               13.7%
erasure (6,3)               1.50             530.17              6               41.0%
erasure (10,4)              1.40              81.33             10               63.7%
erasure (12,4)              1.33             174.51             12               72.8%

year-end storage: all erasure (10,4) 4.52 PB, mixed (factor 1.436) 4.64 PB, all 3 copies 9.69 PB
the mix's excess is 697680 GB-month; for small objects, peak part requests/s is 153.33 for 3 copies, 1533.33 for erasure (10,4)

ND9=0.005 -> 3 copies 1.25e-7, erasure (10,4) 6.03e-9, ratio 20.74
ND9=0.01 -> 3 copies 1.00e-6, erasure (10,4) 1.86e-7, ratio 5.39
ND9=0.02 -> 3 copies 8.00e-6, erasure (10,4) 5.51e-6, ratio 1.45
ND9=0.04 -> 3 copies 6.40e-5, erasure (10,4) 1.51e-4, ratio 0.42

Three copies lose on two numbers at once. The storage constraint is 1.50x while three copies’ factor is 3.00; erasure (10,4) stays under it at 1.40 and also wins on durability: 81.33 objects lost per year instead of 438. Erasure (6,3) breaks the “under 500” constraint with an annual loss of 530.17; erasure (12,4) is the cheapest on storage, but its loss is more than double (10,4)’s.

The alternative wins where the object is small. Erasure (10,4) splits a 180 KB object into fourteen 18 KB parts: overhead climbs to 63.7 percent of the object, and a read needs ten requests — 1533.33 part requests at peak instead of 153.33, which breaks the “served by a single request” constraint. The choice is therefore mixed: three copies for small objects, erasure (10,4) for large ones, at a factor of 1.436.

Which constraint change flips the alternative: if ND9 drops to five per thousand, coding’s advantage grows to 20.74x; at 2 percent it falls to 1.45x, at 4 percent to 0.42x — once node loss becomes frequent enough, three copies pulls ahead, because it requires losing all of the copies rather than three of fourteen parts. Independent loss is a model assumption; if losses are correlated, both columns are optimistic.

Failure Behavior and What Is Sacrificed

When a shard of the metadata store goes down, no new upload can start in that hash range and no merge can complete; but part writes already in flight continue, because they go straight to the store through the valet key. This is graceful degradation (the Resilience and Reliability course’s Fault Isolation topic): the upload does not stop, it only cannot finish, and the client can complete it within the 7-day window. On the loss of one storage node, an erasure (10,4) object is read from 13 of its 14 parts; because repair reads k = 10 parts, its traffic runs to the size of the object. Since the peak part-request rate is 188.33/s, the upload path gets throttling (same topic) set with a threshold above that rate.

What is sacrificed: for small objects, the storage factor was left at 3. If everything were coded, year-end storage would be 4.52 PB; the mixed choice is 4.64 PB — 697,680 GB-month of extra storage a year, the price of keeping small-object reads at 153.33 part requests instead of 1533.33.

Summary

• Eight percent of objects carry 98 percent of the bytes; the two size classes are designed separately, and multipart upload multiplies the request count by 4.52 — 188.33 part requests at peak.
• Part size is chosen from the bottom of a trough: the extra-byte ratio runs 1.0326 at 256 KB, 1.0075 at 2 MB, 1.0346 at 16 MB, 1.2015 in a single stream, and the best point sits exactly where sqrt(2 × overhead × interruption interval) puts it.
• Three copies lose on two numbers at once: a storage factor of 3.00 against 1.40, an annual loss of 438 against 81.33 objects; but for small objects, coding pushes reads from 153.33 to 1533.33 requests, so three copies wins there.
• The choice is tied to ND9: if the loss probability rises to 4 percent, the ratio becomes 0.42 and three copies pulls ahead.

Next Step

In this case the object was a meaningless byte sequence: stored as written, served as read, its content never inspected. The next case removes that assumption: the uploaded file gets multiple representations, and which one is served is decided by the requester’s conditions at that moment. The question becomes: in how many forms is the same content kept, how is the processor cost of producing them weighed against the bytes they store, and when does an extra form become waste.