--- title: 'Object Storage Service' source: 'https://academia.sh/en/courses/case-studies/object-storage-service' course: 'Case Studies' language: en updated: '2026-08-23T07:01:24+00:00' license: 'CC BY-SA 4.0' --- # Object Storage Service The first case where volume drives the design: choosing part size against the probability of interruption, measuring the extra bytes sent in an in-process model and verifying it with a closed form, comparing replica count against erasure coding on storage factor and annual loss, and eliminating coding for small objects on request count. The previous topic defended five cases from the write side, and all of them shared one thing in common: the data was **small** and meaningful on its own, per record. The design was driven not by volume but by the record's correctness, order, and delivery guarantee. This topic removes that assumption. Here a single record holds megabytes, and the storage line item overshadows the others. The first case is an **object storage** service: a client writes a binary body under a key, then reads it back by the same key. Two decisions need measuring — the extra bytes a **multipart upload** sends, and how many copies of the data are kept. This lesson's **durability** is about the data not being lost; the Resilience and Reliability course's **resilience** was about the service staying up — a store can stay up and still lose data. ## Constraints Functional requirement: writing, reading, deleting, and prefix-listing objects by key; sending large objects part by part, merging at the end, resuming an unfinished upload. Non-functional requirement, in numbers: the extra bytes sent per object — resends plus per-part overhead — do not exceed **2** percent of the object size; objects lost per year stay under **500**; stored bytes do not exceed **1.5** times the raw data; a small object read is served by a **single** request. Scope reduction: access control and signature verification (the Backend Development curriculum's Authentication and Authorization course), versioning, lifecycle, cross-region replication, and content search are not designed. ## Assumptions | Code | Assumption | Value | Rationale | |---|---|---|---| | ND1 | daily uploaded objects | 1,200,000 | the daily total across clients | | ND2 | small object share | 0.92 | small attachments dominate by count | | ND3 | small object average size | 180 KB | documents and small images | | ND4 | large object average size | 90 MB | recordings and backup files | | ND5 | reads per object over its lifetime | 4 | an object is read a handful of times | | ND6 | peak multiplier | 3 | peak hour's ratio to the daily average | | ND7 | bytes sent between interruptions | 250 MB | clients are on mobile networks | | ND8 | per-part overhead | 8 KB | header, signature, part-ledger entry | | ND9 | annual node loss probability | 0.01 | from hardware and operations | | ND10 | retention period | 365 days | the contract runs on an annual cycle | | ND11 | object metadata record | 400 bytes | key, size, checksum, placement | | ND12 | slowest uploading client | 1 Mbit/s | the lower bound for the valet key's lifetime | ## Scale ```js // object-scale.mjs — the scale computation from the ND table; all of it is arithmetic const ND = { objects: 1_200_000, smallShare: 0.92, smallBytes: 180e3, largeBytes: 90e6, reads: 4, peak: 3, interruptionBytes: 250e6, partOverhead: 8192, retentionDays: 365, metadataBytes: 400, partSize: 2e6, slowClientMbit: 1 }; const DAY = 86_400; const avgBytes = ND.smallShare * ND.smallBytes + (1 - ND.smallShare) * ND.largeBytes; const dailyBytes = ND.objects * avgBytes, objectsPerSec = ND.objects / DAY; const partsPerSec = objectsPerSec * ((1 - ND.smallShare) * Math.ceil(ND.largeBytes / ND.partSize) + ND.smallShare); const r = { "large objects' byte share": ((1 - ND.smallShare) * ND.largeBytes) / avgBytes, "daily uploaded TB": dailyBytes / 1e12, "raw stored PB": (dailyBytes * ND.retentionDays) / 1e15, "peak upload objects/s": objectsPerSec * ND.peak, "peak part requests/s": partsPerSec * ND.peak, "part-split request multiplier": partsPerSec / objectsPerSec, "peak upload Mbit/s": (dailyBytes / DAY) * ND.peak * 8 / 1e6, "peak read Mbit/s": (dailyBytes / DAY) * ND.reads * ND.peak * 8 / 1e6, "metadata stored GB": (ND.objects * ND.retentionDays * ND.metadataBytes) / 1e9, "object/metadata volume ratio": avgBytes / ND.metadataBytes, }; for (const [name, d] of Object.entries(r)) console.log(name.padEnd(30) + (Number.isInteger(d) ? String(d) : d.toFixed(2)).padStart(10)); const best = (k) => Math.sqrt(2 * ND.partOverhead * ND.interruptionBytes * k); // lowest extra bytes console.log(`\nbest part size = sqrt(2 * overhead * interruption interval) = ${(best(1) / 1e6).toFixed(2)} MB` + `; ND7 x2 -> ${(best(2) / 1e6).toFixed(2)} MB, ND7 x0.5 -> ${(best(0.5) / 1e6).toFixed(2)} MB`); console.log(`at the slowest client a 2 MB part takes ${(ND.partSize * 8) / (ND.slowClientMbit * 1e6)} s -> ` + `valet key lifetime 60 s (x${(60 / ((ND.partSize * 8) / (ND.slowClientMbit * 1e6))).toFixed(2)} margin)`); ``` ``` large objects' byte share 0.98 daily uploaded TB 8.84 raw stored PB 3.23 peak upload objects/s 41.67 peak part requests/s 188.33 part-split request multiplier 4.52 peak upload Mbit/s 2455.20 peak read Mbit/s 9820.80 metadata stored GB 175.20 object/metadata volume ratio 18414.00 best part size = sqrt(2 * overhead * interruption interval) = 2.02 MB; ND7 x2 -> 2.86 MB, ND7 x0.5 -> 1.43 MB at the slowest client a 2 MB part takes 16 s -> valet key lifetime 60 s (x3.75 margin) ``` These numbers are in the **calculation** class, and four of them drive the design. Eight percent of objects carry 98 percent of the bytes, so the two size classes are designed separately. Metadata volume is 1/18,414th of object volume: 175.20 GB against 3.23 PB. Splitting into parts multiplies the request count by 4.52: 41.67 object uploads become 188.33 part requests at peak. Peak read throughput is 9820.80 Mbit/s — the dominant line item is not requests, it is bandwidth and storage. ## Part Size When an interruption cuts a stream, the bytes sent up to that point are wasted. As parts shrink, wasted bytes drop, but the overhead (ND8) is paid on every part. The sum of the two curves is measured. ```js // part-size.mjs — in-process model: multipart upload of a 90 MB object. No network, no file; // interruptions are generated at exponential intervals over bytes sent. The seed is visible, the generator is hand-written. const OBJECT = 90e6, INTERRUPTION = 250e6, OVERHEAD = 8192, RUNS = 5000, SEED = 20260730, L = 1 / INTERRUPTION; let state = SEED; // linear congruential generator const rand = () => { state = (state * 1103515245 + 12345) % 2147483648; return state / 2147483648; }; const interval = () => -Math.log(1 - rand()) / L; // bytes sent before an interruption function upload(size) { let sent = 0; for (let remaining = OBJECT; remaining > 0; remaining -= size) { const p = Math.min(size, remaining); for (;;) { sent += OVERHEAD; // every attempt is a request round const g = interval(); if (g >= p) { sent += p; break; } sent += g; // interrupted: this part is resent from the start } } return sent; } function expected(size) { // closed form over the same partitioning let t = 0; for (let remaining = OBJECT; remaining > 0; remaining -= size) { const p = Math.min(size, remaining); t += (Math.exp(L * p) - 1) / L + OVERHEAD * Math.exp(L * p); } return t; } console.log(`model: ${RUNS} uploads, object ${OBJECT / 1e6} MB, interruption interval ${INTERRUPTION / 1e6} MB, ` + `per-part overhead ${OVERHEAD} bytes, seed ${SEED}\n\n` + "part".padStart(14) + "part count".padStart(14) + "sent MB".padStart(15) + "measured ratio".padStart(16) + "expected ratio".padStart(16)); for (const b of [256e3, 1e6, 2e6, 4e6, 16e6, OBJECT]) { let t = 0; for (let i = 0; i < RUNS; i += 1) t += upload(b); const avg = t / RUNS; console.log((b === OBJECT ? "single stream" : `${(b / 1e6).toFixed(2)} MB`).padStart(14) + String(Math.ceil(OBJECT / b)).padStart(14) + (avg / 1e6).toFixed(2).padStart(15) + (avg / OBJECT).toFixed(4).padStart(16) + (expected(b) / OBJECT).toFixed(4).padStart(16)); } ``` ``` model: 5000 uploads, object 90 MB, interruption interval 250 MB, per-part overhead 8192 bytes, seed 20260730 part part count sent MB measured ratio expected ratio 0.26 MB 352 92.94 1.0326 1.0326 1.00 MB 90 90.91 1.0101 1.0102 2.00 MB 45 90.68 1.0075 1.0081 4.00 MB 23 90.90 1.0100 1.0101 16.00 MB 6 93.11 1.0346 1.0319 single stream 1 108.14 1.2015 1.2038 ``` The measured ratio is a valid **measurement** for this run; the expected ratio is a run-independent **calculation**; the two agree on three of four digits. The 2 percent constraint cuts off both ends of the curve: at 256 KB the extra bytes run 3.26 percent, and the cause is not resending but the overhead of 352 parts; at 16 MB it is 3.46 percent, and the cause is resending. Only the range from 1 MB to 4 MB meets the constraint, and the best point sits at 2 MB, the same place the closed form gives. In a single stream, an average of 108.14 MB goes out for a 90 MB object; this ratio worsens as the object grows, because the expected bytes grow exponentially with object size. **Sensitivity.** If ND7 doubles, the best part size shifts to 2.86 MB; if it halves, to 1.43 MB; a fourfold change in ND7 shifts the part size by a factor of two. The decision is weakly tied to ND7 — 2 MB is defensible across a wide range. ## Design The upload path relies on the **Valet Key Pattern** (the Resilience and Reliability course's Distributed Correctness topic). The service does not carry bytes; it returns a time-limited key that authorizes the client to write a single part. The key's scope is an upload ID paired with a part number, and its lifetime is 60 seconds — at the slowest client a 2 MB part takes 16 seconds, leaving a margin of 3.75x. This choice keeps 2455.20 Mbit/s of upload traffic outside the service nodes. A part write is **idempotent** (same topic, Idempotent Operations); the uniqueness key is the same binary data, so a client can send without being certain which part actually landed. The metadata store is a **key-value store** (the Scaling the Data Layer course's Data Distribution topic, Store Types) and is **sharded** by the hash of the object key (same topic, Sharding). Hashing is chosen over range partitioning: keys collected under a shared prefix would overheat a single shard under range partitioning. The **Scheduler–Agent–Supervisor** pattern collects the parts of unfinished uploads (the Application Layer and Service Interaction course's Queues and Workflows topic); its window is 7 days. **Deliberately unused pattern: caching.** Cache-aside from the Scaling the Data Layer course's Cache Architecture topic has no place here: an object is read only 4 times over its lifetime (ND5), and the cost of a layer holding a fraction of the daily 8.84 TB is not worth four reads. The second is **session stickiness** (the Traffic Layer course's Load Balancing topic): part writes are stateless. ## Eliminated Alternative: Three Copies of Every Object The simplest design keeps three copies of every object. The alternative is **erasure coding**: the object is split into k data parts, m parity parts are computed, and any k of the k+m parts return the object. ```js // durability.mjs — the storage, loss, and request cost of replica count versus erasure coding. // Losses are assumed independent (ND9); this is a model assumption, not a measurement. let P = 0.01; // ND9 const N = 1_200_000 * 365, SMALL = 180e3, OVERHEAD = 8192; // ND1xND10, ND3, ND8 const PEAK = (1_200_000 * 4 / 86_400) * 3 * 0.92; // object-scale.mjs: small object reads/s const binom = (n, i) => { let t = 1; for (let j = 0; j < i; j += 1) t = (t * (n - j)) / (j + 1); return t; }; const loss = (k, m) => { if (m === 0) return P ** k; // all k copies are lost let t = 0; // m+1 or more of the k+m parts are lost for (let i = m + 1; i <= k + m; i += 1) t += binom(k + m, i) * P ** i * (1 - P) ** (k + m - i); return t; }; console.log(`ND9 = ${P}, stored objects ${N.toLocaleString("en-US")}\n\n` + "setup".padEnd(16) + "storage factor".padStart(16) + "lost objects/year".padStart(19) + "read requests".padStart(15) + "overhead at 180 KB".padStart(20)); for (const [name, k, m] of [["3 copies", 3, 0], ["erasure (6,3)", 6, 3], ["erasure (10,4)", 10, 4], ["erasure (12,4)", 12, 4]]) console.log(name.padEnd(16) + (m === 0 ? k : (k + m) / k).toFixed(2).padStart(16) + (loss(k, m) * N).toFixed(2).padStart(19) + String(m === 0 ? 1 : k).padStart(15) + `${((((m === 0 ? k : k + m) * OVERHEAD) / SMALL) * 100).toFixed(1)}%`.padStart(20)); const RAW = 3.23, LARGE_SHARE = 0.9775; // object-scale.mjs const mix = LARGE_SHARE * 1.4 + (1 - LARGE_SHARE) * 3; // large erasure (10,4), small 3 copies // GB-month: under linear growth the year average is half the year-end figure, x12 months -> x6 console.log(`\nyear-end storage: all erasure (10,4) ${(RAW * 1.4).toFixed(2)} PB, mixed ` + `(factor ${mix.toFixed(3)}) ${(RAW * mix).toFixed(2)} PB, all 3 copies ${(RAW * 3).toFixed(2)} PB`); console.log(`the mix's excess is ${(RAW * (mix - 1.4) * 6e6).toFixed(0)} GB-month; for small objects, peak ` + `part requests/s is ${PEAK.toFixed(2)} for 3 copies, ${(PEAK * 10).toFixed(2)} for erasure (10,4)\n`); for (const p of [0.005, 0.01, 0.02, 0.04]) { P = p; console.log(`ND9=${p} -> 3 copies ${loss(3, 0).toExponential(2)}, erasure (10,4) ` + `${loss(10, 4).toExponential(2)}, ratio ${(loss(3, 0) / loss(10, 4)).toFixed(2)}`); } ``` ``` ND9 = 0.01, stored objects 438,000,000 setup storage factor lost objects/year read requests overhead at 180 KB 3 copies 3.00 438.00 1 13.7% erasure (6,3) 1.50 530.17 6 41.0% erasure (10,4) 1.40 81.33 10 63.7% erasure (12,4) 1.33 174.51 12 72.8% year-end storage: all erasure (10,4) 4.52 PB, mixed (factor 1.436) 4.64 PB, all 3 copies 9.69 PB the mix's excess is 697680 GB-month; for small objects, peak part requests/s is 153.33 for 3 copies, 1533.33 for erasure (10,4) ND9=0.005 -> 3 copies 1.25e-7, erasure (10,4) 6.03e-9, ratio 20.74 ND9=0.01 -> 3 copies 1.00e-6, erasure (10,4) 1.86e-7, ratio 5.39 ND9=0.02 -> 3 copies 8.00e-6, erasure (10,4) 5.51e-6, ratio 1.45 ND9=0.04 -> 3 copies 6.40e-5, erasure (10,4) 1.51e-4, ratio 0.42 ``` Three copies lose on two numbers at once. The storage constraint is 1.50x while three copies' factor is 3.00; erasure (10,4) stays under it at 1.40 and also wins on durability: 81.33 objects lost per year instead of 438. Erasure (6,3) breaks the "under 500" constraint with an annual loss of 530.17; erasure (12,4) is the cheapest on storage, but its loss is more than double (10,4)'s. The alternative wins where the object is small. Erasure (10,4) splits a 180 KB object into fourteen 18 KB parts: overhead climbs to 63.7 percent of the object, and a read needs ten requests — 1533.33 part requests at peak instead of 153.33, which breaks the "served by a single request" constraint. The choice is therefore mixed: three copies for small objects, erasure (10,4) for large ones, at a factor of 1.436. **Which constraint change flips the alternative:** if ND9 drops to five per thousand, coding's advantage grows to 20.74x; at 2 percent it falls to 1.45x, at 4 percent to 0.42x — once node loss becomes frequent enough, three copies pulls ahead, because it requires losing all of the copies rather than three of fourteen parts. Independent loss is a model assumption; if losses are correlated, both columns are optimistic. ## Failure Behavior and What Is Sacrificed When a shard of the metadata store goes down, no new upload can start in that hash range and no merge can complete; but part writes already in flight continue, because they go straight to the store through the valet key. This is **graceful degradation** (the Resilience and Reliability course's Fault Isolation topic): the upload does not stop, it only cannot finish, and the client can complete it within the 7-day window. On the loss of one storage node, an erasure (10,4) object is read from 13 of its 14 parts; because repair reads k = 10 parts, its traffic runs to the size of the object. Since the peak part-request rate is 188.33/s, the upload path gets **throttling** (same topic) set with a threshold above that rate. **What is sacrificed:** for small objects, the storage factor was left at 3. If everything were coded, year-end storage would be 4.52 PB; the mixed choice is 4.64 PB — 697,680 GB-month of extra storage a year, the price of keeping small-object reads at 153.33 part requests instead of 1533.33. ## Summary - Eight percent of objects carry 98 percent of the bytes; the two size classes are designed separately, and multipart upload multiplies the request count by 4.52 — 188.33 part requests at peak. - Part size is chosen from the bottom of a trough: the extra-byte ratio runs 1.0326 at 256 KB, 1.0075 at 2 MB, 1.0346 at 16 MB, 1.2015 in a single stream, and the best point sits exactly where sqrt(2 × overhead × interruption interval) puts it. - Three copies lose on two numbers at once: a storage factor of 3.00 against 1.40, an annual loss of 438 against 81.33 objects; but for small objects, coding pushes reads from 153.33 to 1533.33 requests, so three copies wins there. - The choice is tied to ND9: if the loss probability rises to 4 percent, the ratio becomes 0.42 and three copies pulls ahead. ## Next Step In this case the object was a meaningless byte sequence: stored as written, served as read, its content never inspected. The next case removes that assumption: the uploaded file gets multiple representations, and which one is served is decided by the requester's conditions at that moment. The question becomes: in how many forms is the same content kept, how is the processor cost of producing them weighed against the bytes they store, and when does an extra form become waste.