--- title: 'Storage and Data in the Cloud' source: 'https://academia.sh/en/courses/cloud-storage' language: en updated: '2026-09-13T05:07:36+00:00' license: 'CC BY-SA 4.0' lessons: 14 outcomes: - 'Selects the storage type by access pattern and counts its cost' - 'Draws out the decision handed over in a managed data service item by item' - 'Ties data transfer and egress cost to the design decision' --- # Storage and Data in the Cloud Renting data and the decisions handed over: the bucket, object, and key model, storage classes by access frequency, the automatic transition of versioning and lifecycle rules, the volume types and input-output budget of block storage, the shared file system, the retrieval time of the archive tier, presigned links that grant time-limited access, backup, patching, and failover automation in a managed relational database, read replicas, the managed key-value service with its partition key and capacity model, secondary indexes and change streams, the managed cache service, zonal and internet egress fees, and snapshots with point-in-time restore. ## Storage Types Where data lands: how the bucket, object, and key model differs from a file system, the price and retrieval trade-off of storage classes by access frequency, the automatic transitions and deletions of versioning and lifecycle rules, block storage's volume types and IOPS budget, the multi-writer use of a shared file system, the archive tier's retrieval time, and presigned URLs that grant time-limited access. ### Object Storage Source: https://academia.sh/en/courses/cloud-storage/object-storage Separating the bucket, object, and key model from a file system item by item: prefix instead of directory, copy instead of rename, no partial update, and listing as a billed operation; translating every difference into request counts and bytes, and counting the key layout as an unrecoverable item. The previous course ended by renting the machine and the network: instances launched, settled into subnets, passed through a filter, and opened outward. One question stayed open — where the **data** of everything running on those machines lives. The local disk is not the answer: when the instance shuts down the disk goes with it, and within a scaling group there is no telling which instance holds which data. In the rented layer, data has a place of its own, and choosing that place hands a set of decisions to the provider. This is the course's measure: every lesson counts **which decision is delegated, which one stays**, which of the ones that stay **cannot be undone**, and what data costs **on the way out**. The first type is the most common default: object storage. It is true that it is cheap, and that carries no decision; the decision is carried by the fact that object storage **is not** a file system. #### Bucket, Object, and Key There are three concepts. A **bucket** is a namespace; an **object** sits inside it; every object is reached by a **key**. A key looks like a path but is not one: `meter/2026/03/14/A118.json` is a plain string, and the slashes inside it carry no meaning for the store. The store looks this string up in a dictionary. Slashes only matter in a **prefix** query: keys that start with a given string can be listed. There is no such object as a directory, so there is no empty directory either, no directory delete, and no field to read for a directory's size. This single difference has four consequences, and all four can be counted: prefix instead of directory, copy instead of rename, full rewrite instead of partial update, and listing — assumed free — turning out to be a billed operation. The model below counts these over the fictional measurement network's object set. - **ST1** — The regional measurement network is fictional: 4,000 meters, 310 days of history, four object families. Counts and average sizes are written into the model. - **ST2** — Fees are denominated in a fictional **unit**, not a real provider's price; storage, request, and egress rates are written out explicitly in the model. - **ST3** — In the model a list request returns at most 1,000 keys; the page size varies from provider to provider. A request's round trip takes 45 ms and requests are sequential on a single stream. - **ST4** — Copying happens server-side: it incurs no egress fee, only a request fee. ```js // object/measurement-network.mjs — the regional measurement network is FICTIONAL: the object set and the fees are both a model. // Every fee is denominated in a fictional "unit"; it is not a real provider's price. export const METERS = 4000, DAYS = 310; // fictional network: 4,000 meters, 310 days of history export const FAMILIES = [ { name: "meter-readings", count: METERS * DAYS, avg: 2400, cadence: "daily" }, { name: "invoice-documents", count: 86_000, avg: 380 * 1024, cadence: "monthly" }, { name: "field-photos", count: 42_000, avg: Math.round(3.6 * 1024 * 1024), cadence: "once" }, { name: "work-order-records", count: 310_000, avg: 11 * 1024, cadence: "daily" }, ]; export const RATE = { // fictional unit storage: 0.023, // unit / GiB / month writeRequest: 0.0050 / 1000, // unit / request (put, copy, list) readRequest: 0.0004 / 1000, // unit / request (get, head) egressInternet: 0.085, // unit / GiB egressCrossRegion: 0.020, // unit / GiB }; export const PAGE = 1000; // a list request returns at most 1,000 keys export const ROUND_TRIP_MS = 45; // round-trip time of a request in the model export const gib = (bytes) => bytes / 1024 ** 3; export const totalBytes = FAMILIES.reduce((a, f) => a + f.count * f.avg, 0); export const totalCount = FAMILIES.reduce((a, f) => a + f.count, 0); export const listRequests = (keys) => Math.ceil(keys / PAGE); ``` #### Differences from a File System, Item by Item - **ST5** — Invoice documents receive 4,300 edits a month; each edit appends 2 KiB to the document. - **ST6** — Range requests are supported: only the requested byte range of an object can be read. ```js // object/model-difference.mjs — file system operations vs. object storage equivalents, item by item import { FAMILIES, RATE, PAGE, ROUND_TRIP_MS, gib, totalBytes, totalCount, listRequests } from "./measurement-network.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); console.log(`fictional bucket: ${totalCount.toLocaleString("en-US")} objects, ${g(gib(totalBytes), 0, 1)} GiB, ` + `${PAGE} keys per page, round-trip time in the model ${ROUND_TRIP_MS} ms`); console.log("\nfamily | count | GiB | list requests | list time (s)"); console.log("-------------------|----------|--------|---------------|--------------"); for (const f of FAMILIES) { const li = listRequests(f.count); console.log(`${f.name.padEnd(18)} | ${s(f.count, 8)} | ${g(gib(f.count * f.avg), 6, 1)} | ` + `${s(li, 13)} | ${g(li * ROUND_TRIP_MS / 1000, 13, 1)}`); } // Four operations: work that is a single call on a file system, translated into object storage. const invoice = FAMILIES[1], photo = FAMILIES[2], meter = FAMILIES[0]; const EDITS = 4300, APPEND = 2 * 1024, RANGE = 64 * 1024; const OP = [ { name: "learn directory size", fsCalls: 1, fsBytes: 0, objRequests: listRequests(meter.count), objBytes: 0 }, { name: "rename directory", fsCalls: 1, fsBytes: 0, objRequests: listRequests(meter.count) + 2 * meter.count, objBytes: meter.count * meter.avg }, { name: "append 2 KiB to a file", fsCalls: 1, fsBytes: APPEND, objRequests: 1, objBytes: invoice.avg }, { name: "monthly edits", fsCalls: EDITS, fsBytes: EDITS * APPEND, objRequests: EDITS, objBytes: EDITS * invoice.avg }, { name: "read a 64 KiB range", fsCalls: 1, fsBytes: RANGE, objRequests: 1, objBytes: RANGE }, { name: "read the whole photo", fsCalls: 1, fsBytes: photo.avg, objRequests: 1, objBytes: photo.avg }, ]; console.log("\noperation | on a file system | on object storage | byte amplification"); console.log(" | calls / bytes | requests / bytes | "); console.log("-------------------------|---------------------|--------------------------|-------------------"); for (const o of OP) { const factor = o.fsBytes === 0 ? (o.objBytes === 0 ? "1x" : "unmeasured") : (o.objBytes / o.fsBytes).toFixed(1) + "x"; console.log(`${o.name.padEnd(24)} | ${s(o.fsCalls, 6)} / ${s(o.fsBytes, 10)} | ` + `${s(o.objRequests, 9)} / ${s(o.objBytes, 12)} | ${factor.padStart(18)}`); } const rn = OP[1]; const requestFee = rn.objRequests * RATE.writeRequest; console.log(`\nrename: ${rn.objRequests.toLocaleString("en-US")} requests = ${g(requestFee, 0, 2)} units ` + `(fictional), moving ${g(gib(rn.objBytes), 0, 2)} GiB; on the file system 1 call, 0 bytes.`); const ed = OP[3]; console.log(`monthly edits: ${g(ed.fsBytes / 1024 ** 2, 0, 1)} MiB changed, ` + `${g(gib(ed.objBytes), 0, 2)} GiB written; with no partial update the factor is ${(ed.objBytes / ed.fsBytes).toFixed(0)}x.`); console.log(`range reads work: requesting ${RANGE / 1024} KiB returns ${RANGE / 1024} KiB — ` + `partial reads exist, partial writes do not.`); ``` ``` fictional bucket: 1,678,000 objects, 184.8 GiB, 1000 keys per page, round-trip time in the model 45 ms family | count | GiB | list requests | list time (s) -------------------|----------|--------|---------------|-------------- meter-readings | 1240000 | 2.8 | 1240 | 55.8 invoice-documents | 86000 | 31.2 | 86 | 3.9 field-photos | 42000 | 147.7 | 42 | 1.9 work-order-records | 310000 | 3.3 | 310 | 13.9 operation | on a file system | on object storage | byte amplification | calls / bytes | requests / bytes | -------------------------|---------------------|--------------------------|------------------- learn directory size | 1 / 0 | 1240 / 0 | 1x rename directory | 1 / 0 | 2481240 / 2976000000 | unmeasured append 2 KiB to a file | 1 / 2048 | 1 / 389120 | 190.0x monthly edits | 4300 / 8806400 | 4300 / 1673216000 | 190.0x read a 64 KiB range | 1 / 65536 | 1 / 65536 | 1.0x read the whole photo | 1 / 3774874 | 1 / 3774874 | 1.0x rename: 2,481,240 requests = 12.41 units (fictional), moving 2.77 GiB; on the file system 1 call, 0 bytes. monthly edits: 8.4 MiB changed, 1.56 GiB written; with no partial update the factor is 190x. range reads work: requesting 64 KiB returns 64 KiB — partial reads exist, partial writes do not. ``` The table separates the four differences from one another. Learning a directory's size is a single call on a file system; on object storage it is 1,240 list requests and takes 55.8 seconds in the model — while moving zero bytes. The meter family holds **1.5% of the bucket's bytes** but generates most of the list requests: listing depends not on bytes but on **key count**. Renaming a directory is a single call and zero bytes on a file system; on object storage it is 2,481,240 requests and moves 2.77 GiB start to finish. There is no equivalent, no multiplier — the operation type has changed. The partial-update difference is the sneakiest of the four because it throws no error, it only inflates the bill: appending 2 KiB to a 380 KiB document rewrites the whole document, a factor of 190x. At 4,300 edits a month, the changed 8.4 MiB becomes 1.56 GiB written. The last two rows balance the picture: partial **reads** exist. Requesting a 64 KiB range returns 64 KiB. Object storage allows reading in pieces but not writing in pieces; this asymmetry is the one-sentence reason small, frequently updated records do not belong in object storage. #### Key Layout Is an Unrecoverable Item Because there is no directory, the only thing that determines the access pattern is the key **itself**. The same data can be kept in two layouts and both are correct; the difference is which query can narrow by prefix. ```js // object/key-layout.mjs — same data, two key layouts: how many list requests do the queries need import { METERS, DAYS, FAMILIES, RATE, ROUND_TRIP_MS, gib, listRequests } from "./measurement-network.mjs"; const READING = FAMILIES[0], TOTAL = READING.count; // Prefix tree: a query narrows to few keys only if it can narrow by prefix; otherwise it scans all. const LAYOUT = { "date-first": { pattern: "meter////.json", day: METERS, meter: TOTAL }, "meter-first": { pattern: "meter////.json", day: TOTAL, meter: DAYS }, }; const QUERY = [ { name: "all readings of one day", field: "day", monthly: 30 }, { name: "all history of one meter", field: "meter", monthly: 900 }, ]; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); console.log(`${TOTAL.toLocaleString("en-US")} reading objects = ${METERS} meters x ${DAYS} days. ` + `Data is the same, only the key's first field changes.`); for (const [name, d] of Object.entries(LAYOUT)) console.log(` ${name.padEnd(11)} ${d.pattern}`); console.log("\nquery | layout | keys read | list requests | time (s)"); console.log("--------------------------|-------------|-----------|----------------|---------"); const cost = {}; for (const q of QUERY) for (const [name, d] of Object.entries(LAYOUT)) { const keys = d[q.field], li = listRequests(keys); cost[name] = (cost[name] ?? 0) + q.monthly * li; console.log(`${q.name.padEnd(25)} | ${name.padEnd(11)} | ${s(keys, 9)} | ` + `${s(li, 14)} | ${g(li * ROUND_TRIP_MS / 1000, 7, 1)}`); } console.log("\nmonthly total (30 daily queries + 900 meter queries):"); for (const [name, requests] of Object.entries(cost)) console.log(` ${name.padEnd(11)} ${s(requests.toLocaleString("en-US"), 11)} list requests = ` + `${(requests * RATE.writeRequest).toFixed(2).padStart(7)} units (fictional)`); const [a, b] = Object.values(cost); console.log(` difference ${(Math.max(a, b) / Math.min(a, b)).toFixed(0)}x; both are correct, both hold the same data.`); // Changing the layout later: no directory rename — every object is copied, then the old one deleted. const migration = listRequests(TOTAL) + 2 * TOTAL, bytes = TOTAL * READING.avg; console.log(`\ncost of changing the layout: ${listRequests(TOTAL)} listings + ${TOTAL.toLocaleString("en-US")} copies + ` + `${TOTAL.toLocaleString("en-US")} deletes = ${migration.toLocaleString("en-US")} requests`); console.log(` request fee ${(migration * RATE.writeRequest).toFixed(2)} units, moving ${gib(bytes).toFixed(2)} GiB, ` + `${(migration * ROUND_TRIP_MS / 3600000).toFixed(1)} hours in a single stream in the model`); console.log(` the same bytes moved out of the region: cross-region ${(gib(bytes) * RATE.egressCrossRegion).toFixed(3)} units, ` + `to the internet ${(gib(bytes) * RATE.egressInternet).toFixed(3)} units`); console.log(` migration fee equals ${(migration * RATE.writeRequest / (gib(bytes) * RATE.storage)).toFixed(0)} months of storage fees.`); ``` ``` 1,240,000 reading objects = 4000 meters x 310 days. Data is the same, only the key's first field changes. date-first meter////.json meter-first meter////.json query | layout | keys read | list requests | time (s) --------------------------|-------------|-----------|----------------|--------- all readings of one day | date-first | 4000 | 4 | 0.2 all readings of one day | meter-first | 1240000 | 1240 | 55.8 all history of one meter | date-first | 1240000 | 1240 | 55.8 all history of one meter | meter-first | 310 | 1 | 0.0 monthly total (30 daily queries + 900 meter queries): date-first 1,116,120 list requests = 5.58 units (fictional) meter-first 38,100 list requests = 0.19 units (fictional) difference 29x; both are correct, both hold the same data. cost of changing the layout: 1240 listings + 1,240,000 copies + 1,240,000 deletes = 2,481,240 requests request fee 12.41 units, moving 2.77 GiB, 31.0 hours in a single stream in the model the same bytes moved out of the region: cross-region 0.055 units, to the internet 0.236 units migration fee equals 195 months of storage fees. ``` Both layouts hold the same bytes, house the same objects, and pay the same storage fee. The difference is in the query: with date first, one day's readings arrive in 4 requests, and one meter's whole history is scanned in 1,240; with meter first, it is exactly reversed. This workload has 900 meter queries and 30 daily queries a month, so the wrong layout multiplies the monthly list-request count by 29x. The real item is in the last block. Changing the layout is not a configuration task: 2,481,240 requests, 2.77 GiB rewritten, and 31 hours on a single stream in the model. The migration's request fee equals **195 months of the same data's storage fee** — sixteen years of storage money spent on a one-time rename. **Key layout is this course's first unrecoverable item:** it is technically reversible, but in practice not without moving the data. The exit side stands apart too: if the migration stays inside the region it incurs no egress fee; moving the same bytes out of the region costs 0.055 units, downloading them to the internet costs 0.236 units. Data is cheap coming in, expensive going out. #### The Delegated Decision and What Stays When object storage is rented, disk placement, replica count, repair of a corrupted block, disk replacement, and the durability target all pass to the provider. These are not unrecoverable decisions, because they are not decisions at all: they have no visibility, no dial to turn. What stays is four things, and all four are measured in this course's later lessons: **key layout** (in this lesson), **storage class**, **versioning**, and **access scope**. What the four have in common is that once chosen, none can change without moving the data. The bucket name is a fifth, and the harshest one: the namespace is shared, the name cannot be changed later, and changing it means opening a new bucket and copying every object — the same 2,481,240 requests as above. #### Summary - Object storage is a dictionary: a bucket is a namespace, a key is a plain string, and a slash only means something in a prefix query. There is no such concept as a directory, an empty directory, or a directory's size. - The four differences separate out by number: directory size is 1,240 requests instead of 1 call, rename is 2,481,240 requests instead of 1 call, appending 2 KiB is a 190x rewrite, and range reads are one to one. - Listing depends on key count, not bytes: the meter family, holding 1.5% of the bucket's bytes, generates most of the list requests and takes 55.8 seconds for a single scan in the model. - Key layout fixes the access pattern in place: keeping the same data in two layouts changes the monthly list-request count by 29x, and changing the layout later costs 195 months of storage fees. - The delegated decisions are the invisible ones (placement, replication, repair); what stays is key layout, storage class, versioning, and access scope — none of the four can be undone without moving the data. #### Next Step Every number in this lesson came out under a single storage fee: every object was billed at the same rate. Yet the four families inside the fictional bucket resemble each other not at all — meter readings are written daily and read daily, invoice documents are written once a month and opened a few times a year, field photos are written once and almost never read. The next lesson carries this difference over to the price side: storage classes exist by access frequency, and as the storage fee drops the access fee rises. Three things will be measured — each class's break-even access frequency, the cost of keeping rarely accessed data in the frequent class, and the cost of dropping frequently accessed data into the infrequent class. ### Storage Classes Source: https://academia.sh/en/courses/cloud-storage/storage-classes Measuring tiers by access frequency: how the storage fee and the access fee run in opposite directions, scanning the break-even access frequency by object size, distributing objects into classes over access history, and counting the wrong class both ways — the annual cost of keeping infrequent data in the frequent class, and frequent data in archive. The previous lesson billed the four families in the fictional bucket at a single storage rate. Yet the families do not resemble each other: meter readings are small and numerous, invoice documents are large and rarely read, field photos are written once and almost never opened again. Object storage carries this difference over to the price side with the **storage class**: objects in the same bucket can be kept in different classes, and the class is chosen per object. Class is not a quality dial. Durability is identical across all four classes; the only thing that changes is how the fee splits between storage and access. So the sentence "the infrequent class is cheaper" carries no decision; the number that carries the decision is the **break-even access frequency**. - **ST7** — The four classes and their fees are fictional; classes are called only by access frequency, fees are denominated in a fictional **unit**. - **ST8** — Access history is a mixture model: the hot share and the hot and cold annual averages are written into the model, the generator lives in the model file, the seed is 40127. - **ST9** — Class is chosen per object; the annual read count stays fixed across the year. ```js // class/class-model.mjs — storage classes and access history are a MODEL. Fees are a fictional // unit; classes are called by access frequency, no commercial name is used. export const CLASS = [ // storage: unit/GiB/month, request and transition: unit/request, retrieval: unit/GiB { name: "frequent", storage: 0.0230, request: 0.00040 / 1000, retrieval: 0.000, minDays: 0, transition: 0.0050 / 1000 }, { name: "infrequent", storage: 0.0125, request: 0.00100 / 1000, retrieval: 0.010, minDays: 30, transition: 0.0100 / 1000 }, { name: "rare", storage: 0.0040, request: 0.00250 / 1000, retrieval: 0.020, minDays: 90, transition: 0.0250 / 1000 }, { name: "archive", storage: 0.0010, request: 0.00500 / 1000, retrieval: 0.030, minDays: 180, transition: 0.0500 / 1000 }, ]; export const EGRESS_INTERNET = 0.085; // unit/GiB, fictional // Access history is a mixture model: every family has a hot share, the rest is cold. export const FAMILIES = [ // count, average size, hot share, hot and cold annual read average { name: "meter-readings", count: 1_240_000, avg: 2400, hotShare: 0.06, hotAvg: 60, coldAvg: 0.4 }, { name: "invoice-documents", count: 86_000, avg: 380 * 1024, hotShare: 0.15, hotAvg: 26, coldAvg: 0.8 }, { name: "field-photos", count: 42_000, avg: Math.round(3.6 * 1024 * 1024), hotShare: 0.03, hotAvg: 40, coldAvg: 0.15 }, { name: "work-order-records", count: 310_000, avg: 11 * 1024, hotShare: 0.20, hotAvg: 34, coldAvg: 1.2 }, ]; export const gib = (bytes) => bytes / 1024 ** 3; // Annual fee: storage + read requests + retrieved bytes. Changing class is a separate fee. export const annualFee = (c, bytes, readsPerYear) => 12 * c.storage * gib(bytes) + readsPerYear * (c.request + gib(bytes) * c.retrieval); export const SEED = 40127; // we write the generator ourselves, the seed is visible: fictional export function* accessPattern(family) { // annual read count for each object (integer) let x = (SEED ^ family.count) >>> 0; const u = () => (x = (Math.imul(x, 1664525) + 1013904223) >>> 0) / 4294967296; for (let i = 0; i < family.count; i++) { const avg = u() < family.hotShare ? family.hotAvg : family.coldAvg; yield Math.floor(-avg * Math.log(1 - u())); } } ``` #### Break-Even Access Frequency ```js // class/break-even.mjs — storage and access fees run in opposite directions: the break-even access frequency is scanned import { CLASS, FAMILIES, gib, annualFee } from "./class-model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); console.log("class | storage | request (per 1,000) | retrieval unit/GiB | min retention"); console.log("---------------------|---------|----------------------|---------------------|--------------"); for (const c of CLASS) console.log(`${c.name.padEnd(20)} | ${g(c.storage, 7, 4)} | ${g(c.request * 1000, 20, 5)} | ` + `${g(c.retrieval, 19, 3)} | ${String(c.minDays + " days").padStart(13)}`); console.log("As storage drops, request and retrieval rise: there is no single dial to turn."); // Break-even scan: with a step of 0.1 reads a year, where the cheap-storage class turns expensive. const breakEven = (cheap, costly, bytes) => { for (let k = 0; k <= 2000; k++) { const r = k / 10; if (annualFee(cheap, bytes, r) > annualFee(costly, bytes, r)) return r; } return null; }; console.log("\nbreak-even access frequency (reads a year); above it the cheap-storage class turns costly:"); console.log("family | avg size | infrequent>frequent | rare>infrequent | archive>rare "); console.log("-------------------|------------|----------------------|------------------|-------------"); for (const f of FAMILIES) { const c = [breakEven(CLASS[1], CLASS[0], f.avg), breakEven(CLASS[2], CLASS[1], f.avg), breakEven(CLASS[3], CLASS[2], f.avg)]; const size = f.avg < 1024 ** 2 ? (f.avg / 1024).toFixed(1) + " KiB" : (f.avg / 1024 ** 2).toFixed(1) + " MiB"; console.log(`${f.name.padEnd(18)} | ${size.padStart(10)} | ` + c.map((x, i) => x.toFixed(1).padStart([20, 16, 13][i])).join(" | ")); } console.log("Break-even drops for small objects: the request fee outweighs the byte fee, and cheap"); console.log("storage only pays off within a very narrow frequency range."); // Same family, same bytes, two different frequencies: the point where the ranking reverses. const f = FAMILIES[1], y = (r) => CLASS.map((c) => annualFee(c, f.avg, r) * f.count); const [t0, t12] = [y(0), y(12)]; console.log(`\n${f.name} (${gib(f.count * f.avg).toFixed(1)} GiB) annual total, fictional unit:`); console.log(` zero reads : frequent ${g(t0[0], 6)} | archive ${g(t0[3], 6)} -> archive ${(t0[0] / t0[3]).toFixed(0)}x cheaper`); console.log(` 12 reads/year: frequent ${g(t12[0], 6)} | archive ${g(t12[3], 6)} -> ranking reversed`); ``` ``` class | storage | request (per 1,000) | retrieval unit/GiB | min retention ---------------------|---------|----------------------|---------------------|-------------- frequent | 0.0230 | 0.00040 | 0.000 | 0 days infrequent | 0.0125 | 0.00100 | 0.010 | 30 days rare | 0.0040 | 0.00250 | 0.020 | 90 days archive | 0.0010 | 0.00500 | 0.030 | 180 days As storage drops, request and retrieval rise: there is no single dial to turn. break-even access frequency (reads a year); above it the cheap-storage class turns costly: family | avg size | infrequent>frequent | rare>infrequent | archive>rare -------------------|------------|----------------------|------------------|------------- meter-readings | 2.3 KiB | 0.5 | 0.2 | 0.1 invoice-documents | 380.0 KiB | 10.9 | 7.3 | 2.2 field-photos | 3.6 MiB | 12.4 | 9.8 | 3.4 work-order-records | 11.0 KiB | 1.9 | 0.7 | 0.2 Break-even drops for small objects: the request fee outweighs the byte fee, and cheap storage only pays off within a very narrow frequency range. invoice-documents (31.2 GiB) annual total, fictional unit: zero reads : frequent 8.60 | archive 0.37 -> archive 23x cheaper 12 reads/year: frequent 9.01 | archive 16.75 -> ranking reversed ``` The break-even table shows that class choice depends on **object size**. If a 380 KiB invoice document stays under 10.9 reads a year, the infrequent class pays off; for a 2.3 KiB meter reading the same threshold is 0.5 — putting a record that is read once a year into the cheap-storage class is a loss. The reason is the flat request fee: for a small object, the cost of access is measured not in bytes but in **request count**. The last two lines show the same thing on a single family. #### Distribution over Access History - **ST10** — Changing class is a transition request, billed separately, and the minimum retention period starts at the transition. - **ST11** — The retrieval fee is separate from the egress fee; both are paid together for the same bytes. - **ST12** — In the model, an object read more than 24 times a year counts as "hot." ```js // class/distribution.mjs — classes are distributed over access history; the wrong class is counted both ways import { CLASS, FAMILIES, EGRESS_INTERNET, gib, annualFee, SEED, accessPattern } from "./class-model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const cheapest = (bytes, r) => { const t = CLASS.map((c) => annualFee(c, bytes, r)); return t.indexOf(Math.min(...t)); }; // Single pass: every family's access history is scanned, summaries and strategy totals come out together. const SUMMARY = [], STRATEGY = { "all frequent": 0, "all infrequent": 0, "all archive": 0, "by family": 0, "per object": 0 }; let neverRead = 0, neverReadBytes = 0, neverReadExcess = 0; let hotInArchive = 0, hotInArchiveReads = 0, hotInArchiveBytes = 0, hotInArchiveExcess = 0, retrievedBytes = 0; for (const f of FAMILIES) { let totalReads = 0, zero = 0, distribution = [0, 0, 0, 0], perObject = 0; const familyTotal = CLASS.map(() => 0), threshold = 24; // an object read more than 24 times a year counts as hot for (const r of accessPattern(f)) { totalReads += r; if (r === 0) zero++; const i = cheapest(f.avg, r); distribution[i]++; perObject += annualFee(CLASS[i], f.avg, r); for (let k = 0; k < CLASS.length; k++) familyTotal[k] += annualFee(CLASS[k], f.avg, r); if (r === 0) { // a never-read object kept in the frequent class neverRead++; neverReadBytes += f.avg; neverReadExcess += annualFee(CLASS[0], f.avg, 0) - annualFee(CLASS[3], f.avg, 0); } else if (r > threshold) { // a hot object dropped into archive hotInArchive++; hotInArchiveReads += r; hotInArchiveBytes += f.avg; retrievedBytes += r * f.avg; hotInArchiveExcess += annualFee(CLASS[3], f.avg, r) - annualFee(CLASS[0], f.avg, r); } } const best = familyTotal.indexOf(Math.min(...familyTotal)); STRATEGY["all frequent"] += familyTotal[0]; STRATEGY["all infrequent"] += familyTotal[1]; STRATEGY["all archive"] += familyTotal[3]; STRATEGY["by family"] += familyTotal[best]; STRATEGY["per object"] += perObject; SUMMARY.push({ f, totalReads, zero, distribution }); } console.log(`access history is a model; the generator is in this file, seed ${SEED}. Annual read counts:`); console.log("\nfamily | never read | avg reads/yr | cheapest class per object"); console.log(" | | | frequent | infrequent | rare | archive"); console.log("-------------------|------------|--------------|-----------|------------|-------|--------"); for (const o of SUMMARY) console.log(`${o.f.name.padEnd(18)} | ${g(100 * o.zero / o.f.count, 9, 1)}% | ${g(o.totalReads / o.f.count, 12, 2)} | ` + o.distribution.map((x, i) => String(x).padStart([9, 10, 5, 7][i])).join(" | ")); console.log("\nstrategy | annual total | excess over best | share of excess"); console.log("----------------|--------------|-------------------|----------------"); const best = STRATEGY["per object"]; for (const [name, t] of Object.entries(STRATEGY)) console.log(`${name.padEnd(15)} | ${g(t, 12)} | ${g(t - best, 17)} | ${g(100 * (t - best) / best, 14, 1)}%`); console.log("\nthe wrong class, both ways:"); console.log(` cold data in the frequent class: ${neverRead.toLocaleString("en-US")} objects, ` + `${g(gib(neverReadBytes), 0, 1)} GiB never read; keeping them in the frequent class instead of archive costs ` + `${g(neverReadExcess, 0, 2)} units a year extra.`); console.log(` hot data in the archive class: ${hotInArchive.toLocaleString("en-US")} objects, ` + `${g(gib(hotInArchiveBytes), 0, 1)} GiB, ${hotInArchiveReads.toLocaleString("en-US")} reads a year; ` + `keeping them in archive costs ${g(hotInArchiveExcess, 0, 2)} units a year extra.`); console.log(` if the ${g(gib(retrievedBytes), 0, 1)} GiB retrieved by these reads also leaves to the internet, egress adds ` + `${g(gib(retrievedBytes) * EGRESS_INTERNET, 0, 2)} units on top: the retrieval fee does not replace egress, it stacks on it.`); // Changing class is reversible, not free: it carries the minimum retention period. const f = FAMILIES[1], N = 12_000, DAY = 8, infrequent = CLASS[1]; const transitionFee = 2 * N * infrequent.transition; const minRetentionDebt = N * infrequent.storage * gib(f.avg) * (infrequent.minDays / 30); const retrievalFee = N * gib(f.avg) * infrequent.retrieval; const savings = N * (CLASS[0].storage - infrequent.storage) * gib(f.avg) * (DAY / 30); console.log(`\n${N.toLocaleString("en-US")} invoice documents moved to infrequent, found hot on day ${DAY}, moved back:`); console.log(` round-trip transition request ${g(transitionFee, 0, 3)} | min ${infrequent.minDays}-day retention debt ${g(minRetentionDebt, 0, 3)} | ` + `retrieval ${g(retrievalFee, 0, 3)}`); console.log(` total ${g(transitionFee + minRetentionDebt + retrievalFee, 0, 3)} units; saved over ${DAY} days ${g(savings, 0, 3)} units; ` + `net ${g(transitionFee + minRetentionDebt + retrievalFee - savings, 0, 3)} units loss.`); console.log(` the decision was reversed, the fee was not: minimum retention debt stands even if the object is deleted.`); ``` ``` access history is a model; the generator is in this file, seed 40127. Annual read counts: family | never read | avg reads/yr | cheapest class per object | | | frequent | infrequent | rare | archive -------------------|------------|--------------|-----------|------------|-------|-------- meter-readings | 86.4% | 3.66 | 168911 | 0 | 0 | 1071089 invoice-documents | 61.4% | 4.16 | 8311 | 1095 | 3774 | 72820 field-photos | 97.0% | 1.22 | 920 | 64 | 142 | 40874 work-order-records | 45.7% | 7.34 | 105257 | 63012 | 0 | 141731 strategy | annual total | excess over best | share of excess ----------------|--------------|-------------------|---------------- all frequent | 53.91 | 45.78 | 563.0% all infrequent | 38.39 | 30.26 | 372.2% all archive | 48.65 | 40.52 | 498.4% by family | 16.80 | 8.67 | 106.6% per object | 8.13 | 0.00 | 0.0% the wrong class, both ways: cold data in the frequent class: 1,306,318 objects, 166.2 GiB never read; keeping them in the frequent class instead of archive costs 43.87 units a year extra. hot data in the archive class: 84,485 objects, 4.6 GiB, 6,192,838 reads a year; keeping them in archive costs 35.56 units a year extra. if the 276.3 GiB retrieved by these reads also leaves to the internet, egress adds 23.48 units on top: the retrieval fee does not replace egress, it stacks on it. 12,000 invoice documents moved to infrequent, found hot on day 8, moved back: round-trip transition request 0.240 | min 30-day retention debt 0.054 | retrieval 0.043 total 0.338 units; saved over 8 days 0.012 units; net 0.326 units loss. the decision was reversed, the fee was not: minimum retention debt stands even if the object is deleted. ``` #### The Wrong Class's Two Directions The distribution column confirms the break-even table: for meter readings the two middle classes take zero objects, and the 1,240,000 records split between frequent and archive. As objects get larger the middle classes' range opens up, and for invoice documents all four are used. The strategy table shows that all three single-class approaches lose. Keeping everything in the frequent class is 6.6x the best distribution. Dropping everything into archive is, counter to intuition, more expensive than keeping everything in the infrequent class — 48.65 against 38.39 — because the retrieval fee of the hot objects swallows what was saved. A single class at the family level brings the total down to 16.80, but that is still 106% in excess; half the gain lies in the **hot–cold split within each family**. The two directions are counted separately. The 1,306,318 objects that are never read — 166.2 GiB — cost 43.87 units a year extra when kept in the frequent class; this direction is silent, it throws no error. The reverse direction holds less data but costs more money: the 84,485 objects read more than 24 times a year are only 4.6 GiB, but keeping them in archive costs 35.56 units a year extra. If the retrieved 276.3 GiB also leaves to the internet, a 23.48-unit egress fee stacks **on top** of that. #### The Decision Reversed, the Fee Not Unlike key layout, storage class looks reversible: one request moves an object to another class. The last block counts the cost. 12,000 invoice documents are moved to the infrequent class and moved back on day eight: 0.240 transition request, 0.054 minimum retention debt, 0.043 retrieval, total 0.338 units. What was saved over eight days is 0.012 units; the net loss is twenty-seven times the saving. **The unrecoverable item is the minimum retention debt.** The moment an object enters a class, that class's period is committed; the debt stands whether the object is moved or deleted. The delegated side is constant: durability, replication, and placement belong to the provider in all four classes. #### Summary - Class is not a quality dial but a trade-off: as storage drops, request and retrieval rise, durability does not change. - Break-even frequency depends on object size: 10.9 reads a year for a 380 KiB document, 0.5 reads for a 2.3 KiB record. For small objects the middle classes never pay off. - Per-object selection costs 8.13 units; keeping everything in the frequent class costs 53.91, dropping everything into archive costs 48.65 — archive is not automatically cheap. - The wrong class costs in both directions: 166.2 GiB that is never read costs 43.87 units a year extra in the frequent class, and 4.6 GiB that is read often costs 35.56 units a year extra in archive. The retrieval fee stacks on top of egress. - The class decision is reversible, its fee is not: a transition reversed in eight days leaves a loss twenty-seven times the saving; minimum retention debt stands even if the object is deleted. #### Next Step This lesson's class distribution was done by hand: every object's annual read count was known, and the cheapest class was chosen. In a real bucket this information is not known in advance; a script that moves every object one by one carries away the gain in the fee of the requests it makes. The next lesson hands the decision to the store: a lifecycle rule changes an object's class and deletes it by age. Three things will be measured — the bytes that accumulate without a rule, the number of objects transitioned and deleted by the rule, and an early transition's retrieval cost. Versioning is also introduced in the same lesson: once turned on, it is shown with a number that deletion does not actually delete. ### Versioning and Lifecycle Rules Source: https://academia.sh/en/courses/cloud-storage/versioning-and-lifecycle-rules Running automatic transitions and deletion over a 36-month object lifetime: the bytes that accumulate without a rule, the number of objects transitioned and deleted by the rule, the transition fee swallowing the gain on small objects, an early transition's retrieval cost, and the deleted object not actually being deleted once versioning is turned on. The previous lesson's class distribution rested on an assumption: every object's annual read count is known and it is placed by hand into the right class. Across millions of objects this is neither known nor done by hand. Object storage offers a **lifecycle rule** for this: objects under a prefix change class or get deleted once they reach a given age. The rule looks at **age**, not access, and this distinction carries a measurable cost. The same lesson also introduces versioning. In this model versioning is a bucket-level switch, and once turned on there is no going back; with real providers the level of that switch and whether it can be turned off vary, but the old versions that pile up stay everywhere. Both are measured together: versioning grows bytes, the lifecycle rule shrinks them. - **ST13** — The 36-month object lifetime is a model: monthly production, average size, and the age at which reads halve are written into the model. There is no randomness, every number comes from a cohort. - **ST14** — A transition is a request, billed per object; the minimum retention period starts when the class is entered. - **ST15** — The rule only looks at age; it does not see whether an object was actually read. ```js // lifecycle/model.mjs — a 36-month object lifetime is a MODEL: no randomness, every number comes from a cohort. // Fees are a fictional unit; classes are called only by access frequency. export const MONTHS = 36; export const CLASS = { // storage unit/GiB/month, request and transition unit/request, retrieval unit/GiB frequent: { storage: 0.0230, request: 0.00040 / 1000, retrieval: 0.000, minMonths: 0, transition: 0.0050 / 1000 }, infrequent: { storage: 0.0125, request: 0.00100 / 1000, retrieval: 0.010, minMonths: 1, transition: 0.0100 / 1000 }, archive: { storage: 0.0010, request: 0.00500 / 1000, retrieval: 0.030, minMonths: 6, transition: 0.0500 / 1000 }, }; // births: objects created each month; base: monthly reads at age zero; halfLife: the month reads halve export const FAMILIES = [ { name: "meter-readings", births: 120_000, avg: 2400, base: 3.0, halfLife: 1.5 }, { name: "invoice-documents", births: 7_200, avg: 380 * 1024, base: 2.2, halfLife: 4 }, { name: "field-photos", births: 3_500, avg: Math.round(3.6 * 1024 * 1024), base: 0.9, halfLife: 2 }, { name: "work-order-records", births: 26_000, avg: 11 * 1024, base: 4.0, halfLife: 3 }, ]; export const gib = (bytes) => bytes / 1024 ** 3; export const reads = (f, age) => f.base * Math.pow(0.5, age / f.halfLife); // Rule: change class at an age (month) threshold; remove the object at the delete threshold. export const RULE = { "no rule": { transitions: [], delete: null }, moderate: { transitions: [[3, "infrequent"], [12, "archive"]], delete: 30 }, aggressive: { transitions: [[1, "infrequent"], [3, "archive"]], delete: 7 }, "early-archive": { transitions: [[1, "archive"]], delete: null }, }; export const classAt = (rule, age) => { // the class occupied at this age and the age it was entered let name = "frequent", entered = 0; for (const [threshold, c] of rule.transitions) if (age >= threshold) { name = c; entered = threshold; } return { c: CLASS[name], name, entered }; }; ``` #### The Rule Looks at Age, Not Access ```js // lifecycle/rule.mjs — lifecycle rules are run over cohorts across 36 months import { MONTHS, CLASS, FAMILIES, RULE, gib, reads, classAt } from "./model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); const empty = () => ({ storage: 0, request: 0, retrieval: 0, transitionFee: 0, transitionCount: 0, debt: 0, deleted: 0, remainingBytes: 0 }); const run = (rule, family) => { // one family's 36-month lifetime const r = empty(); for (let birth = 0; birth < MONTHS; birth++) // the cohort born each month is aged month by month for (let month = birth; month < MONTHS; month++) { const age = month - birth; if (rule.delete !== null && age >= rule.delete) { // minimum retention debt falls due at deletion if (age === rule.delete) { const at = classAt(rule, age - 1); r.debt += family.births * Math.max(0, at.c.minMonths - (age - at.entered)) * at.c.storage * gib(family.avg); r.deleted += family.births; } continue; } const at = classAt(rule, age), readCount = reads(family, age); if (rule.transitions.some(([t]) => t === age)) { r.transitionCount += family.births; r.transitionFee += family.births * at.c.transition; } r.storage += family.births * gib(family.avg) * at.c.storage; r.request += family.births * readCount * at.c.request; r.retrieval += family.births * readCount * gib(family.avg) * at.c.retrieval; if (month === MONTHS - 1) r.remainingBytes += family.births * family.avg; } r.total = r.storage + r.request + r.retrieval + r.transitionFee + r.debt; return r; }; const T = {}; // T[rule][family] and T[rule].all for (const [name, rule] of Object.entries(RULE)) { T[name] = { all: empty() }; for (const f of FAMILIES) { const r = T[name][f.name] = run(rule, f); for (const a of ["storage", "request", "retrieval", "transitionFee", "transitionCount", "debt", "deleted", "remainingBytes", "total"]) T[name].all[a] = (T[name].all[a] ?? 0) + r[a]; } } console.log(`${FAMILIES.reduce((a, f) => a + f.births, 0).toLocaleString("en-US")} objects/month, ${MONTHS} months; ` + `reads halve with age, no randomness. Fees are a fictional unit.`); console.log("\nrule | GiB | transitions | storage | retrieval | transition fee | total"); console.log("---------------|--------|-------------|---------|-----------|-----------------|------"); for (const [name, r] of Object.entries(T).map(([a, x]) => [a, x.all])) console.log(`${name.padEnd(14)} | ${g(gib(r.remainingBytes), 6, 1)} | ${s(r.transitionCount.toLocaleString("en-US"), 11)} | ` + `${g(r.storage, 7)} | ${g(r.retrieval, 9)} | ${g(r.transitionFee, 15)} | ${g(r.total, 6)}`); console.log("\n36-month total, family by family (fictional unit):"); console.log("family |" + Object.keys(RULE).map((a) => a.padStart(14)).join(" |") + " | cheapest rule"); console.log("-------------------|" + Object.keys(RULE).map(() => "-".repeat(15)).join("|") + "|--------------"); for (const f of FAMILIES) { const v = Object.keys(RULE).map((a) => T[a][f.name].total); console.log(`${f.name.padEnd(18)} |` + v.map((x) => g(x, 14)).join(" |") + ` | ${Object.keys(RULE)[v.indexOf(Math.min(...v))]}`); } const [none, moderate, aggressive, early] = ["no rule", "moderate", "aggressive", "early-archive"].map((a) => T[a]); console.log(`\nwith no rule, ${gib(none.all.remainingBytes).toFixed(1)} GiB accumulates by month 36, no object is ever deleted. ` + `The moderate rule deletes ${moderate.all.deleted.toLocaleString("en-US")} objects, cutting storage by ` + `${(100 * (1 - moderate.all.storage / none.all.storage)).toFixed(1)} points, but multiplies the transition-fee ` + `total by ${(moderate.all.total / none.all.total).toFixed(2)}x.`); const mixed = FAMILIES.reduce((a, f) => a + Math.min(...Object.keys(RULE).map((k) => T[k][f.name].total)), 0); console.log(`the mixed rule chosen by family costs ${mixed.toFixed(2)} units: below the best single rule's ` + `(${Math.min(...Object.values(T).map((x) => x.all.total)).toFixed(2)}). A rule is written by object type.`); console.log(`the cost of an early transition is in the retrieval column: moderate ${moderate.all.retrieval.toFixed(2)}, early-archive ` + `${early.all.retrieval.toFixed(2)} units (${(early.all.retrieval / moderate.all.retrieval).toFixed(1)}x); early-archive saves ` + `${(moderate.all.storage - early.all.storage).toFixed(2)} in storage and loses ` + `${(early.all.retrieval + early.all.request - moderate.all.retrieval - moderate.all.request).toFixed(2)} units in retrieval and requests.`); console.log(`the aggressive rule deletes at month 7, and archive carries a minimum ${CLASS.archive.minMonths}-month debt: ` + `${aggressive.all.debt.toFixed(2)} units are paid for objects that no longer exist.`); ``` ``` 156,700 objects/month, 36 months; reads halve with age, no randomness. Fees are a fictional unit. rule | GiB | transitions | storage | retrieval | transition fee | total ---------------|--------|-------------|---------|-----------|-----------------|------ no rule | 556.4 | 0 | 236.74 | 0.00 | 0.00 | 257.94 moderate | 463.6 | 8,931,900 | 92.06 | 13.05 | 239.75 | 379.90 aggressive | 108.2 | 10,655,600 | 28.07 | 30.28 | 313.40 | 460.64 early-archive | 556.4 | 5,484,500 | 22.53 | 58.48 | 274.22 | 540.24 36-month total, family by family (fictional unit): family | no rule | moderate | aggressive | early-archive | cheapest rule -------------------|---------------|---------------|---------------|---------------|-------------- meter-readings | 17.46 | 203.62 | 292.04 | 318.98 | no rule invoice-documents | 41.19 | 37.32 | 38.33 | 56.13 | moderate field-photos | 188.63 | 83.22 | 44.92 | 51.49 | aggressive work-order-records | 10.66 | 55.74 | 85.34 | 113.64 | no rule with no rule, 556.4 GiB accumulates by month 36, no object is ever deleted. The moderate rule deletes 940,200 objects, cutting storage by 61.1 points, but multiplies the transition-fee total by 1.47x. the mixed rule chosen by family costs 110.36 units: below the best single rule's (257.94). A rule is written by object type. the cost of an early transition is in the retrieval column: moderate 13.05, early-archive 58.48 units (4.5x); early-archive saves 69.52 in storage and loses 195.39 units in retrieval and requests. the aggressive rule deletes at month 7, and archive carries a minimum 6-month debt: 0.90 units are paid for objects that no longer exist. ``` With no rule, 556.4 GiB accumulates over thirty-six months and no object is ever deleted; storage is nearly the whole total. The moderate rule cuts this item by 61.1%, deletes 940,200 objects, pulls bytes down to 463.6 GiB — and **raises** the total. The reason is in the transition-fee column: 8,931,900 transition requests cost 239.75 units, more than what was gained in storage. The family table shows where the rule pays off. For field photos the aggressive rule brings the total from 188.63 down to 44.92 units: the object is large, so the transition request carries many bytes. For meter readings the same rule raises it from 17.46 to 292.04 units; for a 2.3 KiB object, the transition request costs more than storing that object for years. This is why the rule is written **at the prefix level**: the mixed rule, at 110.36 units, is under half of the best single rule. Early transition opens a third item. The early-archive rule gains 69.52 units in storage and loses 195.39 units in retrieval and requests: when an object still being read in its first month is sent to archive, every read incurs a retrieval fee. The objects the aggressive rule deletes at month seven, meanwhile, never fill archive's six-month period; 0.90 units are paid for objects that no longer exist. #### Versioning: Where Deletion Doesn't Delete - **ST16** — The update rate is a model: overwrites grow rarer as an object ages, and every overwrite leaves an old version behind. - **ST17** — Versioning is bucket-level; while it is on, deletion places a delete marker and turns the current version into an old version. - **ST18** — Old versions are cleaned up only by a separate lifecycle rule. ```js // lifecycle/versioning.mjs — what deleting and disabling do while versioning is on is counted (MODEL) import { MONTHS, CLASS, FAMILIES, gib } from "./model.mjs"; // Update MODEL: overwrites grow rarer as an object ages. Every overwrite leaves an old version behind. const UPDATE = { "work-order-records": { base: 0.85, halfLife: 2 }, "invoice-documents": { base: 0.30, halfLife: 3 } }; const DELETE_MONTH = 30, VERSION_LIFE = 3, DISABLE_MONTH = 24; // old-version rule: clean up after 3 months const selected = FAMILIES.filter((f) => UPDATE[f.name]); const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); // Every month: current-object bytes and produced old-version bytes (by the month produced) are summed. const run = ({ versioning, versionRule, disable }) => { const r = { current: 0, currentBytes: 0, oldVersions: 0, oldVersionBytes: 0, storage: 0, deleted: 0, freed: 0 }; for (const f of selected) { const u = UPDATE[f.name], produced = []; // produced[m] = old versions born that month for (let m = 0; m < MONTHS; m++) produced.push(0); for (let m = 0; m < MONTHS; m++) { let current = 0; for (let birth = 0; birth <= m; birth++) { const age = m - birth; if (age >= DELETE_MONTH) { // delete: bytes are freed if versioning is off if (age === DELETE_MONTH) { r.deleted += f.births; if (versioning && !(disable && m >= DISABLE_MONTH)) produced[m] += f.births; else r.freed += f.births * f.avg; } continue; } current += f.births; const write = f.births * u.base * Math.pow(0.5, age / u.halfLife); if (versioning && !(disable && m >= DISABLE_MONTH)) produced[m] += write; } let old = 0; // old versions still standing that month for (let k = 0; k <= m; k++) if (!versionRule || m - k < VERSION_LIFE) old += produced[k]; r.storage += gib(f.avg) * (current + old) * CLASS.frequent.storage; if (m === MONTHS - 1) { r.current += current; r.currentBytes += current * f.avg; r.oldVersions += old; r.oldVersionBytes += old * f.avg; } } } return r; }; const D = { "versioning off": run({ versioning: false }), "versioning on": run({ versioning: true }), "on + version rule": run({ versioning: true, versionRule: true }), "disabled at month 24": run({ versioning: true, disable: true }), }; console.log(`${selected.map((f) => f.name).join(" + ")}; ${MONTHS} months, delete at month ${DELETE_MONTH}, ` + `old-version rule ${VERSION_LIFE} months. Update rate is in the model.`); console.log("\nstate | current objects | old versions | old version GiB | 36-month storage"); console.log("----------------------|-----------------|--------------|------------------|-----------------"); for (const [name, r] of Object.entries(D)) console.log(`${name.padEnd(21)} | ${s(Math.round(r.current).toLocaleString("en-US"), 15)} | ` + `${s(Math.round(r.oldVersions).toLocaleString("en-US"), 12)} | ${g(gib(r.oldVersionBytes), 16, 1)} | ${g(r.storage, 16)}`); const [off, on, rule, disabled] = Object.values(D); console.log(`\ndeletion experiment: ${Math.round(off.deleted).toLocaleString("en-US")} objects deleted. Bytes freed — ` + `versioning off ${gib(off.freed).toFixed(1)} GiB, on ${gib(on.freed).toFixed(1)} GiB.`); console.log(` old-version to current-data ratio ${(on.oldVersionBytes / on.currentBytes).toFixed(2)}; the version rule ` + `recovers ${gib(on.oldVersionBytes - rule.oldVersionBytes).toFixed(1)} GiB.`); console.log(`disable experiment: disabled at month ${DISABLE_MONTH}. No new versions are produced, the accumulated ` + `${gib(disabled.oldVersionBytes).toFixed(1)} GiB still stands and is billed for ${MONTHS - DISABLE_MONTH} more months;`); console.log(` deletions after disabling free ${gib(disabled.freed).toFixed(1)} GiB; the only thing that can be done ` + `for the earlier ones is to write a separate version rule.`); ``` ``` invoice-documents + work-order-records; 36 months, delete at month 30, old-version rule 3 months. Update rate is in the model. state | current objects | old versions | old version GiB | 36-month storage ----------------------|-----------------|--------------|------------------|----------------- versioning off | 996,000 | 0 | 0.0 | 42.75 versioning on | 996,000 | 3,070,001 | 165.9 | 102.12 on + version rule | 996,000 | 357,335 | 22.4 | 53.68 disabled at month 24 | 996,000 | 1,839,941 | 93.6 | 92.51 deletion experiment: 199,200 objects deleted. Bytes freed — versioning off 17.3 GiB, on 0.0 GiB. old-version to current-data ratio 1.92; the version rule recovers 143.5 GiB. disable experiment: disabled at month 24. No new versions are produced, the accumulated 93.6 GiB still stands and is billed for 12 more months; deletions after disabling free 17.3 GiB; the only thing that can be done for the earlier ones is to write a separate version rule. ``` The current object count is the same across all four rows: 996,000. What the application sees does not change; what changes is the bill. Once versioning is turned on, 3,070,001 old versions accumulate, taking up 165.9 GiB — 1.92 times the current data, bytes produced by overwriting itself. The deletion experiment settles the concept in place. 199,200 objects are deleted: with versioning off, 17.3 GiB is freed; with it on, **0.0 GiB**. The deletion succeeds, the object no longer appears in listings — and the bytes stay put, because the deleted object has turned into an old version. Once a separate rule is written for old versions, 143.5 GiB comes back, and storage drops from 102.12 to 53.68 units; until that rule is written, turning on versioning silently makes deletion meaningless. The disable experiment shows the unrecoverability. Versioning is disabled in month twenty-four: no new version is produced, subsequent deletions free bytes again, but the accumulated 93.6 GiB stands and is billed for twelve more months. **Versioning is this course's second unrecoverable item:** turning it on is one switch, turning it off is not the same switch. The provider only **runs** the rule; the thresholds, the deletion, and the old-version lifetime all stay with you. #### Summary - The rule looks at age, not access: the early-archive rule gains 69.52 units in storage and loses 195.39 units in retrieval and requests, a 4.5x increase in the retrieval item. - The transition fee is per object: the moderate rule cuts storage by 61.1% but multiplies the total by 1.47x. A rule is written at the prefix level, not the bucket level. - The mixed rule costs 110.36 units, the best single rule 257.94: the aggressive rule pays off for field photos, the same rule costs seventeen times as much for meter readings. - Early deletion incurs minimum retention debt: 0.90 units are paid for objects deleted at month seven, and the objects no longer exist. - With versioning on, deleting 199,200 objects frees 0.0 GiB, and 165.9 GiB of old versions accumulate. Disabling it does not clean up the accumulated versions, it only stops producing new ones. #### Next Step These three lessons share a single assumption: data is written whole and read whole. One part of the fictional measurement network does not work this way — the collector process makes hundreds of small writes a second to a database file, and every write changes a few kilobytes in the middle of the file. The 190x write amplification measured in the first lesson makes this workload alone impossible on object storage. The next lesson moves to the type that presents data as byte blocks. Three things will be measured — the input/output budget that a volume's type and size give, the point where workload profiles saturate that budget, and the design trap opened by size and performance being tied together. ### Block Storage Source: https://academia.sh/en/courses/cloud-storage/block-storage How a volume's type and size give an input/output budget: applying workload profiles to that budget, counting the saturation point and the idle budget, the design trap opened by size and performance being tied together, and the volume's inability to shrink and its binding to a single instance. The previous three lessons treated data as an object written and read whole. The fictional measurement network's collector process does not work this way: it changes a few kilobytes in the middle of a database file thousands of times a second. The 190x write amplification measured in the first lesson makes this workload impossible on object storage. What is needed is a layer that presents data as addressable **blocks** and on top of which a file system can be built. What is rented in this type is a **volume**. In this model a volume attaches to a single instance, and the operating system sees it as a disk. Some providers allow certain volume types to attach to more than one instance, but that arrangement needs a cluster-aware file system. The delegated side is broad: physical disks, replication, repair of a bad sector, and disk replacement all belong to the provider. What stays comes down to two numbers — **type** and **size**. That these two determine a third, the input/output budget, is the source of every measurement in this lesson. - **ST19** — The volume-type catalog and its fees are fictional; types are called by their capabilities. - **ST20** — An I/O operation rounds up to the volume's **chunk**: on a type that counts 16 KiB chunks a 1 MiB read spends 64 chunks, on a type that counts 1 MiB chunks it spends 1. - **ST21** — Workload profiles are a model: operation size, operation count, and data size are written into the model. ```js // block/model.mjs — the block volume catalog and workload profiles are a MODEL. Fees are a fictional // unit; no product or volume family name is used, types are called by their capabilities. export const TYPE = { // unitKib: the chunk size this type counts I/O operations in; iopsRate: fee per provisioned IOPS "general purpose": { unitKib: 16, perGib: 3, base: 100, maxIops: 16000, mibPerGib: 0.25, maxMib: 250, rate: 0.080, iopsRate: 0 }, "performance-provisioned": { unitKib: 16, perGib: 500, base: 100, maxIops: 64000, mibPerGib: 1.00, maxMib: 1000, rate: 0.125, iopsRate: 0.006 }, "throughput-optimized": { unitKib: 1024, perGib: 0.5, base: 12, maxIops: 500, mibPerGib: 0.40, maxMib: 1000, rate: 0.045, iopsRate: 0 }, cold: { unitKib: 1024, perGib: 0.2, base: 12, maxIops: 250, mibPerGib: 0.12, maxMib: 250, rate: 0.015, iopsRate: 0 }, }; // Workload profiles: operation size, operations a second, the data it sits on. export const WORKLOAD = [ { name: "meter writes", kib: 16, ops: 4200, dataGib: 200, kind: "random small write" }, { name: "work-order query", kib: 8, ops: 900, dataGib: 60, kind: "random small read" }, { name: "invoice generation", kib: 1024, ops: 180, dataGib: 400, kind: "sequential large read" }, { name: "photo upload", kib: 4096, ops: 25, dataGib: 1200, kind: "sequential large write" }, ]; // An operation rounds up to the type's chunk: a 16 KiB write also spends 1 chunk on a 1024 KiB-chunk type. export const chunks = (w, t) => Math.ceil(w.kib / t.unitKib); export const consumedIops = (w, t) => w.ops * chunks(w, t); export const consumedMib = (w, t) => consumedIops(w, t) * t.unitKib / 1024; export const budget = (t, gib) => ({ iops: Math.min(t.maxIops, Math.max(t.base, Math.round(t.perGib * gib))), mib: Math.min(t.maxMib, t.mibPerGib * gib), }); export const cost = (t, gib, provisionedIops = 0) => t.rate * gib + t.iopsRate * provisionedIops; ``` #### Type and Size Give a Budget ```js // block/budget.mjs — volume type and size give a budget; workloads are applied to that budget import { TYPE, WORKLOAD, chunks, consumedIops, consumedMib, budget, cost } from "./model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); console.log("workload | kind | op size | ops/s | raw MiB/s | data GiB"); console.log("--------------------|-------------------------|----------|-------|-----------|---------"); for (const w of WORKLOAD) console.log(`${w.name.padEnd(19)} | ${w.kind.padEnd(23)} | ${s(w.kib + " KiB", 8)} | ${s(w.ops, 5)} | ` + `${g(w.ops * w.kib / 1024, 9, 1)} | ${s(w.dataGib, 8)}`); const general = TYPE["general purpose"]; console.log("\nbuying a general-purpose volume sized to the data (the natural choice), 16 KiB chunk:"); console.log("workload | budget iops | consumed | utilization | budget MiB/s | consumed | utilization | saturated axis"); console.log("--------------------|-------------|----------|-------------|--------------|----------|-------------|---------------"); for (const w of WORKLOAD) { const b = budget(general, w.dataGib), dIops = consumedIops(w, general) / b.iops, dMib = consumedMib(w, general) / b.mib; console.log(`${w.name.padEnd(19)} | ${s(b.iops, 11)} | ${s(consumedIops(w, general), 8)} | ${g(100 * dIops, 10, 1)}% | ` + `${g(b.mib, 12, 1)} | ${g(consumedMib(w, general), 8, 1)} | ${g(100 * dMib, 10, 1)}% | ` + `${(dIops >= dMib ? "iops" : "MiB/s").padStart(14)}`); } const photo = WORKLOAD[3], pb = budget(general, photo.dataGib); console.log(`idle budget: while ${photo.name} saturates on the iops axis, ` + `${(100 * (1 - consumedMib(photo, general) / pb.mib)).toFixed(1)}% of the MiB/s budget goes unused. Also, an 8 KiB read`); console.log(`consumes a 16 KiB chunk: the work-order query's raw ${(WORKLOAD[1].ops * WORKLOAD[1].kib / 1024).toFixed(1)} MiB/s is ` + `written into the budget as ${consumedMib(WORKLOAD[1], general).toFixed(1)} MiB/s.`); console.log("\nsmallest size needed to satisfy the workload, and monthly fee (fictional unit):"); console.log("workload | type | chunks/op | required GiB | idle GiB | monthly fee"); console.log("--------------------|--------------------------|-----------|--------------|----------|------------"); const cheapest = {}; for (const w of WORKLOAD) for (const [name, t] of Object.entries(TYPE)) { let size = null; for (let gib = w.dataGib; gib <= 65536; gib += 4) { // size is grown until the budget catches up const b = budget(t, gib); if (b.iops >= consumedIops(w, t) && b.mib >= consumedMib(w, t)) { size = gib; break; } } if (name === "performance-provisioned" && consumedMib(w, t) <= t.maxMib && consumedIops(w, t) <= t.maxIops) size = w.dataGib; if (size === null) { console.log(`${w.name.padEnd(19)} | ${name.padEnd(24)} | ${s(chunks(w, t), 9)} | ${"infeasible".padStart(12)} | ` + `${"-".padStart(8)} | ${"-".padStart(11)}`); continue; } const c = cost(t, size, name === "performance-provisioned" ? consumedIops(w, t) : 0); if (!cheapest[w.name] || c < cheapest[w.name].c) cheapest[w.name] = { name, c, size }; console.log(`${w.name.padEnd(19)} | ${name.padEnd(24)} | ${s(chunks(w, t), 9)} | ${s(size, 12)} | ` + `${s(size - w.dataGib, 8)} | ${g(c, 11)}`); } console.log("\ncheapest choice: " + WORKLOAD.map((w) => `${w.name} -> ${cheapest[w.name].name} ${cheapest[w.name].size} GiB ` + `(${cheapest[w.name].c.toFixed(2)})`).join("; ")); ``` ``` workload | kind | op size | ops/s | raw MiB/s | data GiB --------------------|-------------------------|----------|-------|-----------|--------- meter writes | random small write | 16 KiB | 4200 | 65.6 | 200 work-order query | random small read | 8 KiB | 900 | 7.0 | 60 invoice generation | sequential large read | 1024 KiB | 180 | 180.0 | 400 photo upload | sequential large write | 4096 KiB | 25 | 100.0 | 1200 buying a general-purpose volume sized to the data (the natural choice), 16 KiB chunk: workload | budget iops | consumed | utilization | budget MiB/s | consumed | utilization | saturated axis --------------------|-------------|----------|-------------|--------------|----------|-------------|--------------- meter writes | 600 | 4200 | 700.0% | 50.0 | 65.6 | 131.3% | iops work-order query | 180 | 900 | 500.0% | 15.0 | 14.1 | 93.8% | iops invoice generation | 1200 | 11520 | 960.0% | 100.0 | 180.0 | 180.0% | iops photo upload | 3600 | 6400 | 177.8% | 250.0 | 100.0 | 40.0% | iops idle budget: while photo upload saturates on the iops axis, 60.0% of the MiB/s budget goes unused. Also, an 8 KiB read consumes a 16 KiB chunk: the work-order query's raw 7.0 MiB/s is written into the budget as 14.1 MiB/s. smallest size needed to satisfy the workload, and monthly fee (fictional unit): workload | type | chunks/op | required GiB | idle GiB | monthly fee --------------------|--------------------------|-----------|--------------|----------|------------ meter writes | general purpose | 1 | 1400 | 1200 | 112.00 meter writes | performance-provisioned | 1 | 200 | 0 | 50.20 meter writes | throughput-optimized | 1 | infeasible | - | - meter writes | cold | 1 | infeasible | - | - work-order query | general purpose | 1 | 300 | 240 | 24.00 work-order query | performance-provisioned | 1 | 60 | 0 | 12.90 work-order query | throughput-optimized | 1 | infeasible | - | - work-order query | cold | 1 | infeasible | - | - invoice generation | general purpose | 64 | 3840 | 3440 | 307.20 invoice generation | performance-provisioned | 64 | 400 | 0 | 119.12 invoice generation | throughput-optimized | 1 | 452 | 52 | 20.34 invoice generation | cold | 1 | 1500 | 1100 | 22.50 photo upload | general purpose | 256 | 2136 | 936 | 170.88 photo upload | performance-provisioned | 256 | 1200 | 0 | 188.40 photo upload | throughput-optimized | 4 | 1200 | 0 | 54.00 photo upload | cold | 4 | 1200 | 0 | 18.00 cheapest choice: meter writes -> performance-provisioned 200 GiB (50.20); work-order query -> performance-provisioned 60 GiB (12.90); invoice generation -> throughput-optimized 452 GiB (20.34); photo upload -> cold 1200 GiB (18.00) ``` The second table gives the saturation points. Buying a volume sized to the data — the natural choice — falls short on all four workloads: utilization runs from 177.8% to 960%. The saturated axis is iops in all four, but the reasons differ. Meter writes truly perform a great many operations; invoice generation performs only 180 operations a second, but since each operation spends 64 chunks, the budget records 11,520 iops. **Chunk size changes how a workload looks in the budget.** Idle budget is the other face of the same table. While photo upload saturates on the iops axis, 60% of the MiB/s budget is never spent. There is a loss in the reverse direction too: an 8 KiB work-order read consumes a full 16 KiB chunk, so the raw 7.0 MiB/s is recorded in the budget as 14.1 MiB/s — a direct consequence of the operation being smaller than the chunk. The third table reduces type choice to a calculation. Sequential large reads cost 15x less on types that count 1 MiB chunks: invoice generation costs 307.20 on the general-purpose volume, 20.34 on the throughput-optimized volume. The same types, though, **cannot satisfy** random small writes — a 16 KiB write spends a full 1 MiB chunk, so 4,200 operations a second records 4,200 MiB/s of consumption. The wrong type here is not expensive, it is infeasible. #### Size and Performance Being Tied Together - **ST22** — On the performance-provisioned type, iops is billed separately; on the other types the budget depends only on size. - **ST23** — A volume can grow but cannot shrink; the copy speed in the model is 250 MiB/s. - **ST24** — A volume is bound to a single instance and a single region; the snapshot compression ratio is 0.55. ```js // block/trap.mjs — size and performance being tied together, no shrinking, and single-instance binding are counted import { TYPE, WORKLOAD, consumedIops, budget, cost } from "./model.mjs"; const general = TYPE["general purpose"], meter = WORKLOAD[0]; const EGRESS = { crossRegion: 0.020, internet: 0.085 }; const COMPRESSION = 0.55, COPY_MIB = 250, ATTACH_SEC = 40, COLLECTORS = 12, WRITERS = 1; const g = (x, d = 2) => x.toFixed(d); const requiredSize = (iops) => { // size is grown until the budget catches up for (let gib = 4; gib <= 65536; gib += 4) if (budget(general, gib).iops >= iops) return gib; }; const INITIAL = requiredSize(consumedIops(meter, general)), DROPPED = 900, NEW_SIZE = requiredSize(DROPPED); console.log(`${meter.name}: data ${meter.dataGib} GiB, required ${consumedIops(meter, general)} iops.`); console.log(` on a general-purpose volume, iops is tied to size, so ${INITIAL} GiB must be bought: ` + `${INITIAL - meter.dataGib} GiB stands there not for data but for budget (${g(cost(general, INITIAL))} units a month).`); console.log(` the same budget is bought on the performance-provisioned type with ${meter.dataGib} GiB: ` + `${g(cost(TYPE["performance-provisioned"], meter.dataGib, consumedIops(meter, general)))} units a month.`); console.log(`\nthe workload dropped from ${consumedIops(meter, general)} iops to ${DROPPED} iops; now ${NEW_SIZE} GiB is enough.`); console.log(` growing is free: the ${INITIAL} GiB volume can be grown without moving data.`); console.log(` no shrinking: a new ${NEW_SIZE} GiB volume is opened, ${meter.dataGib} GiB is copied, the connection is swapped.`); const copySec = meter.dataGib * 1024 / COPY_MIB; console.log(` copying takes ${g(copySec / 60, 1)} min (${COPY_MIB} MiB/s), and both volumes are billed at once during the move: ` + `${g(cost(general, INITIAL) + cost(general, NEW_SIZE))} units/month.`); console.log(` the extra paid for not being able to shrink: ${g(cost(general, INITIAL) - cost(general, NEW_SIZE))} a month, ` + `${g(12 * (cost(general, INITIAL) - cost(general, NEW_SIZE)))} units a year.`); const snapshot = meter.dataGib * COMPRESSION; console.log(`\na volume is bound to a single instance and a single region: only ${WRITERS} of ${COLLECTORS} collector instances can write at once.`); console.log(` handing it to another instance means detaching and attaching; in the model, attaching takes ${ATTACH_SEC} s with no writer during that time.`); console.log(` moving it to another region needs a snapshot: ${g(snapshot, 1)} GiB (compression ${COMPRESSION}); ` + `cross-region egress ${g(snapshot * EGRESS.crossRegion, 2)}, to the internet ${g(snapshot * EGRESS.internet, 2)} units.`); console.log(` an in-region snapshot incurs no egress fee; taking the volume out of the region does.`); ``` ``` meter writes: data 200 GiB, required 4200 iops. on a general-purpose volume, iops is tied to size, so 1400 GiB must be bought: 1200 GiB stands there not for data but for budget (112.00 units a month). the same budget is bought on the performance-provisioned type with 200 GiB: 50.20 units a month. the workload dropped from 4200 iops to 900 iops; now 300 GiB is enough. growing is free: the 1400 GiB volume can be grown without moving data. no shrinking: a new 300 GiB volume is opened, 200 GiB is copied, the connection is swapped. copying takes 13.7 min (250 MiB/s), and both volumes are billed at once during the move: 136.00 units/month. the extra paid for not being able to shrink: 88.00 a month, 1056.00 units a year. a volume is bound to a single instance and a single region: only 1 of 12 collector instances can write at once. handing it to another instance means detaching and attaching; in the model, attaching takes 40 s with no writer during that time. moving it to another region needs a snapshot: 110.0 GiB (compression 0.55); cross-region egress 2.20, to the internet 9.35 units. an in-region snapshot incurs no egress fee; taking the volume out of the region does. ``` The trap stands in the first lines. Because iops is tied to size on the general-purpose type, 200 GiB of data demands a 1,400 GiB volume: 1,200 GiB stands there not for data but for **budget**. The same budget is bought at 200 GiB for 50.20 units on the type that sells iops separately — buying capacity for performance means paying for empty bytes again every month. The second block gives the unrecoverable item. When the workload drops from 4,200 iops to 900, 300 GiB is enough — but **the volume cannot shrink**. Growing happens without moving data; shrinking means opening a new volume, copying 200 GiB, and swapping the connection — 13.7 minutes in the model, double billing during the move, and 1,056 units overpaid a year. Size is a one-way dial. The third block writes the type's own limit. A volume is bound to a single instance: only one of twelve collector instances can write at once; handing it to another instance is a matter of detaching and attaching, and in the model there is no writer at all for forty seconds. A volume is also bound to a single region, and moving it requires a snapshot — 110 GiB, 2.20 units cross-region, 9.35 units to the internet. A snapshot that stays inside the region incurs no egress fee. #### Summary - Type and size together give an input/output budget; the budget is spent on the iops and MiB/s axes, and the workload saturates on whichever it fills first. - An operation rounds up to the volume's chunk: a 1 MiB read spends 64 chunks on a 16 KiB chunk; an 8 KiB read's raw 7.0 MiB/s shows up in the budget as 14.1 MiB/s. - Idle budget is measured: while photo upload saturates on the iops axis, 60% of the MiB/s budget goes unused. The wrong type is sometimes not expensive but infeasible. - Size and performance being tied together is a trap: a 1,400 GiB volume is bought for 200 GiB of data, and 1,200 GiB stands there for budget. The same budget is 50.20 units instead of 112.00 on the type that sells iops separately. - The unrecoverable item is size: a volume can grow but cannot shrink. Shrinking means a new volume, a 13.7-minute copy, and double billing. #### Next Step The last block left a door open: only one of twelve collector instances can write at once. In the fictional measurement network this is a direct obstacle — field teams drop work-order attachments into the same directory, invoice generation reads from the same directory, and all of it runs on separate instances. Block storage cannot satisfy this, and object storage offers no directory or lock. The next lesson measures a third type: a shared file system that multiple instances mount at the same time on the same data. Three things will be measured — how multi-writer access changes across the three types, how long the lock makes things wait, and the workload where a shared file system gains nothing at all. ### File Storage Source: https://academia.sh/en/courses/cloud-storage/file-storage Measuring the shared file system under multi-writer access: keeping the same data as object, block, and shared file and counting how many instances can write at once, how the lock works, latency, and monthly fee; naming the case where the shared file system gains nothing, and working out the access protocol as an unrecoverable item. The previous lesson ended at a door: a block volume is bound to a single instance, and only one of twelve collector instances can write at once. In the fictional measurement network this is not a detail but a direct obstacle — field teams drop work-order attachments into the same files, invoice generation reads from the same directory, and all of it runs on separate instances. Object storage accepts multiple writers but offers no lock or partial update; a block volume offers both but allows only a single writer. A third type fills this gap: the **shared file system**. It connects over the network, multiple instances mount it at once, and it offers a full file system interface — directory, rename, partial write, and byte-range lock. What is measured is how much of this is real gain: the same data can be kept on all three types, and the difference among them only shows up under multi-writer access. - **ST25** — Fictional workload: twelve field instances, 340 appends a second, each append 2 KiB, 4,800 active work-order files. Which file a write lands on is set by the generator in the model, seed 71549. - **ST26** — Latencies are a model: an object storage request 25 ms, a block volume 0.6 ms, a shared file system 2.5 ms; the network round trip from a remote instance to the single writer 0.4 ms. - **ST27** — On object storage the read-modify-write window is 50 ms; a second write landing on the same file within that window overwrites the first. ```js // file/model.mjs — three storage types, a multi-writer workload, and fees are a MODEL. Fees are a // fictional unit; types are called by access form, not product name. export const INSTANCES = 12, WRITES_PER_SEC = 340, APPEND_KIB = 2, FILE_KIB = 11, ACTIVE = 4800; export const MONTH_SEC = 2_592_000; // a 30-day month, in seconds export const TYPE = { "object storage": { writers: INSTANCES, lock: "none", latencyMs: 25, storage: 0.023, request: 0.0050 / 1000, throughput: 0 }, "block volume": { writers: 1, lock: "single writer", latencyMs: 0.6, storage: 0.080, request: 0, throughput: 0 }, "shared file": { writers: INSTANCES, lock: "byte range", latencyMs: 2.5, storage: 0.300, request: 0, throughput: 0.60 }, }; export const NETWORK_RTT_MS = 0.4; // the round trip from a remote instance to the single writer export const COLLISION_MS = 50; // object storage's read-modify-write window export const gib = (bytes) => bytes / 1024 ** 3; // Which file a write lands on is a MODEL: the generator is here, the seed is visible. The distribution is skewed. export const SEED = 71549; export function* writes(count) { let x = SEED >>> 0; const u = () => (x = (Math.imul(x, 1664525) + 1013904223) >>> 0) / 4294967296; for (let i = 0; i < count; i++) yield { t: (i / WRITES_PER_SEC) * 1000, file: Math.floor(ACTIVE * Math.pow(u(), 3)) }; } // Single-server queue: mean wait with a fixed service time (utilization p, service S). export const wait = (p, S) => (p >= 1 ? Infinity : (p / (2 * (1 - p))) * S); ``` #### Same Data, Three Types, Multiple Writers ```js // file/multi-writer.mjs — the same data on three types: how many writers, how the lock works, latency and fee import { INSTANCES, WRITES_PER_SEC, APPEND_KIB, FILE_KIB, ACTIVE, MONTH_SEC, TYPE, NETWORK_RTT_MS, COLLISION_MS, gib, SEED, writes, wait } from "./model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); const DURATION_SEC = 60, COUNT = WRITES_PER_SEC * DURATION_SEC; // 1) Object storage: no lock. A second write landing on the same file within COLLISION_MS overwrites the first. const lastWrite = new Map(); let lost = 0, hottest = 0; const counts = new Map(); for (const w of writes(COUNT)) { const previous = lastWrite.get(w.file); if (previous !== undefined && w.t - previous < COLLISION_MS) lost++; lastWrite.set(w.file, w.t); counts.set(w.file, (counts.get(w.file) ?? 0) + 1); } for (const n of counts.values()) hottest = Math.max(hottest, n); const hottestRate = hottest / DURATION_SEC; // 2) Block volume: single writer. The others send over the network to that instance, forming a queue there. const S_BLOCK = TYPE["block volume"].latencyMs, pBlock = WRITES_PER_SEC * S_BLOCK / 1000; const blockLatency = NETWORK_RTT_MS + S_BLOCK + wait(pBlock, S_BLOCK); // 3) Shared file system: a byte-range lock per file; the queue forms only within the same file. const S_SHARED = TYPE["shared file"].latencyMs, pHottest = hottestRate * S_SHARED / 1000; const sharedLatency = S_SHARED + wait(pHottest, S_SHARED); const dataGib = gib(ACTIVE * FILE_KIB * 1024); const monthlyWrites = WRITES_PER_SEC * MONTH_SEC; const FEE = { "object storage": TYPE["object storage"].storage * dataGib + monthlyWrites * TYPE["object storage"].request, "block volume": TYPE["block volume"].storage * Math.ceil(WRITES_PER_SEC / 3 / 4) * 4, // iops budget demands size "shared file": TYPE["shared file"].storage * dataGib + TYPE["shared file"].throughput * (WRITES_PER_SEC * APPEND_KIB / 1024), }; console.log(`${INSTANCES} field instances, ${WRITES_PER_SEC} appends/s (${APPEND_KIB} KiB), ${ACTIVE} active work-order files ` + `(${FILE_KIB} KiB). Distribution skewed, generator in the model, seed ${SEED}. ${DURATION_SEC} s run.`); console.log(`the hottest file took ${hottest} writes in ${DURATION_SEC} s: ${g(hottestRate, 0, 1)} writes/s.`); console.log("\ntype | writers at once | lock | latency (ms) | lost update | monthly fee"); console.log("-----------------|-----------------|----------------|--------------|--------------|------------"); const LATENCY = { "object storage": TYPE["object storage"].latencyMs, "block volume": blockLatency, "shared file": sharedLatency }; const LOST = { "object storage": lost, "block volume": 0, "shared file": 0 }; for (const [name, t] of Object.entries(TYPE)) console.log(`${name.padEnd(16)} | ${s(t.writers, 15)} | ${t.lock.padEnd(14)} | ${g(LATENCY[name], 12, 2)} | ` + `${s(LOST[name], 12)} | ${g(FEE[name], 11)}`); console.log(`on object storage, ${lost} of the ${COUNT} writes (${g(100 * lost / COUNT, 0, 2)}%) were lost silently: ` + `no lock, last writer wins.`); console.log(`on the block volume, 11 instances cannot write directly; the single writer's utilization is ${g(100 * pBlock, 0, 1)}%, ` + `wait ${g(wait(pBlock, S_BLOCK), 0, 3)} ms. If that instance drops, ${INSTANCES}/${INSTANCES} writers stop.`); console.log(`on the shared file system, the hottest file's utilization is ${g(100 * pHottest, 0, 2)}%, ` + `lock wait ${g(wait(pHottest, S_SHARED), 0, 4)} ms; the lock is per file, not per bucket.`); // The case it gains nothing: single writer, large sequential read. Same data, same job, fee on three types. const INVOICE = { dataGib: 400, mibS: 180 }; const COST = { "object storage": TYPE["object storage"].storage * INVOICE.dataGib, "block volume": 0.045 * 452, // throughput-optimized volume, computed in the previous lesson "shared file": TYPE["shared file"].storage * INVOICE.dataGib + TYPE["shared file"].throughput * INVOICE.mibS, }; console.log(`\nthe case it gains nothing — invoice generation: single writer, ${INVOICE.mibS} MiB/s sequential read, ${INVOICE.dataGib} GiB:`); for (const [name, u] of Object.entries(COST)) console.log(` ${name.padEnd(16)} ${g(u, 8)} units/month${name === "shared file" ? ` — ${g(u / Math.min(...Object.values(COST)), 0, 1)}x the cheapest, the only thing it gains is an unused lock` : ""}`); ``` ``` 12 field instances, 340 appends/s (2 KiB), 4800 active work-order files (11 KiB). Distribution skewed, generator in the model, seed 71549. 60 s run. the hottest file took 1168 writes in 60 s: 19.5 writes/s. type | writers at once | lock | latency (ms) | lost update | monthly fee -----------------|-----------------|----------------|--------------|--------------|------------ object storage | 12 | none | 25.00 | 1057 | 4406.40 block volume | 1 | single writer | 1.08 | 0 | 9.28 shared file | 12 | byte range | 2.56 | 0 | 0.41 on object storage, 1057 of the 20400 writes (5.18%) were lost silently: no lock, last writer wins. on the block volume, 11 instances cannot write directly; the single writer's utilization is 20.4%, wait 0.077 ms. If that instance drops, 12/12 writers stop. on the shared file system, the hottest file's utilization is 4.87%, lock wait 0.0639 ms; the lock is per file, not per bucket. the case it gains nothing — invoice generation: single writer, 180 MiB/s sequential read, 400 GiB: object storage 9.20 units/month block volume 20.34 units/month shared file 228.00 units/month — 24.8x the cheapest, the only thing it gains is an unused lock ``` The table separates the three types on a single screen. Object storage accepts twelve writers but has no lock: 1,057 of 20,400 writes in sixty seconds, 5.18%, are lost silently. No error is returned, no trace is left in a log, the field team's note is nowhere to be found. Its monthly fee is also the highest — 4,406.40 units, almost all from write requests: small, frequent writes are object storage's most expensive use. A block volume gives lock and partial write, its latency is the lowest (1.08 ms), its fee 9.28 units — but its writer count is one. Eleven instances cannot write directly, they send their writes to the single instance. That instance's utilization is 20.4%, comfortable today; the real item is not in performance but in structure: when that instance goes down, all twelve writers stop. The single-writer funnel is not a performance constraint, it is an availability item. The shared file system gives both at once: twelve writers, a byte-range lock, zero lost updates. The lock is per file, not per bucket — even the hottest file only takes 19.5 writes a second, its utilization is 4.87%, and the mean lock wait is 0.064 ms. Its latency is more than double a block volume's (2.56 ms against 1.08 ms), because every operation goes through the network. The last block writes the type's limit by name. Invoice generation is single-writer and does a large sequential read; it needs no lock, no directory tree. On the shared file system it costs 228.00 units a month — 24.8 times the cheapest option. **A shared file system is bought for multi-writer access; on a single-writer large sequential read, the only thing it gains is an unused lock.** #### The Access Protocol Is an Unrecoverable Item - **ST28** — The fictional application's file access points and how many sites each appears at are written into the model. - **ST29** — A shared file system is bound to a region; copying data elsewhere incurs an egress fee. - **ST30** — Fees are a fictional unit; a shared file system bills used size and throughput, a block volume bills allocated size. ```js // file/protocol.mjs — the access protocol is an unrecoverable item: call sites are counted (MODEL) import { ACTIVE, FILE_KIB, gib } from "./model.mjs"; // The fictional application's file access points: call, how many sites it appears at, object-model equivalent. const POINT = [ ["read the whole file", 9, "yes", "full object get"], ["read a range of the file", 6, "yes", "range request"], ["write the whole file", 5, "yes", "full object put"], ["write into the middle of the file", 7, "no", "no partial update, the object is rewritten from scratch"], ["append to the file", 8, "no", "no append, the object is rewritten from scratch"], ["truncate the file", 2, "no", "no equivalent"], ["list the directory", 4, "partial", "prefix listing, paginated and billed"], ["read the directory's size", 2, "no", "every key is scanned"], ["rename", 3, "no", "copy and delete"], ["acquire a byte-range lock", 5, "no", "no lock"], ["force the change to disk", 3, "no", "a write is already all-or-nothing"], ]; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); const total = POINT.reduce((a, n) => a + n[1], 0); const direct = POINT.filter((n) => n[2] === "yes").reduce((a, n) => a + n[1], 0); const partial = POINT.filter((n) => n[2] === "partial").reduce((a, n) => a + n[1], 0); console.log("call | sites | object model | equivalent"); console.log("-----------------------------------|-------|--------------|--------------------------------------"); for (const [name, sites, status, equivalent] of POINT) console.log(`${name.padEnd(34)} | ${s(sites, 5)} | ${status.padEnd(12)} | ${equivalent}`); console.log(`\n${total} call sites in total: ${direct} carry over directly, ${partial} carry over by changing ` + `behavior, ${total - direct - partial} have to be rewritten.`); console.log(`the share that must be rewritten is ${g(100 * (total - direct - partial) / total, 0, 1)}%; this number ` + `is independent of data size — even at ${gib(ACTIVE * FILE_KIB * 1024).toFixed(3)} GiB the same ${total - direct - partial} points are rewritten.`); // Egress and transfer: the shared file system is bound to a region. const EGRESS = { "in-region": 0, "cross-region": 0.020, "to the internet": 0.085 }; const dataGib = gib(ACTIVE * FILE_KIB * 1024), ARCHIVE_GIB = 640; console.log(`\negress: if a ${ARCHIVE_GIB} GiB field archive is read off the shared file system and copied elsewhere`); for (const [where, rate] of Object.entries(EGRESS)) console.log(` ${where.padEnd(16)} ${g(ARCHIVE_GIB * rate, 7)} units`); console.log(`the active work-order data is ${dataGib.toFixed(3)} GiB; the egress fee is measured not by the data's ` + `size but by where it moves to.`); ``` ``` call | sites | object model | equivalent -----------------------------------|-------|--------------|-------------------------------------- read the whole file | 9 | yes | full object get read a range of the file | 6 | yes | range request write the whole file | 5 | yes | full object put write into the middle of the file | 7 | no | no partial update, the object is rewritten from scratch append to the file | 8 | no | no append, the object is rewritten from scratch truncate the file | 2 | no | no equivalent list the directory | 4 | partial | prefix listing, paginated and billed read the directory's size | 2 | no | every key is scanned rename | 3 | no | copy and delete acquire a byte-range lock | 5 | no | no lock force the change to disk | 3 | no | a write is already all-or-nothing 54 call sites in total: 20 carry over directly, 4 carry over by changing behavior, 30 have to be rewritten. the share that must be rewritten is 55.6%; this number is independent of data size — even at 0.050 GiB the same 30 points are rewritten. egress: if a 640 GiB field archive is read off the shared file system and copied elsewhere in-region 0.00 units cross-region 12.80 units to the internet 54.40 units the active work-order data is 0.050 GiB; the egress fee is measured not by the data's size but by where it moves to. ``` When a shared file system is chosen, the delegated-decision list is short: the provider takes over the metadata server, replication, and growth. What stays holds one more thing, and it does not show up in the billing column — the **access protocol**. The application is written against file paths, a directory tree, and locks. This is not a configuration, it is a shape of code. The table gives its measure. Of fifty-four access points, twenty carry over directly to the object model, four carry over by changing behavior, thirty must be rewritten: 55.6%. This number is **independent of data size** — even at 0.050 GiB of work-order data the same thirty points are rewritten. Key layout was an unrecoverable item because the data had to be moved; the access protocol is an unrecoverable item because the **application** has to be moved, and that move's cost does not shrink with the data. The egress side closes out the third item. A shared file system is bound to a single region; reading a 640 GiB field archive within the region incurs no egress fee, moving it cross-region costs 12.80 units, downloading it to the internet costs 54.40 units. The same rule has held since this course's first lesson: data is free coming in, billed going out, and its fee is measured by where it goes. #### Summary - The three types diverge under multi-writer access: object storage takes twelve writers but has no lock, a block volume gives the lock but takes only one writer, a shared file system gives both at once. - Lockless access costs silently: 1,057 of 20,400 writes in sixty seconds, 5.18%, are lost with no error returned. Under the same workload, object storage is the most expensive option at 4,406.40 units. - The single-writer funnel is not a performance constraint but an availability item: utilization sits comfortably at 20.4%, but when that instance goes down, all twelve writers stop. - The lock is per file: the hottest file takes 19.5 writes a second, its utilization is 4.87%, lock wait is 0.064 ms. Latency is more than double a block volume's (2.56 ms against 1.08 ms), because every operation goes through the network. - The case where it gains nothing is named: single-writer large sequential reads cost 228.00 units, 24.8 times the cheapest option. The unrecoverable item is the access protocol — thirty of fifty-four call sites are rewritten regardless of data size. #### Next Step Five lessons leave one gap. The fictional measurement network's field photos are written once and almost never read; in the distribution measured in the second lesson, 97% of this family was never opened all year. For such data the frequent class is an obvious waste, but what the archive tier is has not yet been measured: its storage fee is nearly nothing, and in exchange the data **cannot be read right away**. The next lesson counts this trade-off — retrieval speed options are scanned, time and fee are worked out together for each, the hours an audit request waits are computed, and what the minimum retention period makes an early-deleted object pay is shown. ### Archive Storage Source: https://academia.sh/en/courses/cloud-storage/archive-storage How the archive tier's storage fee drops while retrieval is both billed and delayed: scanning three retrieval speeds along with their time and fee, working out how many hours an audit request waits, the egress fee stacking on top of retrieval, and the minimum retention period making an early-deleted object pay the full period. When access history was scanned in the second lesson, one family stood apart: 97% of field photos were never opened all year. For such data the frequent class is an obvious waste, and the third lesson's lifecycle rule can send it to archive automatically. What data sent to archive actually becomes has not yet been measured. The archive tier is a storage class, but it departs from the others at one point: data **cannot be read right away**. Retrieving an object first requires a retrieval request, the request is processed, and only then does the object become readable. This wait ranges from minutes to hours, and it can be sped up — for a fee. The wait is not the rule for every archive class: some providers' cheapest class is read in milliseconds and differs only in fee and minimum storage duration; this lesson measures the one with the wait. This is the lesson's measure: where what was gained in storage gets paid back, and in which item. - **ST31** — The archive tier, retrieval speeds, and fees are fictional; tiers are called by access frequency. - **ST32** — There are three retrieval speeds; each one's first-response latency, streaming rate, byte fee, and request fee are written into the model. - **ST33** — A retrieval request takes at most 10,000 objects, and four batches are processed at once. ```js // archive/model.mjs — the archive tier, retrieval speeds, and the audit request are a MODEL. Fees are // a fictional unit; tiers are called by access frequency, no commercial name is used. export const CLASS = { // storage unit/GiB/month, transition unit/request, minDays: minimum retention frequent: { storage: 0.0230, transition: 0.0050 / 1000, minDays: 0 }, archive: { storage: 0.0010, transition: 0.0500 / 1000, minDays: 180 }, }; // Retrieval speeds: first-response latency, streaming rate, byte and request fee. export const SPEED = { slow: { firstHour: 12, gibPerHour: 250, perGib: 0.0025, request: 0.030 / 1000 }, medium: { firstHour: 4, gibPerHour: 600, perGib: 0.0100, request: 0.060 / 1000 }, fast: { firstHour: 5 / 60, gibPerHour: 1500, perGib: 0.0300, request: 0.120 / 1000 }, }; export const BATCH = 10_000, CONCURRENT = 4; // 10,000 objects a request, 4 batches at once export const EGRESS = { "in-region": 0, "cross-region": 0.020, internet: 0.085 }; // Fictional audit request: one province's three years of invoice documents and field photos. export const AUDIT = { invoice: { count: 46_000, kib: 380 }, photo: { count: 80_000, kib: 3686 } }; export const gib = (bytes) => bytes / 1024 ** 3; export const auditBytes = Object.values(AUDIT).reduce((a, k) => a + k.count * k.kib * 1024, 0); export const auditCount = Object.values(AUDIT).reduce((a, k) => a + k.count, 0); ``` #### Time and Fee, Two Ends of the Same Dial - **ST34** — The fictional audit request covers one province's three years of invoice documents and field photos; counts and sizes are written into the model. ```js // archive/retrieval.mjs — retrieval speeds are scanned: time and fee are counted together import { SPEED, BATCH, CONCURRENT, EGRESS, gib, auditBytes, auditCount, AUDIT } from "./model.mjs"; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); const auditGib = gib(auditBytes), batches = Math.ceil(auditCount / BATCH), rounds = Math.ceil(batches / CONCURRENT); console.log(`fictional audit request: ${Object.entries(AUDIT).map(([a, k]) => `${k.count} ${a}`).join(" + ")} = ` + `${auditCount.toLocaleString("en-US")} objects, ${g(auditGib, 0, 1)} GiB.`); console.log(`the request is split into ${BATCH}-object batches: ${batches} batches, ${CONCURRENT} at once, ${rounds} rounds.`); console.log("\nspeed | first resp | stream GiB/hr | one batch (hr) | total wait (hr) | retrieval fee"); console.log("-------|------------|---------------|----------------|------------------|--------------"); const FEE = {}, WAIT = {}; for (const [name, sp] of Object.entries(SPEED)) { const batchGib = auditGib / batches, roundHours = sp.firstHour + (batchGib * CONCURRENT) / sp.gibPerHour; const fee = FEE[name] = auditGib * sp.perGib + auditCount * sp.request; WAIT[name] = rounds * roundHours; console.log(`${name.padEnd(6)} | ${g(sp.firstHour, 10, 2)} | ${s(sp.gibPerHour, 13)} | ${g(sp.firstHour + batchGib / sp.gibPerHour, 14, 2)} | ` + `${g(WAIT[name], 16, 1)} | ${g(fee, 13, 2)}`); } console.log(`same request: wait shrinks ${(WAIT.slow / WAIT.fast).toFixed(0)}x, fee ` + `grows ${(FEE.fast / FEE.slow).toFixed(1)}x. Time and fee are two ends of the same dial.`); console.log(`the request fee dominates: ${auditCount.toLocaleString("en-US")} objects at slow speed cost ` + `${g(auditCount * SPEED.slow.request, 0, 2)} units, the byte fee for the same bytes is ${g(auditGib * SPEED.slow.perGib, 0, 2)} units.`); console.log("\nonce retrieval finishes, egress is paid separately (same bytes):"); for (const [where, rate] of Object.entries(EGRESS)) console.log(` ${where.padEnd(15)} ${g(auditGib * rate, 7)} units`); console.log(`downloading to the internet costs ${g(auditGib * EGRESS.internet / FEE.slow, 0, 1)}x the cheapest retrieval: ` + `archive's cheapness only lasts until the data leaves.`); // Decision: the cheapest speed that meets a given response-time commitment. What narrowing the request changes. const solve = (count, bytes, limitHours) => { const G = gib(bytes), p = Math.ceil(count / BATCH), t = Math.ceil(p / CONCURRENT); let best = null; for (const [name, sp] of Object.entries(SPEED)) { const hours = t * (sp.firstHour + (G / p * CONCURRENT) / sp.gibPerHour), fee = G * sp.perGib + count * sp.request; if (hours <= limitHours && (best === null || fee < best.fee)) best = { name, hours, fee }; } return best; }; console.log("\ncheapest speed that meets a response-time commitment:"); console.log("commitment | full request (126,000 objects) | invoices only (46,000 objects)"); console.log("-----------|-----------------------------------|-------------------------------"); for (const limit of [1, 6, 24, 72]) { const a = solve(auditCount, auditBytes, limit); const f = solve(AUDIT.invoice.count, AUDIT.invoice.count * AUDIT.invoice.kib * 1024, limit); const format = (x) => x ? `${x.name.padEnd(6)} ${g(x.hours, 5, 1)} hr ${g(x.fee, 7)} units` : "infeasible"; console.log(`${s(limit + " hr", 10)} | ${format(a).padEnd(33)} | ${format(f)}`); } ``` ``` fictional audit request: 46000 invoice + 80000 photo = 126,000 objects, 297.9 GiB. the request is split into 10000-object batches: 13 batches, 4 at once, 4 rounds. speed | first resp | stream GiB/hr | one batch (hr) | total wait (hr) | retrieval fee -------|------------|---------------|----------------|------------------|-------------- slow | 12.00 | 250 | 12.09 | 49.5 | 4.52 medium | 4.00 | 600 | 4.04 | 16.6 | 10.54 fast | 0.08 | 1500 | 0.10 | 0.6 | 24.06 same request: wait shrinks 86x, fee grows 5.3x. Time and fee are two ends of the same dial. the request fee dominates: 126,000 objects at slow speed cost 3.78 units, the byte fee for the same bytes is 0.74 units. once retrieval finishes, egress is paid separately (same bytes): in-region 0.00 units cross-region 5.96 units internet 25.32 units downloading to the internet costs 5.6x the cheapest retrieval: archive's cheapness only lasts until the data leaves. cheapest speed that meets a response-time commitment: commitment | full request (126,000 objects) | invoices only (46,000 objects) -----------|-----------------------------------|------------------------------- 1 hr | fast 0.6 hr 24.06 units | fast 0.2 hr 6.02 units 6 hr | fast 0.6 hr 24.06 units | fast 0.2 hr 6.02 units 24 hr | medium 16.6 hr 10.54 units | medium 8.0 hr 2.93 units 72 hr | slow 49.5 hr 4.52 units | slow 24.1 hr 1.42 units ``` The audit request is 126,000 objects and 297.9 GiB. None of the three speeds is wrong; their difference collects into two columns. At slow speed the total wait is 49.5 hours — more than two days — and the retrieval fee is 4.52 units. At fast speed the wait drops to 0.6 hours, the fee rises to 24.06 units. The wait shrinks 86 times, the fee grows 5.3 times. In an audit request the real decision is not which speed is "good," it is **how many hours of waiting is acceptable**; without that answer, no speed can be chosen. The reason the total wait exceeds the first-response latency is the batch calculation. The request is split into 10,000-object batches: 126,000 objects makes 13 batches, four batches are processed at once, so four rounds are needed. At slow speed a single batch takes 12.09 hours, but the total is 49.5 hours. **First-response latency is paid again on every round**, and as object count grows the wait grows not with bytes but with rounds. The line below shows the price side of the same structure: 126,000 objects' request fee at slow speed is 3.78 units, while the same bytes' byte fee is 0.74 units. Four-fifths of the retrieval bill comes **from object count**. This is archive's counterpart to the rule from the first and second lessons — small, numerous objects cost through the request fee at every tier. Egress is a separate item. Retrieval makes data readable; it does not **move** it anywhere. The same 297.9 GiB incurs no egress fee if it stays in the region, costs 5.96 units across regions, and 25.32 units downloaded to the internet. Downloading to the internet is 5.6 times the cheapest retrieval: archive's cheapness lasts only until the data leaves. The last table ties the decision to a response-time commitment and reduces the choice to a single line. At a one-hour commitment only the fast speed suffices, costing 24.06 units; at a twenty-four-hour commitment the medium speed drops to 10.54 units; at a seventy-two-hour commitment the slow speed suffices at 4.52 units. The right column opens a second lever: when the audit request is narrowed to only invoice documents, both time and fee drop at every commitment — 2.93 units instead of 10.54 units at the twenty-four-hour commitment. **Narrowing the request is cheaper than raising the speed;** the first thing to measure in archive is what actually needs to be retrieved. #### Minimum Retention Period - **ST35** — The archive tier's minimum retention period is 180 days, and it starts the moment the object enters the class. - **ST36** — The lifecycle rule transitions the object to archive on day 30 and deletes it when its lifetime ends. ```js // archive/minimum.mjs — minimum retention: an object deleted early is still billed for the full period import { CLASS, gib } from "./model.mjs"; const DAYS_PER_MONTH = 30, archive = CLASS.archive, frequent = CLASS.frequent; const minMonths = archive.minDays / DAYS_PER_MONTH; // Two families, two lifetimes: a small, short-lived document and a large, long-lived photo. const FAMILIES = [ { name: "invoice-documents", count: 86_000, kib: 380, lifeMonths: 3 }, { name: "field-photos", count: 42_000, kib: 3686, lifeMonths: 60 }, ]; const TRANSITION_MONTH = 1; // the rule moves to archive on day 30 const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); console.log(`the archive tier's minimum retention period is ${archive.minDays} days = ${minMonths} months. ` + `The rule transitions to archive in month ${TRANSITION_MONTH}, deletes when the lifetime ends.`); console.log("\nfamily | life | in archive | billed | transition fee | no archive | archive | ratio"); console.log("------------------|-------|------------|--------|-----------------|------------|---------|------"); for (const f of FAMILIES) { const dataGib = gib(f.count * f.kib * 1024), inArchive = f.lifeMonths - TRANSITION_MONTH; const billed = Math.max(inArchive, minMonths); const transitionFee = f.count * archive.transition; const withoutArchive = dataGib * frequent.storage * f.lifeMonths; const withArchive = dataGib * frequent.storage * TRANSITION_MONTH + transitionFee + dataGib * archive.storage * billed; console.log(`${f.name.padEnd(17)} | ${s(f.lifeMonths + " mo", 5)} | ${s(inArchive + " mo", 10)} | ${s(billed + " mo", 6)} | ` + `${g(transitionFee, 15)} | ${g(withoutArchive, 10)} | ${g(withArchive, 7)} | ${g(withArchive / withoutArchive, 5, 2)}`); } // Early deletion: even if the object is deleted on day 45, the 180-day debt stands. const photo = FAMILIES[1], dataGib = gib(photo.count * photo.kib * 1024), DELETE_DAY = 45; const used = DELETE_DAY / DAYS_PER_MONTH, owed = minMonths - used; console.log(`\nearly deletion: ${photo.count.toLocaleString("en-US")} photos moved to archive, deleted on day ${DELETE_DAY}.`); console.log(` used ${g(used, 0, 1)} months, billed ${minMonths} months, paid for ${g(owed, 0, 1)} idle months.`); console.log(` the bill for deleting on day ${DELETE_DAY} is ${g(dataGib * archive.storage * minMonths, 0, 3)} units; ` + `never deleting and holding it for ${minMonths} months bills the same ${g(dataGib * archive.storage * minMonths, 0, 3)} units.`); console.log(` the ${g(owed, 0, 1)} months in between, ${g(dataGib * archive.storage * owed, 0, 3)} units, are paid for objects ` + `that no longer exist: early deletion gains nothing.`); console.log(`archive is not a storage class for data that will live under ${minMonths} months, it is a debt paid up front.`); ``` ``` the archive tier's minimum retention period is 180 days = 6 months. The rule transitions to archive in month 1, deletes when the lifetime ends. family | life | in archive | billed | transition fee | no archive | archive | ratio ------------------|-------|------------|--------|-----------------|------------|---------|------ invoice-documents | 3 mo | 2 mo | 6 mo | 4.30 | 2.15 | 5.20 | 2.42 field-photos | 60 mo | 59 mo | 59 mo | 2.10 | 203.74 | 14.21 | 0.07 early deletion: 42,000 photos moved to archive, deleted on day 45. used 1.5 months, billed 6 months, paid for 4.5 idle months. the bill for deleting on day 45 is 0.886 units; never deleting and holding it for 6 months bills the same 0.886 units. the 4.5 months in between, 0.664 units, are paid for objects that no longer exist: early deletion gains nothing. archive is not a storage class for data that will live under 6 months, it is a debt paid up front. ``` The table applies a single rule to two families and gets opposite results. Field photos live sixty months; the fifty-nine months spent in archive are far above the minimum retention period, so the billed period is also fifty-nine months. The family that costs 203.74 units without archive drops to 14.21 units with it — a ratio of 0.07, a fourteenfold gain. Invoice documents live three months; they spend two months in archive but are **billed for six months**. The family that costs 2.15 units without archive rises to 5.20 units with it: a 2.42x loss. Same rule, same tier, opposite result. Two things drive the difference — the object's lifetime and the object's size. The transition fee for 86,000 small documents alone is 4.30 units, twice the archive-free total. The early-deletion block names the item. Forty-two thousand photos are placed in archive and deleted on day forty-five: 1.5 months used, 6 months billed. The bill for deleting is 0.886 units, and the bill for never deleting and holding it six months is also 0.886 units — identical. The 0.664 units paid for the 4.5 months in between are for objects that no longer exist. **The minimum retention period is this lesson's unrecoverable item:** the debt is born the moment the object enters the class, and neither deletion nor a class change closes it. The delegated-decision side is at its widest here. In the archive tier, durability, placement, storage medium, repair, and keeping the data readable all belong entirely to the provider; none of it is visible, none of it has a dial. What stays comes down to three numbers: which object enters archive and when, how long it lives, and at what speed it gets retrieved when needed. All three are decided before the data is written, because all three are paid for afterward. #### Summary - Storage in the archive tier is cheap, but data cannot be read right away: the same audit request waits 49.5 hours and costs 4.52 units at slow speed, waits 0.6 hours and costs 24.06 units at fast speed. - The wait grows with rounds, not bytes: 126,000 objects split into 10,000-object batches make four rounds, and first-response latency is paid again on every round. - Four-fifths of the retrieval bill comes from object count: 126,000 objects' request fee is 3.78 units, the same bytes' byte fee is 0.74 units. - Retrieval makes data readable, it does not move it: the same 297.9 GiB, downloaded to the internet, adds a 25.32-unit egress fee on top of retrieval, 5.6 times the cheapest retrieval. - The minimum retention period is an unrecoverable item: archive gains fourteen times for photos that live sixty months, and costs 2.42 times more for documents that live three months; an object deleted on day forty-five still pays for six months. #### Next Step Six lessons measured **where** data goes: object, block, shared file, and archive. None of them asked a single question — who can read this data. In the fictional measurement network, field teams open photos from their phones, auditors request invoice documents from outside, and neither group has bucket access. Granting an identity opens a permanent door; what is usually wanted, though, is a **time-limited and narrow** door. The next lesson measures this: a presigned URL is generated, and its expiry and scope are actually verified. Three things will be measured — the regeneration load that grows as the link's lifetime shortens, the leak window that grows as it lengthens, and the object count a single link opens as scope widens. ### Access Control and Signed URLs Source: https://academia.sh/en/courses/cloud-storage/access-control-and-signed-urls Measuring time-limited, narrow access: generating a presigned URL with a fictional key and actually verifying its lifetime, method, and scope checks; scanning link lifetime to compare regeneration load against the leak window, counting the surface scope width opens, and showing a presigned URL's unrecoverability through its cancellation blast radius. Six lessons measured where data goes. None of them asked a single question: who can read this data. In the fictional measurement network, field teams open photos from their phones, auditors request invoice documents from outside, and neither group has bucket access. Granting an identity opens a permanent door; what is usually wanted, though, is a **time-limited and narrow** door — this object, for this long, read only. A **presigned URL** offers this. A link signed with a key proves that the fields it carries have not been changed; the store verifies the link and hands over the object. Whoever holds the link has no account, no identity, no permission — the link itself is the permission. This puts its power and its danger in the same sentence: wherever the link goes, the permission goes with it. - **ST37** — The signing key is fictional, not a real secret; the clock is fixed so the output stays the same run to run. - **ST38** — The canonical string consists of the method, scope, resource, and expiry fields; the signature is this string's keyed digest. - **ST39** — Verification runs four checks: signature, clock, method, and scope. ```js // signature/sign.mjs — a presigned URL is generated and lifetime and scope are ACTUALLY verified. // The key is fictional, not a real secret; the clock is fixed so the output stays the same run to run. import { createHmac, timingSafeEqual } from "node:crypto"; const KEY = Buffer.from("fictional-signing-key-example-invalid"); // no real secret is ever written const NOW = 1_774_000_000; // fixed clock (seconds); Date.now() is not used const canonical = (method, scope, resource, expiry) => [method, scope, resource, expiry].join("\n"); const sign = (method, scope, resource, lifetimeSec) => { const expiry = NOW + lifetimeSec; const signature = createHmac("sha256", KEY).update(canonical(method, scope, resource, expiry)).digest("hex"); return { method, scope, resource, expiry, signature }; }; // Verification: does the signature hold, has time expired, is the requested key in scope, is the method the same. const verify = (link, requestedMethod, requestedKey, time) => { const expected = createHmac("sha256", KEY).update(canonical(link.method, link.scope, link.resource, link.expiry)).digest("hex"); const expectedBuf = Buffer.from(expected, "hex"), actualBuf = Buffer.from(link.signature, "hex"); if (expectedBuf.length !== actualBuf.length || !timingSafeEqual(expectedBuf, actualBuf)) return "signature mismatch"; if (time > link.expiry) return `expired ${time - link.expiry} s ago`; if (link.method !== requestedMethod) return "wrong method"; if (!requestedKey.startsWith(link.scope)) return "out of scope"; return "valid"; }; const PREFIX = "field/2026/03/14/"; const OBJECT = PREFIX + "photo-00412.jpg"; const link = sign("read", OBJECT, OBJECT, 900); // single object, 15-minute lifetime const prefixLink = sign("read", PREFIX, OBJECT, 900); // one day's worth of photos const TRIAL = [ ["correct key, within lifetime", link, "read", OBJECT, NOW + 300], ["after expiry", link, "read", OBJECT, NOW + 1200], ["different object requested", link, "read", PREFIX + "photo-00413.jpg", NOW + 300], ["write with a read link", link, "write", OBJECT, NOW + 300], ["last character of signature changed", { ...link, signature: link.signature.slice(0, -1) + (link.signature.endsWith("a") ? "b" : "a") }, "read", OBJECT, NOW + 300], ["prefix scope, within scope", prefixLink, "read", PREFIX + "photo-09999.jpg", NOW + 300], ["prefix scope, outside scope", prefixLink, "read", "field/2026/03/15/photo-00001.jpg", NOW + 300], ]; console.log(`signed with a fictional key; fixed clock ${NOW}. Signature length ${link.signature.length} hex digits.`); console.log(`single-object link: scope ${link.scope}, expiry ${link.expiry} (${link.expiry - NOW} s lifetime)`); console.log(`prefix link : scope ${prefixLink.scope}`); console.log("\ntrial | requested key | result"); console.log("--------------------------------------|-----------------------------------|---------------------"); for (const [name, l, method, key, time] of TRIAL) console.log(`${name.padEnd(37)} | ${key.padEnd(33)} | ${verify(l, method, key, time)}`); console.log("\nrejection reasons are distinct: one from the signature, one from the clock, two from the fields written into the canonical string."); console.log("The signature only says the link was not modified; how long it lives and what it opens is said by"); console.log("the scope and expiry fields — both are inside the signature, and cannot be changed from outside."); ``` ``` signed with a fictional key; fixed clock 1774000000. Signature length 64 hex digits. single-object link: scope field/2026/03/14/photo-00412.jpg, expiry 1774000900 (900 s lifetime) prefix link : scope field/2026/03/14/ trial | requested key | result --------------------------------------|-----------------------------------|--------------------- correct key, within lifetime | field/2026/03/14/photo-00412.jpg | valid after expiry | field/2026/03/14/photo-00412.jpg | expired 300 s ago different object requested | field/2026/03/14/photo-00413.jpg | out of scope write with a read link | field/2026/03/14/photo-00412.jpg | wrong method last character of signature changed | field/2026/03/14/photo-00412.jpg | signature mismatch prefix scope, within scope | field/2026/03/14/photo-09999.jpg | valid prefix scope, outside scope | field/2026/03/15/photo-00001.jpg | out of scope rejection reasons are distinct: one from the signature, one from the clock, two from the fields written into the canonical string. The signature only says the link was not modified; how long it lives and what it opens is said by the scope and expiry fields — both are inside the signature, and cannot be changed from outside. ``` Seven trials show the four checks one by one. The fifteen-minute link is valid at the three-hundredth second, and at the twelve-hundredth second it says "expired 300 s ago" — expiry is not a suggestion, it is part of verification. Requesting a neighboring object with the same link trips the scope check; attempting a write with a read link trips the method check. Changing the signature's last digit stops all four at the signature check. The distinction matters. The signature only says one thing: the link **has not been modified** since it was generated. What says how long the link lives and what it opens is not the signature but the scope and expiry fields written into the canonical string. Because these fields are inside the signature, they cannot be changed from outside; but whoever writes their content at generation time is the one who draws the boundary of access. Generating a link is an act of delegating permission, and it comes down to two numbers: lifetime and scope. #### Lifetime and Scope - **ST40** — The field application's workload is a model: 12,000 sessions a day, a session is 1,500 seconds, the screen refreshes eighteen times, forty photos per screen. - **ST41** — The leak rate is fictional: one in 200,000 generated links leaks into a log, an address bar, or a share. Generating one signature costs 0.012 ms of CPU time in the model. ```js // signature/lifetime.mjs — a link's lifetime and scope are scanned: regeneration load, leak window, // cancellation blast radius. Workload and rates are a MODEL, no randomness. const SESSIONS = 12_000, SESSION_SEC = 1500, REFRESHES = 18, SCREEN_PHOTOS = 40; const LEAK_RATE = 1 / 200_000; // the leaking share of generated links (fictional) const SIGN_MS = 0.012; // a signature generation's CPU time in the model const SCOPE = { // the surface a single leaked link opens "single object": { objects: 1, gib: 0.0035 }, "one day's prefix": { objects: 3_500, gib: 12.3 }, "whole bucket": { objects: 1_678_000, gib: 184.8 }, }; const EGRESS_INTERNET = 0.085; // unit/GiB, fictional const LIFETIME = [60, 300, 1800, 7200, 86_400, 604_800]; const g = (x, n, d = 2) => x.toFixed(d).padStart(n); const s = (x, n) => String(x).padStart(n); const produced = (ttl) => SESSIONS * SCREEN_PHOTOS * Math.min(REFRESHES, Math.ceil(SESSION_SEC / ttl)); console.log(`${SESSIONS} sessions/day, session ${SESSION_SEC} s, screen refreshes ${REFRESHES} times, ` + `${SCREEN_PHOTOS} photos on screen. Leak rate 1/${(1 / LEAK_RATE).toLocaleString("en-US")} (fictional).`); console.log("\nlifetime | links produced/day | CPU s/day | leaked/day | open object-hr | valid at once"); console.log("----------|-----------------------|----------------|------------|-----------------|------------------"); const PRODUCED = {}; for (const ttl of LIFETIME) { const p = PRODUCED[ttl] = produced(ttl), leaked = p * LEAK_RATE; console.log(`${s(ttl + " s", 9)} | ${s(p.toLocaleString("en-US"), 21)} | ${g(p * SIGN_MS / 1000, 14, 1)} | ` + `${g(leaked, 10, 1)} | ${g(leaked * ttl / 3600, 15, 2)} | ${s(Math.round(p / 86400 * ttl).toLocaleString("en-US"), 17)}`); } const elbow = LIFETIME.find((t) => PRODUCED[t] === PRODUCED[LIFETIME[LIFETIME.length - 1]]); console.log(`elbow at ${elbow} s: past this point production levels off at ${PRODUCED[elbow].toLocaleString("en-US")}/day, ` + `but the open window keeps growing —`); console.log(` at ${elbow} s: ${g(PRODUCED[elbow] * LEAK_RATE * elbow / 3600, 0, 2)} object-hours, ` + `at 604800 s: ${g(PRODUCED[604_800] * LEAK_RATE * 604_800 / 3600, 0, 2)} object-hours: ` + `${(PRODUCED[604_800] * 604_800 / (PRODUCED[elbow] * elbow)).toFixed(0)}x. A longer lifetime no longer gains anything.`); console.log("\nscope width — if a single link leaks:"); console.log("scope | objects opened | GiB opened | egress if all downloaded"); console.log("------------------|----------------|------------|--------------------------"); for (const [name, sc] of Object.entries(SCOPE)) console.log(`${name.padEnd(17)} | ${s(sc.objects.toLocaleString("en-US"), 14)} | ${g(sc.gib, 10, 3)} | ` + `${g(sc.gib * EGRESS_INTERNET, 22)} units`); console.log(`widening scope from a single object to the whole bucket multiplies the surface a single leak ` + `opens by ${(SCOPE["whole bucket"].objects / SCOPE["single object"].objects).toLocaleString("en-US")}x.`); console.log("\ncancelability: a presigned link cannot be revoked, it stays valid until it expires."); console.log("The only lever is rotating the signing key; that breaks every link valid at that moment, at once:"); for (const ttl of [1800, 86_400, 604_800]) console.log(` lifetime ${s(ttl + " s", 9)} -> rotating the key breaks ${s(Math.round(PRODUCED[ttl] / 86400 * ttl).toLocaleString("en-US"), 9)} ` + `links; that is the price paid for a single leaked link.`); // Both axes together: the surface a leaked link opens is the product of lifetime and scope. console.log("\nlifetime x scope — daily open object-hours (leaked link count x lifetime x objects in scope):"); console.log("lifetime |" + Object.keys(SCOPE).map((a) => a.padStart(17)).join(" |")); console.log("----------|" + Object.keys(SCOPE).map(() => "-".repeat(18)).join("|")); for (const ttl of [300, 1800, 86_400, 604_800]) { const leaked = PRODUCED[ttl] * LEAK_RATE; console.log(`${s(ttl + " s", 9)} |` + Object.values(SCOPE) .map((sc) => g(leaked * ttl / 3600 * sc.objects, 17, 1)).join(" |")); } console.log("A short lifetime and a narrow scope work in the same direction: since each shrinks one factor of the"); console.log("product, their effects do not add, they multiply — that is the gap between 300 s, single object and 604800 s, whole bucket."); ``` ``` 12000 sessions/day, session 1500 s, screen refreshes 18 times, 40 photos on screen. Leak rate 1/200,000 (fictional). lifetime | links produced/day | CPU s/day | leaked/day | open object-hr | valid at once ----------|-----------------------|----------------|------------|-----------------|------------------ 60 s | 8,640,000 | 103.7 | 43.2 | 0.72 | 6,000 300 s | 2,400,000 | 28.8 | 12.0 | 1.00 | 8,333 1800 s | 480,000 | 5.8 | 2.4 | 1.20 | 10,000 7200 s | 480,000 | 5.8 | 2.4 | 4.80 | 40,000 86400 s | 480,000 | 5.8 | 2.4 | 57.60 | 480,000 604800 s | 480,000 | 5.8 | 2.4 | 403.20 | 3,360,000 elbow at 1800 s: past this point production levels off at 480,000/day, but the open window keeps growing — at 1800 s: 1.20 object-hours, at 604800 s: 403.20 object-hours: 336x. A longer lifetime no longer gains anything. scope width — if a single link leaks: scope | objects opened | GiB opened | egress if all downloaded ------------------|----------------|------------|-------------------------- single object | 1 | 0.004 | 0.00 units one day's prefix | 3,500 | 12.300 | 1.05 units whole bucket | 1,678,000 | 184.800 | 15.71 units widening scope from a single object to the whole bucket multiplies the surface a single leak opens by 1,678,000x. cancelability: a presigned link cannot be revoked, it stays valid until it expires. The only lever is rotating the signing key; that breaks every link valid at that moment, at once: lifetime 1800 s -> rotating the key breaks 10,000 links; that is the price paid for a single leaked link. lifetime 86400 s -> rotating the key breaks 480,000 links; that is the price paid for a single leaked link. lifetime 604800 s -> rotating the key breaks 3,360,000 links; that is the price paid for a single leaked link. lifetime x scope — daily open object-hours (leaked link count x lifetime x objects in scope): lifetime | single object | one day's prefix | whole bucket ----------|------------------|------------------|------------------ 300 s | 1.0 | 3500.0 | 1678000.0 1800 s | 1.2 | 4200.0 | 2013600.0 86400 s | 57.6 | 201600.0 | 96652800.0 604800 s | 403.2 | 1411200.0 | 676569600.0 A short lifetime and a narrow scope work in the same direction: since each shrinks one factor of the product, their effects do not add, they multiply — that is the gap between 300 s, single object and 604800 s, whole bucket. ``` The lifetime scan balances two columns. At a sixty-second lifetime, 8,640,000 signatures are generated a day; the CPU cost is 103.7 seconds a day and the open window is 0.72 object-hours. As lifetime lengthens, production drops, but only to a floor: past 1,800 seconds, production levels off at 480,000, because a link already covers an entire session. **This is the elbow.** Past this point, a longer lifetime reduces no load; the open window keeps growing — 1.20 object-hours at 1,800 seconds, 403.20 object-hours at 604,800 seconds, 336 times over. A seven-day link carries three hundred thirty-six times the risk for something a thirty-minute link does not gain at all. Scope is the second axis, and its scale differs. A single leaked link opens one object at single-object scope; 3,500 objects and 12.3 GiB at one day's prefix; 1,678,000 objects and 184.8 GiB at the whole bucket — this course's entire byte count, in a single string. The gap is 1,678,000 times, and generating the link is equally easy in all three. Prefix scope is often chosen for convenience — a single request fills the whole screen — but that convenience costs a leak's surface thousands of times over. The last table combines both axes into one number: daily open object-hours, the product of the leaked link count, the lifetime, and the object count in scope. At a five-minute lifetime and single-object scope this is 1.0; at a seven-day lifetime and bucket scope, 676,569,600. Loosening both axes does not add the risk, it **multiplies** it; tightening both reduces it the same way. So "we kept the link's lifetime short" is not, by itself, a safeguard — unless scope is also narrowed, only one factor has shrunk. #### Unrevocability - **ST42** — In this model a presigned URL cannot be revoked; the only way is rotating the signing key, and that breaks every link valid at that moment, at once. Some providers offer selective revocation by binding the link to a named access policy; this lesson does not use that mechanism. The last block writes this type's harshest item. Once a link leaks, nothing can be done: the link stays valid until it expires. There is no identity to suspend, no session to close. The only lever is rotating the signing key, and that lever is not selective. Rotating the key at a thirty-minute lifetime breaks 10,000 links; at a twenty-four-hour lifetime, 480,000; at a seven-day lifetime, 3,360,000. The price of a single leak is breaking every link currently in circulation — and on the user's side, that means screens come back empty. This explains why choosing a lifetime is not a security setting: lifetime is also **the cancellation cost**. A short lifetime shrinks both the leak window and the cancellation blast radius, pulling both columns down at once — which is what the elbow point means: 1,800 seconds is where the regeneration load bottoms out and both risks stay smallest. The delegated-decision side is narrow here. The provider only performs verification: it recomputes the signature, checks the clock, compares the scope. Who accesses what, for how long, stays entirely with you, and by this course's measure all three are unrecoverable — lifetime cannot be shortened, scope cannot be narrowed, the link itself cannot be revoked. Egress is plain too: a leaked bucket link allows all 184.8 GiB to be downloaded, incurring a 15.71-unit egress fee, and the bill arrives before the leak is found. #### Summary - A presigned URL passes through four checks: signature, expiry, method, and scope. The signature only says the link has not been modified; the fields in the canonical string say how long it lives and what it opens. - The lifetime scan has an elbow: past 1,800 seconds, production levels off at 480,000 a day, and lengthening the lifetime reduces no load. - The open window grows in direct proportion to lifetime: from 1.20 object-hours to 403.20 object-hours, 336 times over. A seven-day link carries this difference for something it does not gain. - Scope sets the scale of a leak: single object opens 1, one day's prefix opens 3,500, the whole bucket opens 1,678,000 objects and 184.8 GiB; downloading it all incurs a 15.71-unit egress fee. - The unrecoverable item is the link itself: it cannot be revoked, it stays valid until it expires. The only lever is rotating the key, and that breaks 10,000 links at a thirty-minute lifetime, 3,360,000 at a seven-day lifetime, all at once. #### Next Step This topic closed two questions: where data goes and who accesses it. The fictional measurement network's meter readings, invoice documents, field photos, and work-order records are now placed by type, their class chosen, their lifecycle written, and their access granted through time-limited links. One item still stands outside, and has been passing by the edge for six lessons: the **queried** part of this data. Rate tables, work-order statuses, and invoice line items sit not in object storage but in a database set up by hand on an instance — a scheduled job takes its backup, someone upgrades its version by hand, a single copy runs. The next topic measures renting this layer: when a database is taken as a service, which decision is delegated, which stays, and which of the delegated ones cannot be undone. ## Managed Data Services Handing off database operation: the decisions a managed relational database's backup, patching, and failover automation take over, spreading read load across read replicas and the cost of lag, how the partition key and capacity model fix the access pattern in a managed key-value service, secondary indexes and change streams, the managed cache service, zonal and internet egress pricing, and snapshots with point-in-time restore. ### Managed Relational Database Source: https://academia.sh/en/courses/cloud-storage/managed-relational-database An item-by-item comparison of a self-run relational database against the managed service: how many of 28 decisions go to the provider, how many stay with you, and how many become inaccessible altogether; how many of the kept items turn out to be one-way; the failover window split into the provider's fixed share and the application's share, with dropped requests measured across five cache lifetimes; and the data's exit costed as logical export time plus egress fee. The previous topic settled where objects go and who can reach them, and for how long, with numbers. One thing was left unasked: the **queried** data — where meter readings are summarized, work orders change state, and billing documents are linked — still sits in a relational database set up by hand. That database's machine may be rented, but the person taking the backup, applying the patch, and promoting the standby to primary during an outage is still a person. This lesson's question is not "is the managed service better." What gets measured is narrower and more concrete: when the database is rented, **which decisions change hands**. A decision can go to one of three places — it is delegated to the provider, it stays with you, or it lands with no one at all: it becomes inaccessible. That third set is this lesson's real subject, because a database you run yourself has no such set. #### The Decision Splits into Three Sets In a database you run by hand, all 28 items are kept: which disk the hardware runs on, what the kernel setting is. The managed service splits this list. **Delegated** items are now the provider's job, and the provider answers for the outcome. **Kept** items remain as settings; you can change them. **Inaccessible** items are neither delegated nor kept: they have been removed from the settings surface, the provider has picked a number, and that number is not visible. This split is not abstract. On an outage night, an item in the inaccessible column decides how long the night runs, and that item cannot be touched. The catalog was built on a single criterion: what it takes to change the item. If a request changes it, it is counted as kept; if a provider notification changes it, it is counted as delegated; if nothing changes it, it is counted as inaccessible. The line is thin for some items — backup frequency is kept but how the backup is taken is delegated, growing storage is kept but shrinking it is inaccessible. This is not a flaw in the catalog; it is the thing being measured: a managed service does not hand over decisions as one block, it **splits every item on its own**. #### Mechanics **MD1 — the fictional regional measurement network's work-order database is 220 GB and takes 1,400 requests a second.** All numbers are fictional. **MD2 — the health check runs every 5 seconds, three consecutive failed checks trigger a failover, and the first check lands 2.4 seconds after the fault.** **MD3 — promoting the replica to primary takes 20 seconds.** **MD4 — on the application side, the endpoint name's resolver cache lives for T seconds; the renewal moments of sixteen application instances are spread evenly across the T interval.** **MD5 — each instance carries a pool of 120 connections, one connection is established in 25 ms, and eight connections can be established concurrently.** **MD6 — the egress fee is a fictional unit: 0 within a region, 2 across regions, 9 units/GB to the internet.** MD2 and MD3 sit on the provider's side, MD4 and MD5 on the application's; the measurement is built to expose this line. ```js // md/model.mjs — MANAGED RELATIONAL DATABASE model. The decision catalog and failover // timeline defined over the fictional regional measurement network's work-order database. // Not a real product or measurement; there is no randomness, every run yields the same numbers. // Third column: who holds the decision in the managed service. Fourth column: is it reversible. export const DECISION = [ ["hardware selection and placement", "delegated", true], ["operating system patching", "delegated", true], ["minor version patching", "delegated", true], ["backup format", "delegated", true], ["snapshot storage", "delegated", true], ["point-in-time restore infrastructure", "delegated", true], ["buffer pool size", "delegated", true], ["replication layout", "delegated", true], ["major version upgrade decision", "kept", false], ["patch window", "kept", true], ["backup frequency", "kept", true], ["backup retention period", "kept", true], ["manual failover trigger", "kept", true], ["connection limit", "kept", true], ["planner cost settings", "kept", true], ["installing a listed extension", "kept", true], ["storage growth", "kept", false], ["placement region selection", "kept", false], ["character set and collation", "kept", false], ["shell access", "inaccessible", null], ["file system layout", "inaccessible", null], ["data directory path", "inaccessible", null], ["failover threshold", "inaccessible", null], ["write-ahead log threshold", "inaccessible", null], ["extension not on the list", "inaccessible", null], ["storage shrink", "inaccessible", null], ["direct access to log files", "inaccessible", null], ["kernel parameter", "inaccessible", null], ]; // Failover timeline. The provider's two steps are fixed; the third step depends on the // application's resolver-cache lifetime (T) and VARIES PER INSTANCE. export const HEALTH_INTERVAL = 5, HEALTH_PHASE = 2.4, THRESHOLD = 3, UPGRADE = 20; export const INSTANCES = 16, REQUESTS = 1400, POOL = 120, CONNECT_MS = 25, CONCURRENT = 8; export const detect = () => HEALTH_PHASE + (THRESHOLD - 1) * HEALTH_INTERVAL; export const poolSetup = () => (POOL * CONNECT_MS) / CONCURRENT / 1000; // Each application instance renews its own cache at a different moment, spread evenly // across T. An instance learns the new endpoint at the FIRST renewal after the upgrade ends. export function failover(T) { const readiness = detect() + UPGRADE; const instances = []; for (let i = 0; i < INSTANCES; i += 1) { const phase = (i * T) / INSTANCES; const learn = phase + T * Math.max(0, Math.ceil((readiness - phase) / T)); instances.push(learn + poolSetup()); } const dropped = instances.reduce((t, x) => t + x * (REQUESTS / INSTANCES), 0); return { readiness, worst: Math.max(...instances), average: instances.reduce((a, b) => a + b, 0) / INSTANCES, dropped: Math.round(dropped) }; } ``` ```js // md/measure.mjs — delegated decision count and the failover window import { DECISION, failover, detect, poolSetup, INSTANCES, REQUESTS, POOL, UPGRADE } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const count = (k) => DECISION.filter(([, v]) => v === k).length; console.log(`decision catalog: ${DECISION.length} items. In a database you run yourself, all are kept.\n`); console.log("in the managed service | items | share | example"); console.log("------------------------|-------|-------|--------------------------------------------"); for (const k of ["delegated", "kept", "inaccessible"]) { const list = DECISION.filter(([, v]) => v === k).map(([a]) => a); console.log(`${k.padEnd(23)} | ${s(list.length, 5)} | ` + `${s((100 * list.length / DECISION.length).toFixed(1) + "%", 5)} | ${list.slice(0, 2).join(", ")}`); } const irreversible = DECISION.filter(([, v, g]) => v === "kept" && g === false); console.log(`\nof the ${count("kept")} kept items, ${irreversible.length} are one-way:`); for (const [a] of irreversible) console.log(` - ${a}`); console.log(`${count("inaccessible")} inaccessible items are settings when you run it yourself; here there are none.\n`); console.log(`failover: health check detection ${detect().toFixed(1)} s, upgrade ` + `${UPGRADE} s -> provider share ${(detect() + UPGRADE).toFixed(1)} s (fixed).`); console.log(`pool setup ${(poolSetup() * 1000).toFixed(0)} ms, ${INSTANCES} instances x ${POOL} ` + `connections = ${INSTANCES * POOL} new connections hit the new primary at once.`); console.log(`request rate ${REQUESTS}/s; all of it drops during the failover.\n`); console.log("cache TTL T | provider share | app share | outage (avg) | outage (worst) | dropped req"); console.log("----------------|----------------|---------------|---------------|-------------------|-------------"); for (const T of [5, 15, 30, 60, 120]) { const r = failover(T); console.log(`${s(T + " s", 15)} | ${s(r.readiness.toFixed(1) + " s", 14)} | ` + `${s((r.average - r.readiness).toFixed(1) + " s", 13)} | ${s(r.average.toFixed(1) + " s", 13)} | ` + `${s(r.worst.toFixed(1) + " s", 17)} | ${s(r.dropped, 11)}`); } const a = failover(5), b = failover(120); console.log(`\nrun-independent: provider share fixed at ${a.readiness.toFixed(1)} s; as T goes ` + `from 5 to 120, outage rises ${a.average.toFixed(1)} -> ${b.average.toFixed(1)} s, dropped requests ` + `${a.dropped} -> ${b.dropped}.`); console.log(`the failover threshold is inaccessible, so detection time cannot be shortened; the only ` + `thing that can be shortened is T on the application side.`); // Exit: because the snapshot format is inaccessible, data can leave only through a LOGICAL export. const SIZE_GB = 220, READ_MBS = 40, LOAD_MBS = 26, RATE = { "within region": 0, "cross-region": 2, "internet": 9 }; const hours = (mbs) => (SIZE_GB * 1024) / mbs / 3600; console.log(`\nexit: ${SIZE_GB} GB of work-order data, logical export at ${READ_MBS} MB/s, ` + `load at ${LOAD_MBS} MB/s (indexes are rebuilt).`); console.log(`time: export ${hours(READ_MBS).toFixed(2)} h + load ${hours(LOAD_MBS).toFixed(2)} h = ` + `${(hours(READ_MBS) + hours(LOAD_MBS)).toFixed(2)} h. The snapshot cannot travel: the format sits in the inaccessible column.`); console.log("destination | egress fee (fictional unit/GB) | total (fictional unit)"); console.log("-----------------|--------------------------------|------------------------"); for (const [name, rate] of Object.entries(RATE)) console.log(`${name.padEnd(16)} | ${s(rate, 30)} | ${s(rate * SIZE_GB, 20)}`); ``` ``` decision catalog: 28 items. In a database you run yourself, all are kept. in the managed service | items | share | example ------------------------|-------|-------|-------------------------------------------- delegated | 8 | 28.6% | hardware selection and placement, operating system patching kept | 11 | 39.3% | major version upgrade decision, patch window inaccessible | 9 | 32.1% | shell access, file system layout of the 11 kept items, 4 are one-way: - major version upgrade decision - storage growth - placement region selection - character set and collation 9 inaccessible items are settings when you run it yourself; here there are none. failover: health check detection 12.4 s, upgrade 20 s -> provider share 32.4 s (fixed). pool setup 375 ms, 16 instances x 120 connections = 1920 new connections hit the new primary at once. request rate 1400/s; all of it drops during the failover. cache TTL T | provider share | app share | outage (avg) | outage (worst) | dropped req ----------------|----------------|---------------|---------------|-------------------|------------- 5 s | 32.4 s | 2.8 s | 35.2 s | 37.6 s | 49306 15 s | 32.4 s | 7.8 s | 40.2 s | 47.3 s | 56306 30 s | 32.4 s | 15.8 s | 48.2 s | 62.3 s | 67463 60 s | 32.4 s | 29.9 s | 62.3 s | 90.4 s | 87150 120 s | 32.4 s | 61.7 s | 94.1 s | 150.4 s | 131775 run-independent: provider share fixed at 32.4 s; as T goes from 5 to 120, outage rises 35.2 -> 94.1 s, dropped requests 49306 -> 131775. the failover threshold is inaccessible, so detection time cannot be shortened; the only thing that can be shortened is T on the application side. exit: 220 GB of work-order data, logical export at 40 MB/s, load at 26 MB/s (indexes are rebuilt). time: export 1.56 h + load 2.41 h = 3.97 h. The snapshot cannot travel: the format sits in the inaccessible column. destination | egress fee (fictional unit/GB) | total (fictional unit) -----------------|--------------------------------|------------------------ within region | 0 | 0 cross-region | 2 | 440 internet | 9 | 1980 ``` #### The Inaccessible Column Is the Second-Largest Set The delegated count is 8, or 28.6% of the catalog. This is the column the managed service is selling, and what sits in it genuinely cuts workload: operating system patching, minor version patching, backup format, and replication layout are now someone else's night shift. The unexpected part is the third column: **9 items, 32.1%**. These items were not delegated; they vanished. There is no shell access, no data directory path, no direct look at the log files. Some of these go unnoticed in their absence. Some are noticed the day they are lost: the **failover threshold** sits in this column. How many seconds the health check runs at and how many failures trigger a failover cannot be read or adjusted. The same column holds the **write-ahead log threshold** and an **extension not on the list**: if an extension the application needs is not on the provider's list, there is no way to install it — in a database you run yourself, that is a package-install job. The 11 kept items offer a misleading comfort. Four of them are **one-way**: a major version upgrade cannot be undone, storage that has grown cannot shrink, the chosen placement region does not change without moving the data, and the character-set-and- collation choice is not fixed without rewriting tables. All four are dropdown menus on the setup screen, and their consequences last the database's whole lifetime. #### What Gets Measured Is Not the Outage, but Who Owns It Failover is the most talked-about feature of a managed service, and it is usually cited as a single number. The table shows that single number is made of two parts. **The provider's share is 32.4 seconds in all five rows.** Detection is 12.4 seconds (a 2.4-second phase plus two check intervals), the upgrade is 20 seconds. No setting changes this number, because the item that produces it sits in the inaccessible column. **The application's share climbs from 2.8 seconds to 61.7 seconds.** The whole difference comes from the endpoint name's resolver-cache lifetime. At T = 5 seconds, the average outage is 35.2 seconds and 49,306 requests drop; at T = 120 seconds, the average is 94.1 seconds and 131,775 requests drop. Same failover, same provider, 2.67 times the loss. The worst-case column makes it even plainer: at T = 120, one application instance spends 150.4 seconds trying to connect to a dead address. The rule that follows draws the boundary of the delegated decision: **the failover itself is delegated, the moment it finishes inside the application is not.** The pool row points at the same thing — when sixteen instances come back up, the new primary faces 1,920 connection requests at once; the connection limit is a kept item, and if it sits below that number, the failover turns into an outage that outlasts the failover itself. #### The Exit's Cost Is the Format Being Locked The last block applies the course's recurring item to this lesson. When data needs to move somewhere else, the snapshot is useless: it is taken in the delegated column and becomes unreadable in the inaccessible column. Only one path is left, **logical export**: 220 GB of data comes out at 40 MB/s in 1.56 hours, and loads at the destination at 26 MB/s in 2.41 hours — loading is slower than reading because indexes are rebuilt. The total is 3.97 hours, plus an egress fee: 0 within a region, 440 across regions, 1,980 fictional units over the internet. The ratio among these three numbers stays the same for the rest of the course. Data paid no fee coming in; the same data got charged the moment it crossed a region boundary going out. The time itself is also a cost: through those 3.97 hours, either two systems write in parallel and the gap between them is closed by hand, or the work-order records freeze. That is why placement region is counted among the one-way items — changing regions means paying this four-hour window all over again. #### Summary - Of 28 decisions, 8 were delegated to the provider, 11 stayed kept, and **9 became inaccessible**; that third set does not exist in a database you run yourself. - Of the 11 kept items, 4 are one-way: the major version upgrade, storage growth, placement region, and collation choice are not reversed without rewriting the data. - 32.4 seconds of the failover outage is the provider's fixed share; it cannot be shortened because the failover threshold is inaccessible. - The application's share can climb from 2.8 to 61.7 seconds: within the same failover, dropped requests rise from 49,306 to 131,775, and the worst-case outage rises from 37.6 to 150.4 seconds. - Because the snapshot format is inaccessible, exiting forces a logical export: 3.97 hours for 220 GB, plus a 1,980 fictional units egress fee over the internet. #### Next Step This lesson measured under the assumption of a single primary database: no other copy could absorb the load at the moment of failover, the second machine only stood by. But most of the work-order database's request mix is reads — meter-reading summaries, billing queries, field-photo lookups. Opening a standby copy to reads spreads the load, but that copy lags behind the primary. The next lesson scans that lag and counts how many reads come back stale at each value; it then separates out requests with a **read-your-writes** requirement, and measures how many requests routing them to the primary saves, and how much it adds to the primary's load. ### Read Replicas Source: https://academia.sh/en/courses/cloud-storage/read-replicas Measuring how replica lag reflects onto the application: in a stream of 420,000 requests, lag is scanned from 0.1 to 5 seconds and, at each value, the count of stale reads and of requests unable to see their own write is derived; a session-based sticky window's saved requests are compared against the read share it shifts to the primary; how both the per-replica gain and the lag change as replica count rises is counted; and the monthly transfer cost of an out-of-region replica is computed. The previous lesson stood on a single primary: the second machine stood by, doing no work until the moment of failover. Looking at the work-order database's request mix, this waste stands out — most of the stream is reads. Opening the standby copy to reads spreads the load, and in the managed service this is a single switch. What sits under that switch is a delegated decision: how replication works is no longer visible. That mechanism — synchronous or asynchronous, how many members acknowledge — was measured in the Advanced Data Scaling course and is not repeated here. What gets measured is the one symptom that leaks out of the mechanism: **replica lag**. And what gets counted is not "how many seconds of lag," but how many requests the lag **touches**. #### Two Kinds of Stale Read Not all stale reads carry the same weight. If a field crew sees a work order another crew closed half a second ago still listed as open, that is an inconsistency, but it goes unnoticed. If the same crew sees a work order **it just closed** still listed as open, that is a bug: the screen has ignored an action just taken, and the action gets retried. These requests are said to carry a **read-your-writes** requirement, and they need to be counted separately, because their fix is separate too. #### Mechanics **MD7 — the stream is a 300-second window: 1,400 requests a second, 78% reads (420,000 requests, 92,400 writes).** **MD8 — the fictional measurement network holds 2,400 sessions and 50,000 open work-order records; a session is a field crew or an automated collector.** **MD9 — 31% of writes are read back by the same session 120–900 ms later, because the screen refreshes the list after the action.** **MD10 — the base replica lag is 0.25 seconds and each added replica adds 0.15 seconds, because the replication stream is pulled from the primary.** **MD11 — the load unit is fictional: a write is 1.0, a read is 0.4, and per-replica replication delivery is 0.08 of a write.** **MD12 — the routing rule is session-based: after a session writes, its reads go to the primary for W seconds.** MD9 is the direct determinant; if the read-back interval grows, the first table's right column shrinks in the same direction. ```js // md/model.mjs — READ REPLICA model. Generates the fictional regional measurement network's // work-order request stream; replication MECHANICS (measured in M17/K05 and M19/K04) are not // modeled — only how lag reflects onto reads is. Hand-written generator, seed visible: same stream every run. export const REQUESTS = 1400, WINDOW = 300, READ_SHARE = 0.78; export const SESSIONS = 2400, RECORDS = 50000, READBACK_SHARE = 0.31; export const READBACK_MIN = 0.12, READBACK_MAX = 0.90; export function generateStream(seed = 20260802) { let s = seed % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const total = REQUESTS * WINDOW, writeCount = Math.round(total * (1 - READ_SHARE)); const events = []; for (let i = 0; i < writeCount; i += 1) { // work order state change const t = rand() * WINDOW, o = Math.floor(rand() * SESSIONS), e = Math.floor(rand() * RECORDS); events.push([t, o, e, 1]); // 1 = write if (rand() < READBACK_SHARE) // same session reads its own write events.push([t + READBACK_MIN + rand() * (READBACK_MAX - READBACK_MIN), o, e, 0]); } const remaining = total - events.length; for (let i = 0; i < remaining; i += 1) // reads of someone else's data events.push([rand() * WINDOW, Math.floor(rand() * SESSIONS), Math.floor(rand() * RECORDS), 0]); events.sort((a, b) => a[0] - b[0]); return events; } // Three numbers are derived per read: lag = time since the last write to that record (a // replica behind this lag is stale), own = did the session that read it also last write it, // since = time since the reading session's OWN last write (the routing rule knows only this). export function deriveReads(events) { const lastWrite = new Map(), sessionLast = new Map(); const lag = [], own = [], since = []; let writes = 0; for (const [t, o, e, kind] of events) { if (kind === 1) { lastWrite.set(e, [t, o]); sessionLast.set(o, t); writes += 1; continue; } const last = lastWrite.get(e); lag.push(last ? t - last[0] : Infinity); own.push(last ? last[1] === o : false); since.push(sessionLast.has(o) ? t - sessionLast.get(o) : Infinity); } return { lag, own, since, writes, reads: lag.length }; } export const stale = (r, L) => r.lag.reduce((n, d) => n + (d < L ? 1 : 0), 0); export const staleOwn = (r, L) => r.lag.reduce((n, d, i) => n + (d < L && r.own[i] ? 1 : 0), 0); export const routed = (r, W) => r.since.reduce((n, d) => n + (d < W ? 1 : 0), 0); export const saved = (r, L, W) => r.lag.reduce((n, d, i) => n + (d < L && r.own[i] && r.since[i] < W ? 1 : 0), 0); export const LAG_BASE = 0.25, LAG_STEP = 0.15; export const LOAD_WRITE = 1.0, LOAD_READ = 0.4, LOAD_REPLICATION = 0.08; export const lagForReplicas = (R) => LAG_BASE + LAG_STEP * R; ``` ```js // md/measure.mjs — how replica lag reflects onto the application, the routing rule, and diminishing returns import { generateStream, deriveReads, stale, staleOwn, routed, saved, lagForReplicas, REQUESTS, WINDOW, SESSIONS, RECORDS, READBACK_SHARE, LOAD_WRITE, LOAD_READ, LOAD_REPLICATION } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const pct = (a, b) => (100 * a / b).toFixed(2) + "%"; const r = deriveReads(generateStream()); console.log(`${REQUESTS * WINDOW} requests / ${WINDOW} s: ${r.writes} writes, ${r.reads} reads. ` + `${SESSIONS} sessions, ${RECORDS} work orders. Seed 20260802.`); console.log(`${100 * READBACK_SHARE}% of writes are read back by the same session.\n`); console.log("replica lag | stale reads | stale share | RYW misses | miss share"); console.log("----------------|-------------|-------------|---------------------------|-----------"); for (const L of [0.1, 0.25, 0.5, 1, 2, 5]) { const b = stale(r, L), k = staleOwn(r, L); console.log(`${s(L.toFixed(2) + " s", 15)} | ${s(b, 11)} | ${s(pct(b, r.reads), 11)} | ` + `${s(k, 25)} | ${s(pct(k, r.reads), 7)}`); } const L = 0.55; const base = staleOwn(r, L); console.log(`\nrouting rule, replica lag held at ${L} s (${base} requests cannot see their own write):`); console.log("sticky window W | reads sent to primary | saved | remaining | read share shifted to primary"); console.log("-------------------|-----------------------|------------|-----------|------------------------------"); for (const W of [0, 0.25, 0.5, 1, 2, 5, 15]) { const sent = routed(r, W), kept = saved(r, L, W); console.log(`${s(W.toFixed(2) + " s", 18)} | ${s(sent, 21)} | ${s(kept, 10)} | ${s(base - kept, 9)} | ` + `${s(pct(sent, r.reads), 26)}`); } console.log(`\nas replica count rises, the replication stream is pulled harder from the primary: lag ` + `${lagForReplicas(1).toFixed(2)} s (1 replica) -> ${lagForReplicas(8).toFixed(2)} s (8 replicas).`); console.log("replicas | lag | reads/s per replica | marginal gain of added replica | stale share | primary load (units/s)"); console.log("---------|---------|-----------------------|--------------------------------|-------------|-----------------------"); const W = 1, sentPerSec = routed(r, W) / WINDOW, replicaReads = (r.reads / WINDOW) - sentPerSec; let previous = null; for (const R of [1, 2, 3, 4, 5, 6, 7, 8]) { const perReplica = replicaReads / R; const margin = previous === null ? "-" : (previous - perReplica).toFixed(1) + "/s"; const b = stale(r, lagForReplicas(R)); const load = (r.writes / WINDOW) * LOAD_WRITE + sentPerSec * LOAD_READ + R * (r.writes / WINDOW) * LOAD_REPLICATION; console.log(`${s(R, 8)} | ${s(lagForReplicas(R).toFixed(2) + " s", 7)} | ${s(perReplica.toFixed(1), 21)} | ` + `${s(margin, 30)} | ${s(pct(b, r.reads), 11)} | ${s(load.toFixed(1), 23)}`); previous = perReplica; } console.log(`\nrun-independent: once the sticky window exceeds the replica lag, no requests are left that ` + `cannot see their own write; the cost is the number of reads sent to the primary.`); // Transfer cost: every out-of-region replica carries each write across the region boundary. Fictional unit. const RECORD_KB = 1.8, MONTH = 2592000, SIZE_GB = 220; const gbMonth = (r.writes / WINDOW) * RECORD_KB * MONTH / 1048576; console.log(`\nreplication stream: ${(r.writes / WINDOW).toFixed(0)} writes/s x ${RECORD_KB} KB = ` + `${gbMonth.toFixed(0)} GB/month, per replica.`); console.log("replica placement | unit/GB | 1 replica | 2 replicas | 3 replicas (fictional unit/month)"); console.log("------------------|----------|-----------|------------|-----------------------------------"); for (const [name, rate] of [["same region", 0], ["other region", 2]]) console.log(`${name.padEnd(17)} | ${s(rate, 8)} | ${s((rate * gbMonth).toFixed(0), 9)} | ` + `${s((2 * rate * gbMonth).toFixed(0), 10)} | ${s((3 * rate * gbMonth).toFixed(0), 7)}`); console.log(`irreversible item: a promoted replica does not return to its old replication link; rebuilding ` + `the link means a full ${SIZE_GB} GB copy.`); console.log(`within region at 95 MB/s: ${(SIZE_GB * 1024 / 95 / 3600).toFixed(2)} h; across regions ` + `at 30 MB/s: ${(SIZE_GB * 1024 / 30 / 3600).toFixed(2)} h + ${2 * SIZE_GB} fictional units.`); ``` ``` 420000 requests / 300 s: 92400 writes, 327600 reads. 2400 sessions, 50000 work orders. Seed 20260802. 31% of writes are read back by the same session. replica lag | stale reads | stale share | RYW misses | miss share ----------------|-------------|-------------|---------------------------|----------- 0.10 s | 208 | 0.06% | 0 | 0.00% 0.25 s | 5253 | 1.60% | 4755 | 1.45% 0.50 s | 14702 | 4.49% | 13733 | 4.19% 1.00 s | 30039 | 9.17% | 28093 | 8.58% 2.00 s | 31877 | 9.73% | 28093 | 8.58% 5.00 s | 37473 | 11.44% | 28093 | 8.58% routing rule, replica lag held at 0.55 s (15546 requests cannot see their own write): sticky window W | reads sent to primary | saved | remaining | read share shifted to primary -------------------|-----------------------|------------|-----------|------------------------------ 0.00 s | 0 | 0 | 15546 | 0.00% 0.25 s | 14986 | 5078 | 10468 | 4.57% 0.50 s | 33134 | 13843 | 1703 | 10.11% 1.00 s | 64155 | 15546 | 0 | 19.58% 2.00 s | 95648 | 15546 | 0 | 29.20% 5.00 s | 168901 | 15546 | 0 | 51.56% 15.00 s | 278984 | 15546 | 0 | 85.16% as replica count rises, the replication stream is pulled harder from the primary: lag 0.40 s (1 replica) -> 1.45 s (8 replicas). replicas | lag | reads/s per replica | marginal gain of added replica | stale share | primary load (units/s) ---------|---------|-----------------------|--------------------------------|-------------|----------------------- 1 | 0.40 s | 878.1 | - | 3.34% | 418.2 2 | 0.55 s | 439.1 | 439.1/s | 5.08% | 442.8 3 | 0.70 s | 292.7 | 146.4/s | 6.79% | 467.5 4 | 0.85 s | 219.5 | 73.2/s | 8.53% | 492.1 5 | 1.00 s | 175.6 | 43.9/s | 9.17% | 516.7 6 | 1.15 s | 146.4 | 29.3/s | 9.27% | 541.4 7 | 1.30 s | 125.5 | 20.9/s | 9.35% | 566.0 8 | 1.45 s | 109.8 | 15.7/s | 9.44% | 590.7 run-independent: once the sticky window exceeds the replica lag, no requests are left that cannot see their own write; the cost is the number of reads sent to the primary. replication stream: 308 writes/s x 1.8 KB = 1370 GB/month, per replica. replica placement | unit/GB | 1 replica | 2 replicas | 3 replicas (fictional unit/month) ------------------|----------|-----------|------------|----------------------------------- same region | 0 | 0 | 0 | 0 other region | 2 | 2741 | 5482 | 8223 irreversible item: a promoted replica does not return to its old replication link; rebuilding the link means a full 220 GB copy. within region at 95 MB/s: 0.66 h; across regions at 30 MB/s: 2.09 h + 440 fictional units. ``` #### Lag Is One Number, Its Reflection Is a Curve The first table shows how lag converts into requests. At 0.1 seconds, only 208 of 327,600 reads are stale; at half a second, 14,702; at one second, 30,039. The curve is not linear, because the stale-read count grows not with the lag itself but with **the write density that falls inside the lag window**. The right-hand column carries the real information. Of the 5,253 stale reads at 0.25 seconds of lag, 4,755 — 90% — are requests unable to see their own write. The reason is MD9: read-backs come right after the write, so almost every read that falls inside a narrow lag window is a read-back. As lag rises from 1 to 5 seconds, total stale reads climb from 30,039 to 37,473, but requests unable to see their own write hold steady at 28,093: since read-backs arrive within 900 ms at the latest, all of them are already stale once lag passes 1 second. Growing the lag further does not produce a new kind of bug — it just piles more inconsistency around the existing one. #### The Routing Rule Saves Requests, and Bills the Primary The second table holds lag fixed at 0.55 seconds and scans the sticky window. The rule rests on the one thing the application knows: after a session writes, its reads go to the primary instead of the replica for W seconds. At W = 0 there is no rule, and 15,546 requests cannot see their own write. As W rises to 0.25 seconds, 5,078 are saved and 10,468 remain — the window is narrower than the lag, so it is not enough. At W = 0.5, the remainder drops to 1,703. **At W = 1 second the remainder is zero**, and this line is run-independent: the moment the sticky window exceeds the replica lag, every read touched by the session's own write goes to the primary. The cost sits in the right-hand column. The W = 1 setting that zeroes out the remainder shifts 19.58% of reads to the primary: 64,155 of 327,600 reads. The point of opening the replicas was to unload the primary; the rule hands back a fifth of that gain. Pulling W up to 15 seconds for safety pushes the share to 85.16%, and the replicas sit idle. The right setting is not the largest W, it is the W **just above the lag**; since the lag depends on replica count, the two decisions are tied to each other. #### The Diminishing Return of Adding Replicas The third table raises replica count from 1 to 8. Reads per replica drop from 878.1 to 109.8; that much is expected. The gain column shows the unexpected part: the second replica lifts 439.1 reads/s off the primary, the third 146.4, the fourth 73.2, the eighth 15.7. The seventh and eighth replicas together bring in 8% of what the second replica gained. Two more columns turn this table on its head. **Lag grows with replica count** (MD10): 1.45 seconds at 8 replicas, and the stale-read share climbs from 3.34% to 9.44%. **Primary load grows too:** from 418.2 to 590.7 units, because the replication stream is sent to every replica. By the time the eighth replica is added, the primary carries 41% more load and gains only 15.7 requests/s on the read side. Past this point, adding a replica does not spread the load — it moves it. #### Transfer Cost and the Irreversible Item The last block prices replica placement. The replication stream is 308 writes/s x 1.8 KB, or 1,370 GB per replica each month: 0 fictional units in the same region, 2,741 in another region, 8,223 for three replicas. Spreading replicas across regions looks like a durability decision; on the bill it is a transfer item, and it scales with **write rate**, not data size. The irreversible item in this lesson is a subtle one. Once a replica is promoted to primary, it does not return to its old replication link; rebuilding that link means a full 220 GB copy: 0.66 hours within a region, 2.09 hours across regions, and 440 fictional units. Pressing the promote button is a one-second action; undoing it is not. #### Summary - As replica lag rises from 0.1 to 5 seconds, stale reads climb from 208 to 37,473; the curve is not linear, it tracks the write density that falls inside the lag window. - At narrow lag, 90% of stale reads are requests unable to see their own write; past 1 second of lag this count locks at 28,093, and only the inconsistency around it keeps growing. - Once the sticky window exceeds the lag (W = 1 s at 0.55 s of lag), requests unable to see their own write drop to zero; the cost is 19.58% of reads shifting to the primary. - Opening the window to 15 seconds saves the same 15,546 requests but carries 85.16% of reads to the primary: most of the gain from opening the replicas is handed back. - The return on adding replicas shrinks (439.1 -> 15.7 reads/s), and two costs rise: lag from 0.40 to 1.45 seconds, primary load from 418.2 to 590.7 units. - An out-of-region replica is a transfer item that scales with write rate (1,370 GB a month, 2,741 fictional units); a promoted replica does not return to its replication link, and rebuilding it is a full 220 GB copy. #### Next Step These two lessons measured the renting of a relational database, and most of the kept decisions turned out to be settings: a badly chosen sticky window can be changed, an excess replica can be closed. The next lesson asks the same question on the key-value side, where a single item turns the table upside down: data lands according to the chosen **partition key**, and once that key is chosen, which queries can be served is fixed. Three candidate keys will be run against the same real key distribution, the busiest partition's share and the access patterns the key does not support will be counted, and then the cost of changing the key will be computed in bytes and time. ### Managed Key-Value Service Source: https://academia.sh/en/courses/cloud-storage/managed-key-value-service Measuring the choice of partition key across three candidates against a real key distribution: requests per partition and the busiest partition's share for each candidate, and access patterns the key does not support counted as unserved queries; provisioned capacity compared against the per-request model on the same load profile, with the over-provisioning the hot partition forces computed; and the cost of changing the partition key worked out in duration and fictional units. In the previous two lessons, nearly every kept decision turned out to be a setting: a sticky window can be changed, a replica can be closed. In the managed key-value service, the table turns. Data lands according to the value of a field called the **partition key**, and once that key is chosen, which queries can be served is fixed. The mechanics of partitioning — whether the split is by range or by hash, how much data rebalancing moves — were measured in the Advanced Data Scaling course and are not repeated here. What this lesson measures is specific to the cloud context: what the key does to the **capacity model**, and the **irreversibility** of changing it. #### The Unserved Query If a query fixes the partition key by equality, the request goes straight to the relevant partition; if it does not, the service has no way to know where to look, and the only option is scanning the whole table. At this scale, a scan is not a query but a batch job; such an access pattern is counted as an **unserved query** and stands as its own column below. The second measure is how servable requests spread across partitions: in a load spread over sixteen partitions, the gap between requests per partition and what the busiest partition receives is the **hot partition** itself. #### Mechanics **MD13 — the fictional measurement network holds 40,000 meters, 12 regions, and 26,000 subscribers; each meter writes a reading every 15 minutes (96 a day), and retention is 90 days: 345.6M items, 180 bytes each, 62.2 GB.** Region shares run from 27 down to 1; the central region carries roughly a quarter of the meters. **MD14 — the access mix: a meter's last 24 hours is 38%, a single reading record 22%, a region's daily total 17%, a subscriber's latest readings 10%, today's threshold-exceeding readings 8%, the last hour's readings across the board 5%.** **MD15 — the service spreads data across 16 partitions; in provisioned capacity, the total is split equally across the partitions, and in this model a partition cannot exceed its own share. Real services soften that rigidity with adaptive capacity: a hot partition may go above its own share as long as it stays under the table's total capacity and the per-partition ceiling. The multipliers below are therefore an upper bound.** **MD16 — the request rate averages 1,200/s, and the daily profile is a fictional 24-hour multiplier array.** **MD17 — fictional price: provisioned capacity is 0.12 unit/capacity-hour, per-request is 0.25 unit/1,000 requests, a write unit is 5 times a read.** **MD18 — the key-change job runs at 12,000 items/s.** MD14 is the determinant; if the weights change, the unserved-request share moves in the same direction. ```js // md/model.mjs — MANAGED KEY-VALUE SERVICE model: partition key candidates are run over the // fictional regional measurement network's meter readings. Partitioning MECHANICS (range vs. // hash, rebalancing) were measured in M17/K05 and are not repeated; what is measured here is // how well the key's VALUE DISTRIBUTION fits the access pattern. Hand-written generator, seed visible. export const REGION = [["central", 27], ["north", 14], ["harbor", 11], ["industrial", 9], ["west", 8], ["east", 7], ["highland", 6], ["coast", 5], ["plain", 5], ["dam", 4], ["hill", 3], ["island", 1]]; export const METERS = 40000, SUBSCRIBERS = 26000, READINGS_PER_DAY = 96, RETENTION = 90, PARTITIONS = 16; export const ITEMS = METERS * READINGS_PER_DAY * RETENTION, ITEM_BYTES = 180; export const TODAY = RETENTION - 1; // The meter registry is a closure: given a meter_id its region is known, given a subscriber its meters are. export function registry(seed = 20260803) { let s = seed % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const total = REGION.reduce((t, [, w]) => t + w, 0); const regionOf = new Int32Array(METERS), subscriberOf = new Int32Array(METERS); const regionCount = new Array(REGION.length).fill(0), metersOfSubscriber = new Map(); for (let i = 0; i < METERS; i += 1) { let x = rand() * total, region = 0; for (let j = 0; j < REGION.length; j += 1) { if (x < REGION[j][1]) { region = j; break; } x -= REGION[j][1]; } regionOf[i] = region; regionCount[region] += 1; const sub = Math.floor(rand() * SUBSCRIBERS); subscriberOf[i] = sub; if (!metersOfSubscriber.has(sub)) metersOfSubscriber.set(sub, []); metersOfSubscriber.get(sub).push(i); } return { rand, regionOf, subscriberOf, regionCount, metersOfSubscriber }; } // An access pattern is defined by which fields it can FIX when building the query (A1..A6, per MD14); // if the partition key is not among those fields, the query is unserved. export const ACCESS = [38, 17, 22, 8, 10, 5]; export function generateRequests(reg, count = 300000) { const weight = ACCESS, total = weight.reduce((a, b) => a + b, 0); const requests = []; for (let i = 0; i < count; i += 1) { let x = reg.rand() * total, kind = 0; for (let j = 0; j < weight.length; j += 1) { if (x < weight[j]) { kind = j; break; } x -= weight[j]; } const meter = Math.floor(reg.rand() * METERS); const subscriber = reg.subscriberOf[Math.floor(reg.rand() * METERS)]; const past = Math.floor(reg.rand() * RETENTION); // A2 and A3 also look at the past const day = kind === 0 ? TODAY // A1 "last 24 hours" is always today : kind === 3 ? (reg.rand() < 0.7 ? TODAY : TODAY - Math.floor(reg.rand() * 7)) : past; requests.push({ kind, meter, region: reg.regionOf[meter], subscriber, day }); } return requests; } // Which partition-key VALUES a request touches; null = the key cannot be fixed, i.e. the // query is UNSERVED (the whole table is scanned). export function keysFor(q, candidate, reg) { const metersOf = () => reg.metersOfSubscriber.get(q.subscriber) ?? [q.meter]; if (candidate === "region") { if (q.kind === 3 || q.kind === 5) return null; if (q.kind === 4) return [...new Set(metersOf().map((i) => reg.regionOf[i]))].map((b) => "b" + b); return ["b" + q.region]; } if (candidate === "day") { if (q.kind === 0) return ["g" + q.day, "g" + Math.max(0, q.day - 1)]; return ["g" + (q.kind === 4 || q.kind === 5 ? TODAY : q.day)]; } if (q.kind === 1 || q.kind === 3 || q.kind === 5) return null; // meter_id cannot be fixed return q.kind === 4 ? metersOf().map((i) => "s" + i) : ["s" + q.meter]; } export const hash = (m) => { let h = 2166136261; for (const c of m) h = Math.imul(h ^ c.charCodeAt(0), 16777619); return (h >>> 0) % PARTITIONS; }; ``` ```js // md/measure.mjs — three partition key candidates, hot partition, unserved queries, capacity model import { registry, generateRequests, keysFor, hash, ACCESS, REGION, METERS, SUBSCRIBERS, READINGS_PER_DAY, RETENTION, PARTITIONS, ITEMS, ITEM_BYTES } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const pct = (a, b) => (100 * a / b).toFixed(2) + "%"; const reg = registry(); const requests = generateRequests(reg); const CANDIDATE = [["region | time", "region", REGION.length], ["day | meter_id#time", "day", RETENTION], ["meter_id | time", "meter_id", METERS]]; console.log(`${METERS} meters, ${REGION.length} regions, ${SUBSCRIBERS} subscribers, ${READINGS_PER_DAY} readings/day, ` + `${RETENTION} days retention -> ${(ITEMS / 1e6).toFixed(1)}M items, ${(ITEMS * ITEM_BYTES / 1e9).toFixed(1)} GB.`); console.log(`measured over ${requests.length} requests. Seed 20260803.\n`); const results = new Map(); console.log("partition key | distinct vals | unserved patterns | unserved requests | requests/partition (avg / busiest) | hot partition"); console.log("---------------------|---------------|------------------------|----------------------|-------------------------------------|--------------"); for (const [label, candidate, distinct] of CANDIDATE) { const load = new Array(PARTITIONS).fill(0); let unserved = 0; const patterns = new Set(); for (const q of requests) { const keys = keysFor(q, candidate, reg); if (keys === null) { unserved += 1; patterns.add(q.kind); continue; } for (const value of keys) load[hash(value)] += 1; // each value is a separate partition request } const total = load.reduce((a, b) => a + b, 0), busiest = Math.max(...load); results.set(candidate, { hot: busiest / total, unserved }); console.log(`${label.padEnd(20)} | ${s(distinct, 13)} | ${s(patterns.size + " / " + ACCESS.length, 22)} | ` + `${s(unserved + " (" + pct(unserved, requests.length) + ")", 20)} | ` + `${s(Math.round(total / PARTITIONS) + " / " + busiest, 35)} | ${s(pct(busiest, total), 11)}`); } console.log(`ideal hot-partition share for ${PARTITIONS} partitions is ${(100 / PARTITIONS).toFixed(2)}%; ` + `requests per partition were counted over the partitions touched by servable requests.`); // Capacity model. In provisioned capacity, the total is split equally across PARTITIONS shares; // a partition cannot exceed its own share, so the hot partition forces the total up. const BASELINE = 1200, PRICE_PROVISIONED = 0.12, PRICE_REQUEST = 0.25; const PROFILE = [0.22, 0.18, 0.16, 0.16, 0.20, 0.35, 0.70, 1.20, 1.75, 2.10, 2.20, 2.05, 1.80, 1.70, 1.80, 2.00, 2.30, 2.55, 2.45, 1.90, 1.30, 0.85, 0.50, 0.32]; const avg = PROFILE.reduce((a, b) => a + b, 0) / 24; const rate = PROFILE.map((p) => BASELINE * p / avg); const dailyRequests = rate.reduce((a, b) => a + b, 0) * 3600; const peak = Math.max(...rate); console.log(`\nload profile: average ${BASELINE} requests/s, peak ${peak.toFixed(0)} requests/s ` + `(${(peak / BASELINE).toFixed(2)}x), daily ${(dailyRequests / 1e6).toFixed(1)}M requests.`); console.log(`fictional price: provisioned capacity ${PRICE_PROVISIONED} unit/capacity-hour, per request ` + `${PRICE_REQUEST} unit/1000 requests.`); console.log("partition key | over-provision | provisioned (unit/day) | per-request | requests throttled at peak"); console.log("---------------------|-----------------|-------------------------|--------------|---------------------------"); for (const [label, candidate] of CANDIDATE) { const h = results.get(candidate).hot; const provisioned = PARTITIONS * h * peak * 24 * PRICE_PROVISIONED; // unthrottled provision = PARTITIONS * h * peak const ceiling = peak / PARTITIONS; // provisioning that only covers the total: C = peak const throttled = rate.reduce((t, r) => t + Math.max(0, h * r - ceiling) * 3600, 0); console.log(`${label.padEnd(20)} | ${s((PARTITIONS * h).toFixed(2) + "x", 15)} | ${s(provisioned.toFixed(0), 23)} | ` + `${s((dailyRequests / 1000 * PRICE_REQUEST).toFixed(0), 12)} | ${s((throttled / 1e6).toFixed(1) + "M", 29)}`); } // Irreversible item: changing the partition key means rewriting the data. const TRANSFER_RATE = 12000, WRITE_FACTOR = 5; const equivalent = ITEMS * (1 + WRITE_FACTOR), duration = ITEMS / TRANSFER_RATE / 3600; const cost = equivalent / 1000 * PRICE_REQUEST, dailyOpCost = dailyRequests / 1000 * PRICE_REQUEST; const exportSize = ITEMS * ITEM_BYTES / 1e9; console.log(`\nchanging the key: ${(ITEMS / 1e6).toFixed(1)}M items are read and rewritten under the new ` + `key (write is ${WRITE_FACTOR}x the read unit) -> equivalent requests ${(equivalent / 1e9).toFixed(2)}G.`); console.log(`cost ${(cost / 1000).toFixed(1)}K fictional units = ${(cost / dailyOpCost).toFixed(0)}x daily ` + `operations (${dailyOpCost.toFixed(0)} units); duration ${duration.toFixed(1)} h ` + `(${TRANSFER_RATE} items/s), and a double-write window for the whole duration.`); console.log(`leaving the service is cheaper: ${exportSize.toFixed(1)} GB export, over the internet at ` + `9 units/GB = ${(exportSize * 9).toFixed(0)} fictional units. Distinct value count (${REGION.length} for region) ` + `is the run-independent ceiling on partition count.`); ``` ``` 40000 meters, 12 regions, 26000 subscribers, 96 readings/day, 90 days retention -> 345.6M items, 62.2 GB. measured over 300000 requests. Seed 20260803. partition key | distinct vals | unserved patterns | unserved requests | requests/partition (avg / busiest) | hot partition ---------------------|---------------|------------------------|----------------------|-------------------------------------|-------------- region | time | 12 | 2 / 6 | 39168 (13.06%) | 18545 / 78103 | 26.32% day | meter_id#time | 90 | 0 / 6 | 0 (0.00%) | 25888 / 184840 | 44.62% meter_id | time | 40000 | 3 / 6 | 89856 (29.95%) | 16010 / 16312 | 6.37% ideal hot-partition share for 16 partitions is 6.25%; requests per partition were counted over the partitions touched by servable requests. load profile: average 1200 requests/s, peak 2389 requests/s (1.99x), daily 103.7M requests. fictional price: provisioned capacity 0.12 unit/capacity-hour, per request 0.25 unit/1000 requests. partition key | over-provision | provisioned (unit/day) | per-request | requests throttled at peak ---------------------|-----------------|-------------------------|--------------|--------------------------- region | time | 4.21x | 28978 | 25920 | 16.8M day | meter_id#time | 7.14x | 49127 | 25920 | 34.7M meter_id | time | 1.02x | 7010 | 25920 | 0.0M changing the key: 345.6M items are read and rewritten under the new key (write is 5x the read unit) -> equivalent requests 2.07G. cost 518.4K fictional units = 20x daily operations (25920 units); duration 8.0 h (12000 items/s), and a double-write window for the whole duration. leaving the service is cheaper: 62.2 GB export, over the internet at 9 units/GB = 560 fictional units. Distinct value count (12 for region) is the run-independent ceiling on partition count. ``` #### Three Candidates, Three Different Failures No candidate comes out ahead; each one sinks in a different column. **`region` matches the operating structure and produces a hot partition.** While the average is 18,545 requests per partition, the busiest partition takes 78,103: 26.32%, four times the ideal 6.25%. The reason is that one of twelve distinct values carries a quarter of the meters; adding partitions does not fix this, because a single key value cannot be split. **`day` serves every query and produces the worst hot partition.** Unserved patterns are zero — the only zero in the table. In exchange, today's partition takes 184,840 requests, seven times the average, 44.62% of requests: on a monotonically increasing key, most new writes and reads pile onto a single value. **`meter_id` balances flawlessly and cannot serve nearly a third of requests.** Average 16,010, busiest partition 16,312 — the hot partition sits at 6.37%, within 2% of the ideal. Against that, three of the six patterns — the region total, the threshold scan, and the last hour's readings — never fix the key at all: 29.95% of requests, 89,856 of them. These do not run slowly; **they do not run at all**. #### The Hot Partition Passes into the Bill Through the Capacity Model The second table converts the hot partition into money. The total allocated splits equally across partitions and no partition may exceed its own share (MD15); if the hot partition's share is h, the capacity that must be provisioned to avoid throttling is 16 x h x peak. On a service that offers adaptive capacity the factor is smaller but not gone — the per-partition ceiling and the total capacity limit stay in place. The first column gives this factor: 1.02x for `meter_id`, 4.21x for `region`, 7.14x for `day`. On a key with a hot partition, five-sixths of the provisioned capacity never gets used: **what the capacity buys is not work, it is imbalance.** The daily cost follows from this. With `meter_id`, provisioned capacity is 7,010 fictional units against 25,920 for the per-request model — the provisioned model is 3.7 times cheaper. With `region`, provisioned rises to 28,978 and overtakes the per-request model; with `day` it reaches 49,127. **What decides which capacity model is cheaper is not the load, it is the partition key.** The per-request model is indifferent to imbalance; the provisioned model bills it by its factor. The last column shows what happens when provisioning is thin: allocate just enough to cover the total peak (2,389 units) and `meter_id` is never throttled, `region` throttles 16.8M requests a day, `day` throttles 34.7M. The total capacity is sufficient; it is only the partition that gets throttled. #### The Cost of Changing the Key Everything measured so far looks like a choice; the last block counts why it is not. Changing the partition key is not a setting change: all 345.6M items are read and rewritten under the new key. Since a write unit is five times a read, the equivalent request count is 2.07 billion, and the fictional cost is 518.4 thousand units — **20 times the daily operating cost.** The duration is 8 hours, and through that whole span the application writes to two tables at once, because writes arriving while the migration runs still need somewhere to land. The major version upgrade on the relational side was also one-way, but the data stayed in place; here, reversing means rewriting the data itself. Leaving the service outright is cheaper: a 62.2 GB export and 560 fictional units. **Changing the key costs 900 times more than leaving the service** — what binds you is not the provider, it is the schema. #### Summary - With the `region` key, the hot partition sits at four times the ideal share (26.32%), and twelve distinct values cap the partition count; adding partitions does not fix the imbalance. - `day` serves all six access patterns, but today's partition takes 184,840 requests (44.62%); `meter_id` balances flawlessly at 16,010 against 16,312, but cannot serve three of the six patterns — 29.95% of requests. - The hot partition converts into over-provisioning in provisioned capacity: 1.02x / 4.21x / 7.14x. What decides which capacity model is cheaper is the partition key, not the load (7,010 against 25,920 fictional units). - Even provisioning that covers the total peak throttles 16.8M and 34.7M requests a day on a hot key. - Changing the key means rewriting 345.6M items: 8 hours, 518.4 thousand fictional units, and an 8-hour double-write window — 900 times the cost of leaving the service outright. #### Next Step The table leaves one gap. On `meter_id`, the best-balanced key, 29.95% of requests go unserved, and the fix is not changing the key — that door closed at 518.4 thousand fictional units. Managed key-value services offer a **secondary index** for this gap: it places the same data a second time, under a different key. The next lesson counts how many of the three unserved patterns a secondary index recovers, and what it costs in exchange — extra writes, extra storage, lag time — and models a **change stream** with a consumer, measuring event order, reprocessing, and the missed window once the stream's retention runs out. ### Secondary Indexes and Streams Source: https://academia.sh/en/courses/cloud-storage/secondary-indexes-and-streams Recovering the previous lesson's three unserved access patterns with a secondary index, and counting what that costs in return: each index's recovered requests, the extra write cost it adds, the stale-result share caused by storage and index lag, the trade-off between projection choice and round trips to the base table, and a change stream modeled with a consumer — measuring event order, reprocessed events, and the missed window once the stream's retention runs out. The previous lesson found `meter_id` to be the best-balanced partition key candidate, and left a debt on the same line: three of the six access patterns — the region's daily total, the threshold scan, the last hour's readings — could never fix the key, so they went entirely unserved: 30% of requests. Since changing the key is a door that closed at 518.4 thousand fictional units, only one path remains: placing the same data **once more, under a different key**. This is called a **secondary index**; building the index, keeping it updated, and partitioning it are delegated decisions. What is not delegated is the choice itself: which key, which attributes, how many indexes. #### Mechanics **MD19 — the base table is built on the previous lesson's `meter_id | time` key; the three unserved patterns are A2 at 17%, A4 at 8%, and A6 at 5%.** **MD20 — three secondary index candidates: `region | day#meter_id`, `day | reading_value`, and `day#hour | meter_id`; the second is a sparse index and carries only items past the threshold (4 in 1,000 of the items).** **MD21 — the index updates asynchronously, with 0.8 seconds of lag.** **MD22 — the projection options are 48, 96, and 180 bytes per item; the fictional storage price is 0.9 unit/GB-month, a write unit is 5 times a read, and the per-request price is 0.25 unit/1,000 requests, same as the previous lesson.** **MD23 — the change stream has 16 shards and 24 hours of retention; the event rate equals the write rate (44.4 events/s), and consumer capacity is 55 events/s.** **MD24 — the consumer checkpoints its position every C seconds, and the network averages 0.4 crashes a day.** The capacity margin in MD23 is the determinant; write 45 instead of 55 and the catch-up-time column blows up. ```js // md/model.mjs — SECONDARY INDEX AND CHANGE STREAM model. Builds on the previous lesson's base // table (partition key meter_id); the three unserved patterns found there are the input. The // randomness lives in the event generator, hand-written, seed visible. export const ITEMS = 345_600_000, DAILY_WRITES = 3_840_000, DAILY_REQUESTS = 103_700_000; export const WRITE_FACTOR = 5, PRICE_REQUEST = 0.25, PRICE_GB_MONTH = 0.9, ITEM_BYTES = 180; export const LAMBDA = DAILY_WRITES / 86400; // events/s = writes/s // [name, pattern recovered, pattern weight, indexed item fraction, writes/s hitting that range] export const INDEX = [ ["region | day#meter_id", "A2 region daily total", 17, 1, LAMBDA * 0.27], ["day | reading_value", "A4 threshold-exceeding readings", 8, 0.004, LAMBDA * 0.004], ["day#hour | meter_id", "A6 last hour", 5, 1, LAMBDA], ]; export const PROJECTION = [["key only", 48], ["selected attributes", 96], ["all attributes", 180]]; export const writeCostMonth = (fraction) => DAILY_WRITES * fraction * 30 * WRITE_FACTOR / 1000 * PRICE_REQUEST; export const storageCostMonth = (fraction, bytes) => ITEMS * fraction * bytes / 1e9 * PRICE_GB_MONTH; export const staleItems = (writeRate, L) => writeRate * L; export const staleQueryShare = (writeRate, L) => 1 - Math.exp(-writeRate * L); // Change stream: every write is an event. Event ORDER is preserved only within the same // partition key; when the consumer reads shards in turn, the overall time order breaks. export function readStream({ count = 60000, shards = 16, batch = 64, seed = 20260804 } = {}) { let s = seed % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const hash = (x) => { let h = 2166136261; for (const c of String(x)) h = Math.imul(h ^ c.charCodeAt(0), 16777619); return (h >>> 0) % shards; }; const buckets = Array.from({ length: shards }, () => []); let t = 0; for (let i = 0; i < count; i += 1) { // events are generated in time order t += rand() * 2 / LAMBDA; const meter = Math.floor(rand() * 40000); buckets[hash(meter)].push({ t, meter }); } const consumed = []; // consumer: `batch` events per shard for (let d = 0; consumed.length < count; d += 1) for (let p = 0; p < shards; p += 1) for (const o of buckets[p].slice(d * batch, (d + 1) * batch)) consumed.push(o); let outOfOrder = 0, maxSkew = 0, highWatermark = -Infinity; const lastByKey = new Map(); let withinKeyOutOfOrder = 0; for (const o of consumed) { if (o.t < highWatermark) { outOfOrder += 1; maxSkew = Math.max(maxSkew, highWatermark - o.t); } highWatermark = Math.max(highWatermark, o.t); if ((lastByKey.get(o.meter) ?? -Infinity) > o.t) withinKeyOutOfOrder += 1; lastByKey.set(o.meter, o.t); } return { count, shards, outOfOrder, maxSkew, withinKeyOutOfOrder, duration: t }; } // Pause: the consumer stops for P seconds, then catches up at K events/s. The stream's // retention is R; every second beyond P > R has its events fall off the stream and NEVER RETURN. export const K = 55, R = 24 * 3600, CRASH_RATE = 0.4; export function pause(P) { const backlog = LAMBDA * P; const catchup = K > LAMBDA ? backlog / (K - LAMBDA) : Infinity; return { backlog, catchup, missed: LAMBDA * Math.max(0, P - R), total: P + catchup }; } export const reprocessed = (C) => LAMBDA * C * CRASH_RATE; ``` ```js // md/measure.mjs — the query and cost a secondary index recovers; the change stream's consumer side import { INDEX, PROJECTION, writeCostMonth, storageCostMonth, staleItems, staleQueryShare, readStream, pause, reprocessed, ITEMS, ITEM_BYTES, DAILY_WRITES, DAILY_REQUESTS, LAMBDA, K, R, CRASH_RATE } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const L = 0.8; console.log(`base table: ${(ITEMS / 1e6).toFixed(1)}M items, ${(DAILY_WRITES / 1e6).toFixed(2)}M writes/day, ` + `${(DAILY_REQUESTS / 1e6).toFixed(1)}M requests/day. Unserved patterns: A2 17%, A4 8%, A6 5% (30% total).`); console.log(`index lag ${L} s, event rate ${LAMBDA.toFixed(1)}/s.\n`); console.log("secondary index | recovered req/day | items indexed | extra writes (unit/mo) | extra storage (unit/mo) | stale items | stale result"); console.log("----------------------|----------------------|----------------|------------------------|-------------------------|-------------|-------------"); let recoveredT = 0, writeT = 0, storageT = 0; for (const [name, , weight, fraction, rate] of INDEX) { const recovered = DAILY_REQUESTS * weight / 100; const write = writeCostMonth(fraction), storage = storageCostMonth(fraction, 180); recoveredT += recovered; writeT += write; storageT += storage; console.log(`${name.padEnd(21)} | ${s((recovered / 1e6).toFixed(1) + "M (" + weight + "%)", 20)} | ` + `${s((ITEMS * fraction / 1e6).toFixed(1) + "M", 14)} | ${s(write.toFixed(0), 22)} | ${s(storage.toFixed(1), 23)} | ` + `${s(staleItems(rate, L).toFixed(1), 11)} | ${s((100 * staleQueryShare(rate, L)).toFixed(1) + "%", 12)}`); } console.log(`total: ${(recoveredT / 1e6).toFixed(1)}M requests recovered a day (unserved 30% -> 0%); ` + `cost ${(writeT + storageT).toFixed(0)} fictional units/month.`); console.log(`extra writes are ${(writeT / storageT).toFixed(0)}x extra storage: the secondary index's ` + `cost is not the bytes it holds, it is every write happening a second time.`); console.log("\nprojection choice (for the A2 index only, 345.6M items):"); console.log("projection | bytes/item | index size | storage (unit/mo) | round trip to base "); console.log("---------------------|-----------------|--------------|--------------------|--------------------"); for (const [name, bytes] of PROJECTION) console.log(`${name.padEnd(20)} | ${s(bytes, 15)} | ${s((ITEMS * bytes / 1e9).toFixed(1) + " GB", 12)} | ` + `${s(storageCostMonth(1, bytes).toFixed(0), 18)} | ${s(bytes === 180 ? "not needed" : "1 read per item", 19)}`); const a = readStream({ batch: 24 }); console.log(`\nchange stream: ${a.count} events, ${a.shards} shards, a ${a.duration.toFixed(0)} s window. Seed 20260804.`); console.log(`events out of order in global time: ${a.outOfOrder} (${(100 * a.outOfOrder / a.count).toFixed(1)}%), ` + `largest time skew ${a.maxSkew.toFixed(2)} s.`); console.log(`events out of order within the same partition key: ${a.withinKeyOutOfOrder}. The guarantee lives here, not in the global order.`); console.log(`reprocessing (checkpoint interval C, ${CRASH_RATE} crashes a day):`); console.log("C | 1 s | 5 s | 15 s | 60 s | 300 s "); console.log("event | " + [1, 5, 15, 60, 300].map((C) => s(reprocessed(C).toFixed(0), C < 10 ? 4 : C < 100 ? 5 : 6)).join(" | ")); console.log(`\npause: consumer capacity ${K} events/s, event rate ${LAMBDA.toFixed(1)}/s ` + `-> catch-up factor ${(K / (K - LAMBDA)).toFixed(2)}. Stream retention ${R / 3600} h.`); console.log("pause | backlog | catch-up | outage to full catch-up | missed "); console.log("----------|---------------|----------------|--------------------------|----------------"); for (const P of [900, 3600, 4 * 3600, 12 * 3600, 24 * 3600, 30 * 3600]) { const d = pause(P); console.log(`${s((P / 3600).toFixed(2) + " h", 9)} | ${s((d.backlog / 1e3).toFixed(0) + "K", 13)} | ` + `${s((d.catchup / 3600).toFixed(2) + " h", 14)} | ${s((d.total / 3600).toFixed(2) + " h", 24)} | ` + `${s(d.missed === 0 ? "0" : (d.missed / 1e3).toFixed(0) + "K", 14)}`); } console.log(`run-independent: events are missed only once the pause exceeds the stream's retention ` + `(${R / 3600} h) and cannot be recovered; catch-up time does not change this, it only lengthens it.`); // Transfer cost: if the consumer is in another region, every event crosses the region boundary // (an event = the item's before and after image, i.e. twice the item size). Fictional unit. const gbMonth = DAILY_WRITES * 2 * ITEM_BYTES * 30 / 1e9; console.log(`\ntransfer: ${gbMonth.toFixed(1)} GB/month of event traffic. Consumer in the same region: 0, ` + `another region: ${(gbMonth * 2).toFixed(0)}, outside the region: ${(gbMonth * 9).toFixed(0)} fictional units/month.`); ``` ``` base table: 345.6M items, 3.84M writes/day, 103.7M requests/day. Unserved patterns: A2 17%, A4 8%, A6 5% (30% total). index lag 0.8 s, event rate 44.4/s. secondary index | recovered req/day | items indexed | extra writes (unit/mo) | extra storage (unit/mo) | stale items | stale result ----------------------|----------------------|----------------|------------------------|-------------------------|-------------|------------- region | day#meter_id | 17.6M (17%) | 345.6M | 144000 | 56.0 | 9.6 | 100.0% day | reading_value | 8.3M (8%) | 1.4M | 576 | 0.2 | 0.1 | 13.3% day#hour | meter_id | 5.2M (5%) | 345.6M | 144000 | 56.0 | 35.6 | 100.0% total: 31.1M requests recovered a day (unserved 30% -> 0%); cost 288688 fictional units/month. extra writes are 2572x extra storage: the secondary index's cost is not the bytes it holds, it is every write happening a second time. projection choice (for the A2 index only, 345.6M items): projection | bytes/item | index size | storage (unit/mo) | round trip to base ---------------------|-----------------|--------------|--------------------|-------------------- key only | 48 | 16.6 GB | 15 | 1 read per item selected attributes | 96 | 33.2 GB | 30 | 1 read per item all attributes | 180 | 62.2 GB | 56 | not needed change stream: 60000 events, 16 shards, a 1349 s window. Seed 20260804. events out of order in global time: 56199 (93.7%), largest time skew 94.73 s. events out of order within the same partition key: 0. The guarantee lives here, not in the global order. reprocessing (checkpoint interval C, 0.4 crashes a day): C | 1 s | 5 s | 15 s | 60 s | 300 s event | 18 | 89 | 267 | 1067 | 5333 pause: consumer capacity 55 events/s, event rate 44.4/s -> catch-up factor 5.21. Stream retention 24 h. pause | backlog | catch-up | outage to full catch-up | missed ----------|---------------|----------------|--------------------------|---------------- 0.25 h | 40K | 1.05 h | 1.30 h | 0 1.00 h | 160K | 4.21 h | 5.21 h | 0 4.00 h | 640K | 16.84 h | 20.84 h | 0 12.00 h | 1920K | 50.53 h | 62.53 h | 0 24.00 h | 3840K | 101.05 h | 125.05 h | 0 30.00 h | 4800K | 126.32 h | 156.32 h | 960K run-independent: events are missed only once the pause exceeds the stream's retention (24 h) and cannot be recovered; catch-up time does not change this, it only lengthens it. transfer: 41.5 GB/month of event traffic. Consumer in the same region: 0, another region: 83, outside the region: 373 fictional units/month. ``` #### The Index Recovers the Query, and Bills the Write The three indexes recover all three unserved patterns: 31.1M requests come back a day, and the unserved share drops from 30% to zero. The bill lands on a single column. **Extra writes are 2,572 times extra storage.** A full index brings 144,000 fictional units of extra writes a month, while the same index's 62.2 GB of storage costs only 56 units. The reason is that an index is less a copy than a **second write path**: every item that goes to the base table is also written to the index. A design that adds two full indexes triples the data layer's write cost. **The sparse index is the exception to this rule.** `day | reading_value` carries only the items past the threshold — 4 in 1,000 — and recovers the same requests for 576 units a month: one two-hundred-fiftieth of a full index. What sets an index's cost is not how many queries it recovers, but **how many writes it touches.** The last two columns count the result of asynchronous updates. Because the index sits 0.8 seconds behind, at any moment the `region` index is missing 9.6 items and the hour index 35.6; the region-total and last-hour queries come back missing something on almost every run (100%), but the missing share is small. The real warning is in the middle row: 13.3% of threshold scans miss a reading that just crossed the threshold — one scan in eight misses a fresh alert. The projection table gives the second trade-off: the key-only index runs 16.6 GB and 15 units a month, the all-attributes index 62.2 GB and 56 units. The 41-unit gap in between is the price of not making an extra read to the base table for every item the index returns, and at a scale of 17.6M queries a day, those reads are far more expensive. #### The Change Stream's Three Numbers The same data has a second consumer: the **change stream**, which emits every write as an event. The consumer that produces billing summaries feeds on this stream, and it has to know three things. **The order guarantee lives within the partition key, not across the whole stream.** 56,199 of 60,000 events — 93.7% — appear out of order to the consumer in global time, with a maximum time skew of 94.73 seconds. Within the same meter's events, the out-of-order count is zero: that is where the guarantee lives. A consumer that assumes two different meters' events arrive in time order is wrong. **Delivery is at least once, not exactly once.** When the consumer crashes, it resumes from the last saved position; at a 1-second checkpoint interval, 18 events a day are processed twice, at 300 seconds, 5,333. This count is not a bug — it is the range across which the consumer needs to be **idempotent**: processing the same event a second time should not add a second line item to the bill. **A missed window cannot be recovered.** The last table stops the consumer for P seconds. A 12-hour pause piles up 1.92M events, and the consumer's catch-up takes 50.53 hours — 62.53 hours from outage to fully caught up. Against that, **missed events are zero**, because the consumer always reads the oldest event first and no event waits longer than the pause itself. The limit sits in exactly one place: the moment the pause exceeds the stream's retention. At a 30-hour pause, 960K events fall off the stream and never come back. This adds a new line to the course's list of irreversible items — and for the first time, what cannot be reversed is not a decision, it is **a span of time**. The transfer line is short: 41.5 GB/month of event traffic is free in the same region, 83 in another region, 373 fictional units outside the region — an item that scales with write rate. #### Summary - Three secondary indexes recover all three unserved patterns (31.1M requests a day, 30% down to 0%), and the cost is writes, not storage: extra writes run **2,572 times** extra storage (144,000 against 56 fictional units/month); the sparse index recovers the same queries for 576 units. - The asynchronous index sits 0.8 seconds behind: 9.6 items are missing in the region index, 35.6 in the hour index, and 13.3% of threshold scans miss a reading that just crossed the threshold. - Projection is a trade-off between storage and reads: key-only runs 16.6 GB and 15 units, all attributes 62.2 GB and 56 units — but the first adds one base-table read for every result item. - In the change stream, order is preserved only within the partition key: 93.7% of events appear out of order in the global order (largest skew 94.73 s), zero out-of-order pairs within a key. - The missed window is the one irreversible item: no loss until the pause exceeds retention, and past that (30 h) 960K events fall off and never return. #### Next Step These four lessons measured both ends of the data layer, and both showed the same thing: handing off a decision is cheap, reversing it is expensive. But every request still hit disk. Yet the measurement network's most frequent question comes from a very narrow set — the latest reading, the meter registry, the open-alarm list — and reading those from the base table every time means producing the same answer thousands of times over the course of a day. The next lesson takes on the managed cache service, which moves that repetition into an in-memory layer, and counts only what **being a service** brings: which decision is delegated, what the scaling unit is, and how wide the data-loss window opens when a node is lost. ### Managed Cache Service Source: https://academia.sh/en/courses/cloud-storage/managed-cache-service Measuring the renting of the in-memory layer as a service: two scaling axes giving the same hit rate and the same bill, the 100% hit-rate row turning out the most expensive, the 3,370,000 keys lost when a node changes putting the purchased capacity 325% over, warm-up not shortening with shard count, and a cache in another zone writing a 722-unit transfer item and coming out pricier than no cache at all. The previous lesson widened the access pattern: the secondary index and the change stream reopened a share of the queries the partition key had closed off. But every reopened query still lands on the data layer. In-memory stores and eviction policies were measured earlier; this lesson does not repeat that mechanics. Here the in-memory layer is a **rented service**, and the three things measured are what the scaling unit is, what is lost when a node changes, and what the cache's location writes onto the bill. - **MD25.** The fictional regional measurement network's read set is three families: latest meter reading (2.4 million keys, 180 bytes), invoice summary (600 thousand, 2,400 bytes), work-order record (4 million, 600 bytes); 4.272 GB total. - **MD26.** Each family is modeled with three popularity tiers. The model fills the cache in order of per-key probability, does not count first-touch misses, and uses no randomness. - **MD27.** Reads average four hundred, peak three thousand requests/s; there is also a steady four hundred requests/s of writes, and writes never touch the cache. A month is counted as 2,592,000 seconds. - **MD28.** Prices are fictional, the currency's name is a fictional unit: a cache node runs sixty units per GB a month; read capacity is bought in hundred-requests/s steps, and a step is thirty units a month. The provider opens up **75%** of the node's memory for use. - **MD29.** Latencies are fictional: the data layer is 6 ms, a managed cache in the same zone 0.8 ms, in another zone 1.9 ms, in-process 0.02 ms. Cross-zone transfer is one unit per GB. - **MD30.** The meter collector accumulates its minute-by-minute totals in the cache and flushes every five seconds; the cache copy's replication lag is 0.25 seconds. Both are fictional. ```js // cache/cache.mjs — model: what changes when the in-memory layer is rented as a service. // Eviction policies were measured in M17/K06 and are not repeated. The working set is modeled // with popularity tiers, no randomness; prices are fictional and 1 GB is counted as 1e9 bytes. const TIER = [ // [family, tier, keys, bytes, access share] ["meter", "hot", 150000, 180, 0.15], ["meter", "warm", 850000, 180, 0.20], ["meter", "cold", 1400000, 180, 0.16], ["invoice", "hot", 18000, 2400, 0.10], ["invoice", "warm", 140000, 2400, 0.09], ["invoice", "cold", 442000, 2400, 0.04], ["workorder", "hot", 80000, 600, 0.06], ["workorder", "warm", 620000, 600, 0.10], ["workorder", "cold", 3300000, 600, 0.10], ]; const ORDER = [...TIER].sort((a, b) => b[4] / b[2] - a[4] / a[2]); // probability per key const fill = (capacity) => { // the cache fills from the most-requested key down let remaining = capacity, keys = 0, hitRate = 0; const retained = []; for (const [, , n, bytes, share] of ORDER) { const taken = Math.min(n, Math.floor(remaining / bytes)); if (taken === 0) break; remaining -= taken * bytes; keys += taken; hitRate += share * taken / n; retained.push([share * taken / n, share / n]); // [share retained, probability per key] } return { keys, hitRate, retained }; }; const PEAK = 3000, WRITES = 400, USABLE_SHARE = 0.75, GB_PRICE = 60; const capacityFee = (toBackend) => Math.ceil((toBackend + WRITES) / 100) * 30; const columns = (widths, left = 1) => (row) => row.map((d, i) => String(d)[i < left ? "padEnd" : "padStart"](widths[i])).join(""); const fmt1 = columns([14, 5, 16, 14, 11, 20, 10, 8, 15]); console.log(fmt1(["node x GB", "GB", "usable", "keys(k)", "hit(%)", "to backend(req/s)", "capacity", "node", "total(unit)"])); for (const [label, nodes, gbPerNode] of [["no cache", 0, 0], ["1 x 1", 1, 1], ["1 x 2", 1, 2], ["2 x 1", 2, 1], ["1 x 4", 1, 4], ["1 x 8", 1, 8]]) { const o = nodes ? fill(nodes * gbPerNode * 1e9 * USABLE_SHARE) : { keys: 0, hitRate: 0 }; const toBackend = Math.round(PEAK * (1 - o.hitRate)); const cap = capacityFee(toBackend), nodeFee = nodes * gbPerNode * GB_PRICE; console.log(fmt1([label, nodes * gbPerNode, (nodes * gbPerNode * USABLE_SHARE).toFixed(2), (o.keys / 1e3).toFixed(0), (o.hitRate * 100).toFixed(1), toBackend, cap, nodeFee, cap + nodeFee])); } // --- When a node changes ----------------------------------------------------------- // 2 GB layout. Since the hash spreads key names, every shard gets an equal share of every tier. const BASE = fill(2 * 1e9 * USABLE_SHARE), TO_BACKEND = Math.round(PEAK * (1 - BASE.hitRate)) + WRITES; const PURCHASED = capacityFee(TO_BACKEND - WRITES) / 30 * 100, PENDING = WRITES * 5, LAG = 0.25; const fmt2 = columns([16, 18, 16, 20, 12, 17]); console.log("\n" + fmt2(["shards", "lost keys", "lost access", "to backend(req/s)", "overage(%)", "pending records"])); for (const [shards, replicated] of [[1, 0], [2, 0], [4, 0], [8, 0], [16, 0], [4, 1]]) { const lostKeys = Math.round((replicated ? TO_BACKEND * LAG : BASE.keys) / shards); const lostAccess = replicated ? 0 : BASE.hitRate / shards; const toBackend = Math.round(PEAK * (1 - BASE.hitRate + lostAccess)) + WRITES; console.log(fmt2([replicated ? `${shards} (replicated)` : shards, lostKeys, (lostAccess * 100).toFixed(1), toBackend, ((toBackend / PURCHASED - 1) * 100).toFixed(0), Math.round((replicated ? WRITES * LAG : PENDING) / shards)])); } const recovered = (t) => BASE.retained.reduce((a, [share, r]) => a + share * (1 - Math.exp(-r * PEAK * t)), 0); const warmup = (target) => { let lo = 1, hi = 1e6; for (let k = 0; k < 60; k++) { const t = Math.sqrt(lo * hi); if (recovered(t) < target * BASE.hitRate) lo = t; else hi = t; } return hi.toFixed(0); }; console.log(`warm-up, independent of shard count: 50% ${warmup(0.5)} s, 90% ${warmup(0.9)} s`); // --- Where the cache sits ----------------------------------------------------------- const DATA_MS = 6, ZONE_PRICE = 1, MONTH = 2592000; // ms; cross-zone unit/GB; seconds/month const AVG_BYTES = TIER.reduce((a, [, , , bytes, share]) => a + bytes * share, 0); const fmt3 = columns([12, 26, 11, 14, 11, 11, 10, 15], 2); const row = (set, label, h, ms, reads, bytes, peak, writes, cacheFee, remote) => { const toBackend = Math.round(peak * (1 - h)); const capacity = Math.max(1, Math.ceil((toBackend + writes) / 100)) * 30; const transfer = (remote ? h * reads * bytes / 1e9 : 0) * ZONE_PRICE; console.log(fmt3([set, label, (h * 100).toFixed(1), (ms + (1 - h) * DATA_MS).toFixed(2), cacheFee, capacity, transfer.toFixed(0), (cacheFee + capacity + transfer).toFixed(0)])); }; console.log("\n" + fmt3(["set", "layout", "hit(%)", "latency(ms)", "cache", "capacity", "transfer", "total(unit)"])); const FULLSET = (label, h, ms, cacheFee, remote) => row("full set", label, h, ms, 400 * MONTH, AVG_BYTES, PEAK, WRITES, cacheFee, remote); FULLSET("no cache", 0, 0, 0, false); FULLSET("in-process 12 x 256 MB", fill(256e6).hitRate, 0.02, 0, false); FULLSET("managed 2 GB same zone", BASE.hitRate, 0.8, 2 * GB_PRICE, false); FULLSET("managed 2 GB other zone", BASE.hitRate, 1.9, 2 * GB_PRICE, true); // Rate table: 20 thousand keys x 2000 bytes = 40 MB, 320 million reads a month, peak 900/s. const RATETABLE = (label, h, ms, cacheFee, remote) => row("rate table", label, h, ms, 320e6, 2000, 900, 0, cacheFee, remote); RATETABLE("in-process 12 x 40 MB", 1, 0.02, 0, false); RATETABLE("managed 1 GB same zone", 1, 0.8, GB_PRICE, false); RATETABLE("managed 1 GB other zone", 1, 1.9, GB_PRICE, true); ``` ``` node x GB GB usable keys(k) hit(%) to backend(req/s) capacity node total(unit) no cache 0 0.00 0 0.0 3000 1020 0 1020 1 x 1 1 0.75 1476 63.8 1085 450 60 510 1 x 2 2 1.50 3370 87.0 390 240 120 360 2 x 1 2 1.50 3370 87.0 390 240 120 360 1 x 4 4 3.00 4880 93.6 193 180 240 420 1 x 8 8 6.00 7000 100.0 0 120 480 600 shards lost keys lost access to backend(req/s) overage(%) pending records 1 3370000 87.0 3400 325 2000 2 1685000 43.5 2095 162 1000 4 842500 21.8 1442 80 500 8 421250 10.9 1116 40 250 16 210625 5.4 953 19 125 4 (replicated) 49 0.0 790 -1 25 warm-up, independent of shard count: 50% 539 s, 90% 3514 s set layout hit(%) latency(ms) cache capacity transfer total(unit) full set no cache 0.0 6.00 0 1020 0 1020 full set in-process 12 x 256 MB 34.7 3.94 0 720 0 720 full set managed 2 GB same zone 87.0 1.58 120 240 0 360 full set managed 2 GB other zone 87.0 2.68 120 240 722 1082 rate table in-process 12 x 40 MB 100.0 0.02 0 30 0 30 rate table managed 1 GB same zone 100.0 0.80 60 30 0 90 rate table managed 1 GB other zone 100.0 1.90 60 30 640 730 ``` #### Two Axes, One Curve The managed cache service offers two scaling levers: growing the node and adding nodes. The first table puts them side by side, and the first reading is this: **no difference.** Two GB is reached with one node just as well as with two; both ways the cache holds 3,370,000 keys, the hit rate is 87%, the bill is 360 units. The scaling-unit decision **cannot be made on hit rate and price alone.** The bill itself is the second reading. The no-cache layout runs 1,020 units; one GB lifts the hit rate to 63.8% and drops the bill to 510, two GB to 360. Then the curve **turns**: 420 at four GB, 600 at eight GB. The 100%-hit-rate row is **67% pricier** than the two-GB row; the remaining 120 units goes to the four hundred requests/s of writes that never touch the cache at all, and there is no cache-side way to bring that down. The usable column is the third reading: buy eight GB and six GB gets used, the remaining quarter goes into the provider's ledger. Since the working set is 4.272 GB, the node needed to hold all of it in memory is not four GB, it is eight. #### When a Node Changes A node change is a routine event — a version upgrade, a patch window, or a failed node being replaced — and the cache is not durable: a changed node's memory comes back empty. In a single-shard layout, **all** 3,370,000 keys are lost. Requests hitting the backend rise from 790 to 3,400, landing **325% over** the purchased 800 requests/s capacity. This is not a crash, it is throttling: the data layer turns away the excess, and the dashboard comes back empty. Shard count brings down the wave's **height**: overage is 80% at four shards, 19% at sixteen. The number falls, but **it never zeroes out at any shard count.** The last row shows the one fix: add a replica to every shard and a node change turns into a failover, and lost keys drop to forty-nine — the keys written within the quarter-second replication lag. The cost is read off the first table: the node fee rises from 120 to 240, the total from 360 to 480. The last column is this lesson's **irreversible item**. The collector accumulates its minute-by-minute total in the cache and writes it out every five seconds; at any moment two thousand records exist only in memory, and when a node is lost in a four-shard layout, five hundred of them go with it. A cached copy can be brought back; a cached **original** cannot. The warm-up row gives the wave's length: half the access share comes back in five hundred thirty-nine seconds, 90% in 3,514. This time is **independent of shard count**: because the hash spreads key names, every shard draws the same share from every popularity tier, and one shard's warm-up curve is the same as the whole's. **Splitting brings down the wave's height, not its length.** In a four-shard layout, the data layer carries above-purchased load for close to an hour; when that window opens is not your decision. #### Where the Cache Sits The third table's first four rows meet the same read load under four layouts. The managed cache is both cheapest and fastest when it sits in the same zone: 360 units, 1.58 ms. The in-process cache row shows what being a service brings. Each of twelve application instances allocates two hundred fifty-six MB — three GB in total — and the hit rate sits at 34.7%; the managed cache reaches 87% using one and a half GB. The difference is not the amount of memory, it is the **singleness of the pool**: twelve separate caches hold twelve copies of the same keys, and each one only warms up on the requests it personally sees. The fourth row is this lesson's transfer item. Putting the cache in another zone does not change the hit rate, and writes a new **722-unit** line onto the bill: 87% of a monthly 1,036,800,000 reads, an average 800-byte response, one unit per GB. The cache node itself costs 120 units; putting it in the wrong zone costs **six times the node** and brings the total to 1,082 units — **pricier than running no cache at all**. #### Where It Gains Nothing The last three rows measure the zonal rate table: forty MB, three hundred twenty million reads a month, a set that fits entirely into every layout. The in-process cache costs 30 units and 0.02 ms; a managed cache in the same zone costs 90 units and 0.80 ms — three times the price, forty times slower. In another zone it is 730 units — **twenty-four times**. The managed cache gains nothing here: since the set already fits in every process's memory, a shared pool is unnecessary, and access over the network is pure cost. The rule is this: **a managed cache pays off when the working set does not fit in a single process's memory.** The in-process layout's cost is not in the table: twelve copies, twelve separate freshness windows. #### The Delegated Decision In a rented cluster, decisions split up as follows: | Group | Yours | Provider's | Fixed at creation | |---|---|---|---| | version | engine version, patch window | when the patch becomes mandatory | — | | maintenance | whether backups are on | change timing, node refresh, backup location | — | | scaling | node size, shard count, replica count | usable memory ratio | shard count | | data | eviction policy, key naming, time to live | data lost on a change | — | | network | the cluster's zone | which node a client lands on | the cluster's zone | **Ten of seventeen items are yours**, seven are the provider's, two are fixed at creation. Three of the maintenance group's four items are delegated; the second table's hour-long warm-up window ties back to this — you are not the one deciding **when that window opens**. You choose the engine version, but the choice is one-way: an upgraded cluster does not go back to the old version. The item fixed in scaling is the **shard count**; change it and every key gets redistributed, with the hit rate dropping through the redistribution. That is why the first table's "same hit rate, same price" result is misleading: the two levers land in the same place, but one is reversible and the other is not. The cluster's zone is also fixed, and the 722-unit item ties directly back to it. #### Summary - Whichever path reaches two GB, the hit rate is 87% and the bill is 360 units: the scaling-unit decision cannot be read off hit rate and price. - The bill's curve turns: 1,020 units with no cache, 510 at one GB, 360 at two, 420 at four, 600 at eight. 100% hit rate is the most expensive row. - A single-shard node change pushes backend-bound requests 325% over purchased capacity; at sixteen shards this falls to 19% but never zeroes out. Warm-up is 3,514 seconds under every layout. The irreversible item is the records pending in write-behind mode. - Putting the cache in another zone writes a 722-unit transfer item and pushes the total above the no-cache layout. On the 40 MB rate table, the in-process cache costs 30 units and the managed cache 90: when the working set fits in a single process, the managed cache gains nothing. #### Next Step In this lesson, transfer was a single column with a single price: cross-zone, one unit per GB. That one column became the biggest item on the bill once the placement decision went wrong. Across the rest of the measurement network, that same column gets written over and over: the collector writes to the data layer, the data layer replicates to the read replica, backups go out, logs stream, dashboard responses go out to the open network. Each flow prices differently depending on where its source and destination **sit**: the same zone, a neighboring zone, a distant region, or the internet. The next lesson is this course's lesson on money: the same architecture is run across three placements and the egress fee is worked out item by item, and then the effect of replica count, log and backup shipping, edge use, and compression decisions on that fee is counted. ### Data Transfer and Egress Cost Source: https://academia.sh/en/courses/cloud-storage/data-transfer-and-egress-cost Measuring zonal and internet egress: the same 6,180 GB running 1,440 units in a single zone, 4,380 in two zones, 7,320 across two regions; compression cutting 1,463 units while a second read replica adds 1,400; the same 1,000 GB costing zero coming in and 9,000 units going out; and the 33,600 GB accumulated over 24 months costing the same as 69 months of billing to export. The previous lesson showed that putting the cache in the wrong zone writes a 722-unit item onto the bill. That item came from a single flow and carried a single price. Looking at the whole measurement network, there are eight separate flows: the meter collector writes to the data layer, the application reads from the data layer and from the read replica, writes to the cache, the data layer replicates to the read replica, backups go to a store, logs stream to a collector, and dashboard responses go out to the open network. Except for the last one, none of these is client traffic; seven of them are the system's own internal traffic. And all of them can carry a fee. What decides the price is not the byte count, it is **where the two ends sit**. This lesson runs the same architecture across three placements, works out the egress fee item by item, measures the effect of five design decisions on that fee one at a time, and counts the asymmetry between data coming in and going out. - **MD31.** The fictional regional measurement network's monthly flows, in GB: collector → data layer 1,400, data layer → app 900, app → cache 780, data layer → read replica 1,400, read replica → app 620, data layer → backup 600, app → log 320, app → internet 160. Total 6,180 GB. - **MD32.** Prices are fictional, the currency's name is a fictional unit: within zone 0, cross-zone 1, cross-region 2, to internet 9 units/GB. Bytes coming in from the internet are free. The one point where the model rests on reality is this ordering. - **MD33.** Three placements are measured: everything in a single zone; the read replica, backup, and log collector in a second zone; the same three in a second region. - **MD34.** Compression applies only to logs, backups, and dashboard responses, at a fictional ratio of 0.38. The replication stream is already compressed, and meter writes are too small to be worth compressing. - **MD35.** The edge hit rate is **67%**, and edge egress is five units per GB. Both were measured in M23/K02; here they are used as an input and are not remeasured. - **MD36.** The data layer and the object store together grow by a net 1,400 GB a month; the bulk export runs over a 1 Gbit/s link. Both are fictional. ```js // data/egress.mjs — model: the same architecture run across three placements, egress fee // worked out item by item. Flows, prices, and growth are fictional; the currency's name is "unit". // The edge's hit rate was measured in M23/K02, here it is an input. No randomness. const FLOW = [ // [source, destination, monthly GB] ["collector", "data", 1400], ["data", "app", 900], ["app", "cache", 780], ["data", "replica", 1400], ["replica", "app", 620], ["data", "backup", 600], ["app", "log", 320], ["app", "internet", 160], ]; const SINGLE = { collector: "r1z1", data: "r1z1", app: "r1z1", cache: "r1z1", replica: "r1z1", backup: "r1z1", log: "r1z1" }; const ZONE = { ...SINGLE, replica: "r1z2", backup: "r1z2", log: "r1z2" }; const REGION = { ...SINGLE, replica: "r2z1", backup: "r2z1", log: "r2z1" }; const PRICE = { "within zone": 0, "cross-zone": 1, "cross-region": 2, "to internet": 9 }; const classify = (loc, src, dst) => dst === "internet" ? "to internet" : loc[src] === loc[dst] ? "within zone" : loc[src][1] === loc[dst][1] ? "cross-zone" : "cross-region"; const compute = (flow, loc) => { const gb = { "within zone": 0, "cross-zone": 0, "cross-region": 0, "to internet": 0 }; for (const [src, dst, v] of flow) gb[classify(loc, src, dst)] += v; const total = Object.keys(gb).reduce((a, s) => a + gb[s] * PRICE[s], 0); return { gb, total, totalGB: flow.reduce((a, f) => a + f[2], 0) }; }; const columns = (widths, left = 1) => (row) => row.map((d, i) => String(d)[i < left ? "padEnd" : "padStart"](widths[i])).join(""); const fmt1 = columns([16, 14, 16, 17, 14, 11, 15]); console.log(fmt1(["layout", "within zone", "cross-zone", "cross-region", "to internet", "total GB", "fee(unit)"])); for (const [label, loc] of [["single zone", SINGLE], ["two zones", ZONE], ["two regions", REGION]]) { const r = compute(FLOW, loc); console.log(fmt1([label, ...Object.values(r.gb), r.totalGB, r.total.toFixed(0)])); } // --- What a design decision writes onto the bill ---------------------------------- // Baseline: two zones. Each decision is applied to the baseline on its own. const BASELINE = compute(FLOW, ZONE); const scale = (flow, target, factor) => flow.map(([src, dst, gb]) => [src, dst, dst === target ? gb * factor : gb]); const COMPRESSION = 0.38, HIT = 0.67, EDGE = 5; // compression ratio; edge hit rate and unit/GB const edgeCost = 160 * EDGE + 160 * (1 - HIT) * PRICE["to internet"]; const DECISION = [ ["second read replica", compute([...FLOW, ["data", "replica2", 1400]], { ...ZONE, replica2: "r1z2" })], ["log sampling 1/5", compute(scale(FLOW, "log", 0.2), ZONE)], ["move backup to a remote region", compute(FLOW, { ...ZONE, backup: "r2z1" })], ["edge, 67% hit", { totalGB: BASELINE.totalGB, total: BASELINE.total - 160 * 9 + edgeCost }], ["compression, ratio 0.38", compute(["log", "backup", "internet"] .reduce((a, dst) => scale(a, dst, COMPRESSION), FLOW), ZONE)], ]; const fmt2 = columns([34, 13, 14, 15]); console.log("\n" + fmt2(["decision", "GB diff", "fee diff", "new total"])); console.log(fmt2(["baseline: two zones", BASELINE.totalGB.toFixed(0), "-", BASELINE.total.toFixed(0)])); for (const [label, r] of DECISION) console.log(fmt2([label, (r.totalGB - BASELINE.totalGB).toFixed(0), (r.total - BASELINE.total).toFixed(0), r.total.toFixed(0)])); // --- Cheap coming in, expensive going out ------------------------------------------ const fmt3 = columns([22, 14, 18]); console.log("\n" + fmt3(["1000 GB to", "unit/GB", "fee(unit)"])); console.log(fmt3(["from the internet", 0, 0])); for (const s of Object.keys(PRICE)) console.log(fmt3([s, PRICE[s], 1000 * PRICE[s]])); const GROWTH = 1400, LINK = 1; // net growth GB/month; export link Gbit/s const fmt4 = columns([10, 16, 18, 20, 14]); console.log("\n" + fmt4(["month", "accumulated GB", "export(unit)", "months of billing", "link(days)"])); for (const month of [12, 24, 36]) { const gb = GROWTH * month, fee = gb * PRICE["to internet"]; console.log(fmt4([month, gb, fee, (fee / BASELINE.total).toFixed(1), (gb * 8 / LINK / 86400).toFixed(1)])); } ``` ``` layout within zone cross-zone cross-region to internet total GB fee(unit) single zone 6020 0 0 160 6180 1440 two zones 3080 2940 0 160 6180 4380 two regions 3080 0 2940 160 6180 7320 decision GB diff fee diff new total baseline: two zones 6180 - 4380 second read replica 1400 1400 5780 log sampling 1/5 -256 -256 4124 move backup to a remote region 0 600 4980 edge, 67% hit 0 -165 4215 compression, ratio 0.38 -670 -1463 2917 1000 GB to unit/GB fee(unit) from the internet 0 0 within zone 0 0 cross-zone 1 1000 cross-region 2 2000 to internet 9 9000 month accumulated GB export(unit) months of billing link(days) 12 16800 151200 34.5 1.6 24 33600 302400 69.0 3.1 36 50400 453600 103.6 4.7 ``` #### Same Bytes, Three Bills The first table's most important column is **total GB**, and it is the same in all three rows: 6,180. No placement moves a single byte more or less. Same components, same flows, same volume. The bill, though, is 1,440, 4,380, and 7,320 units. **More than a fivefold difference.** The only thing that changes is where each component sits. In the single-zone row, 6,020 GB stays within the zone, and within-zone traffic is free; only 160 GB of internet egress is left, and its 1,440 units. This row's entire bill is client-bound traffic; the internal traffic costs nothing. In the second row, the read replica, backup store, and log collector move to the neighboring zone. Three components moved, four flows moved: replication from the data layer to the replica (1,400), reads from the replica to the app (620), data layer to backup (600), and app to log (320). 2,940 GB in total, and the moment it changes tier it writes one unit per GB. The bill jumps to **three times** the baseline, and what it buys is surviving the loss of a single zone. The third row moves the same three components to a second region. The flows are the same, the volume is the same, the only change is the pricing tier: the same 2,940 GB is now priced at two units. The bill is 7,320. The cost of raising durability from a zone to a region is writing 2,940 more units **for the exact same bytes**. The 2,940 GB in the second row does not split evenly across the four flows. Replication alone is 1,400 GB, **47%** of the traffic crossing the boundary; the rest is shared between reads from the replica (620), backup (600), and log (320). The size of the replication flow is no coincidence either: it is exactly the 1,400 GB the collector writes to the data layer. The read replica mirrors the write flow, which is why half the cross-zone bill depends on **write volume, not read volume**. There is no way to shrink this item by caching reads. The decision here has two parts, and only one is yours. The provider decides where each tier starts and ends, and its price; you decide which component sits inside which tier. The model makes this split visible: the price list is a constant, the placement map is a variable. #### What a Design Decision Writes onto the Bill The second table takes the two-zone placement as the baseline and applies five decisions one at a time. **A second read replica** adds 1,400 units to the bill. The reason is direct: every replica has the full replication traffic flowing to it from the data layer written out again. Adding a read replica looks like buying read capacity; on the bill it also shows up as a transfer item, and that item grows linearly with replica count. **Log sampling** is the cheapest win: cutting logs to a fifth drops 256 GB and 256 units. The GB difference and the fee difference are equal because the cross-zone price is one; the same decision would have gained twice as much in a cross-region placement. **Moving the backup to a remote region** adds 600 units while the GB difference stays zero. This is the first row in the table where the two columns diverge, and the lesson's sentence sits right here: **the fee can change without the transferred bytes changing.** Those 600 units are the price of the backup surviving a region loss too. **The edge** reprices the 160 GB of internet egress: 160 GB of edge egress at five units, the 33% share it misses at nine units from the origin. The net difference is minus 165 units — under 4% of the 4,380-unit bill. The edge's real gain does not show up in this table; that gain was in origin load, and it was measured earlier. **Compression** is by far the biggest item: 1,463 units, **a third** of the baseline. A 670 GB drop gaining 1,463 units comes down to the dropped bytes sitting in the expensive tiers. Saving 100 GB from going to the internet is worth 900 units; saving it from crossing to the neighboring zone is worth 100. The same compression ratio pays off ninefold differently depending on which flow it is applied to. The two outer rows should be read together: compression takes off 1,463 units, the second replica adds 1,400. One is a configuration line, the other a durability decision — and on the bill they nearly cancel each other out. Each of the five rows was applied to the baseline **on its own**, and adding them up would be wrong: compression applied to a log that is already sampled does not gain the same share a second time. The point where the rows really diverge is reversibility. Log sampling and compression are configuration lines; they can be switched off the next day and the bill returns to what it was. Adding a replica and moving the backup to another region require moving data; reversing those decisions writes a new transfer item. The same table holds decisions that reverse for free and decisions whose reversal costs money. #### Cheap Coming In, Expensive Going Out The third table sends the same 1,000 GB to five destinations. Coming in from the internet is zero, moving within a zone is zero, crossing to the neighboring zone is 1,000, crossing to the neighboring region is 2,000, going out to the internet is **9,000** units. This table shows a single fact: putting data in is free, taking it out is the top row of the price list. Field photos enter the system at hundreds of GB a month, and none of that entry gets written to any line; the day those same photos need to come back out, nine units a GB will get written. The asymmetry is not a bug, it is the pricing itself. That is also why entry never shows up in the first table's flow list: since field photos and meter readings coming in are free, adding them to the model does not change the bill. The flow that grows the system is the flow that never appears on the bill. Watching monthly spend to track data growth therefore does not work; while spend stays flat, the accumulated volume — and with it, the fourth table's export column — climbs every month. The fourth table measures what this asymmetry does over time. A data layer growing by a net 1,400 GB a month accumulates 16,800 GB in twelve months, 33,600 in twenty-four, 50,400 in thirty-six. Pushing that accumulation out to the internet all at once costs 151,200, 302,400, and 453,600 units respectively — equal to **34.5, 69.0, and 103.6 months of billing**. The last column shows a second limit. Pushing 33,600 GB down a 1 Gbit/s link takes 3.1 days, 50,400 GB takes 4.7. Through that whole span, the old system keeps running, new data keeps accumulating, and both sides keep getting billed. #### Egress Fee and Exit Cost These two terms sound alike and need to be pulled apart. **Egress fee** is a line that sits on the bill every month; in this model it is a slice of the 4,380 units, and design decisions can shrink it. **Exit cost** is the concept built in M23/K01: the total price of leaving a rented place. The link is one sentence: **the egress fee is the exit cost's price tag, and it grows every month along with the accumulated data.** This course's **irreversible item** shows up here. The placement decision is reversible on paper; moving a component back from the second zone to the first is a configuration change. But a component does not move without its **data** moving, and that move itself writes an egress item: bringing 33,600 GB back from a neighboring region is 67,200 units, fifteen months of billing. Getting out of the wrong placement can cost more than staying in it. The delegated decision also reads off this table. The provider holds four items: the price of the four tiers, where the tier boundaries are drawn, which tier a flow falls into, and whether entry stays free. You hold five items: which component sits where, how many replicas are kept, how much of the log gets shipped, where the backup gets written, and whether compression is on. **Five of nine items are yours**, but the multipliers that set the size of the numbers in the table belong to the other four. #### Summary - The same 6,180 GB writes 1,440, 4,380, and 7,320 units across three placements. It is not the byte count that sets the price, it is where the two ends sit. - Spreading across two zones pulls 2,940 GB out of the within-zone tier and triples the bill; moving the same three components to a second region adds 2,940 more units. - Measured one at a time, compression takes off 1,463 units and a second read replica adds 1,400. Moving the backup to a remote region writes 600 units while the GB difference stays zero: the fee can change without the transferred bytes changing. - The same 1,000 GB costs zero coming in, 1,000 to the neighboring zone, 2,000 to the neighboring region, 9,000 to the internet. Exporting the 33,600 GB accumulated over 24 months costs 302,400 units, sixty-nine months of billing, and 3.1 days over a 1 Gbit/s link. - The irreversible item is not the placement itself, it is the data underneath it: moving a component back means moving its data too, and that move writes 67,200 units for 33,600 GB. #### Next Step This lesson worked out the price of moving data from one place to another, and treated the backup flow as an ordinary item: 600 GB a month, 600 or 1,200 units depending on its destination. A backup exists not because it moves, but because it can be **restored**. What that 600 GB on the price list buys is which moment you can return to the day a bad write or a corrupted table turns up. In a managed database, that question no longer depends on a file kept by hand, it depends on the provider's snapshot schedule and the logs it keeps. The next lesson measures this: where the oldest restorable point sits, how long a restore takes, what window is lost while restoring, and where the balance between retention period and storage fee settles. ### Backup and Restore Source: https://academia.sh/en/courses/cloud-storage/backup-and-restore Comparing the snapshot against the continuous log: scanning frequency and counting the lost window together with the storage fee, a 24-hour snapshot plus a 7-day log coming out both cheaper and sixty times narrower than an hourly snapshot, restore time splitting into four components with a 58-minute baseline, and the portable copy's extra cost — starting at 301 units a month — because backups sit in the service's own format. The previous lesson worked out, item by item, where data sits and what moving it costs. The backup flow was an ordinary line there: 600 GB a month, 600 or 1,200 units depending on its destination. A backup exists not because it moves, but because it can be **restored**: which moment you can return to the day a bad bulk update turns up. In a managed data layer, that question no longer depends on a file kept by hand, it depends on the provider's snapshot schedule and the logs it keeps. Backup scope and restore drills were measured earlier; this lesson does not repeat that, it takes it as input. Three things get counted here: the trade-off between the lost window and the storage fee, the components of restore time, and whose format the backup sits in. - **MD37.** The fictional regional measurement network's work-order data layer is 220 GB; 4% of the data changes in 24 hours, and the changed share falls off exponentially at shorter intervals, because the same blocks get written over and over. - **MD38.** The continuous log produces 46 GB a day and is archived every sixty seconds. Snapshots are kept for thirty days. Both are fictional. - **MD39.** The price is fictional, the currency's name is a fictional unit: backup storage is 0.9 units per GB a month; transfer is 0 units per GB in the same zone, 1 in a neighboring zone, 2 in a neighboring region, 9 to the internet. - **MD40.** Restore speeds are fictional: opening a new instance is a fixed 210 seconds, loading runs at 95 MB/s, replaying the log at 12 MB/s, verifying at 260 MB/s. Replay is the slowest step, because records are applied one at a time. - **MD41.** A logical export reads at 40 MB/s, the compression ratio is 0.38; restoring from the exported copy runs at 26 MB/s, because indexes are rebuilt. The portable copy is taken weekly, and four are kept a month. All of it is fictional. ```js // backup/restore.mjs — model: backup and point-in-time restore in the managed data layer. // Backup scope and restore drills were measured in M22/K07 and are not repeated here. // Sizes, speeds, and prices are fictional; the currency's name is "unit". No randomness. const DATA_GB = 220, DAILY_CHANGE = 0.04, EXPONENT = 0.7; // share changed in 24h; decay exponent const LOG_GB = 46, PRICE = 0.9; // continuous log output GB/day; unit/GB-month const SNAPSHOT_DAYS = 30, ARCHIVE_SEC = 60; // snapshot retention; log archiving interval const changedShare = (hours) => DAILY_CHANGE * Math.pow(hours / 24, EXPONENT); const snapshotGB = (hours) => DATA_GB + (SNAPSHOT_DAYS * 24 / hours) * changedShare(hours) * DATA_GB; const columns = (widths, left = 1) => (row) => row.map((d, i) => String(d)[i < left ? "padEnd" : "padStart"](widths[i])).join(""); const fmt1 = columns([28, 16, 22, 14, 13, 16]); console.log(fmt1(["layout", "lost window", "continuous restore", "oldest point", "stored GB", "fee(unit/month)"])); const row = (label, hours, logDays) => { const gb = snapshotGB(hours) + logDays * LOG_GB; console.log(fmt1([label, logDays ? ARCHIVE_SEC + " s" : hours + " h", logDays ? logDays + " d" : "none", SNAPSHOT_DAYS + " d", gb.toFixed(0), (gb * PRICE).toFixed(0)])); }; for (const hours of [24, 12, 6, 3, 1]) row(`snapshot ${hours} h`, hours, 0); for (const days of [1, 7, 35]) row(`24 h + log ${days} d`, 24, days); // --- Restore time splits into components ----------------------------------------- const INSTANCE_SEC = 210, LOAD_MBS = 95, REPLAY_MBS = 12, VERIFY_MBS = 260; const logPerHour = (gb) => LOG_GB / 24 * (gb / DATA_GB); // log GB produced per hour const restore = (gb, distance) => { const load = gb * 1024 / LOAD_MBS, replay = logPerHour(gb) * distance * 1024 / REPLAY_MBS; const verify = gb * 1024 / VERIFY_MBS; return { load, replay, verify, total: INSTANCE_SEC + load + replay + verify }; }; const fmt2 = columns([14, 12, 15, 19, 17, 14, 10]); console.log("\n" + fmt2(["data (GB)", "instance (s)", "load (s)", "replay (s)", "verify (s)", "total (s)", "hours"])); for (const gb of [55, 110, 220, 440, 880]) { const r = restore(gb, 18); console.log(fmt2([gb, INSTANCE_SEC, r.load.toFixed(0), r.replay.toFixed(0), r.verify.toFixed(0), r.total.toFixed(0), (r.total / 3600).toFixed(2)])); } const BASELINE = INSTANCE_SEC + DATA_GB * 1024 / LOAD_MBS + DATA_GB * 1024 / VERIFY_MBS; const fmt3 = columns([22, 17, 19, 15, 17]); console.log("\n" + fmt3(["snapshot interval", "worst distance", "replay (s)", "total (h)", "fee(unit/month)"])); for (const hours of [24, 12, 6, 3, 1]) { const r = restore(DATA_GB, hours); console.log(fmt3([hours + " h", hours + " h", r.replay.toFixed(0), (r.total / 3600).toFixed(2), (snapshotGB(hours) * PRICE).toFixed(0)])); } console.log(`baseline (instance + load + verify, replay zero): ${(BASELINE / 3600).toFixed(2)} hours`); // --- Portable backup --------------------------------------------------------------- // The service's own backup is in its own format. The portable copy is a logical export. const COMPRESSION = 0.38, WEEKLY = 4, EXPORT_MBS = 40, IMPORT_MBS = 26; const DESTINATION = { "same zone": 0, "neighboring zone": 1, "neighboring region": 2, "to internet": 9 }; const COPY_GB = DATA_GB * COMPRESSION, SERVICE = (snapshotGB(24) + 7 * LOG_GB) * PRICE; const fmt4 = columns([21, 12, 16, 18, 16, 16]); console.log("\n" + fmt4(["destination", "unit/GB", "transfer(mo)", "storage(mo)", "extra total", "service + extra"])); for (const [name, rate] of Object.entries(DESTINATION)) { const transfer = COPY_GB * WEEKLY * rate, storage = COPY_GB * WEEKLY * PRICE; console.log(fmt4([name, rate, transfer.toFixed(0), storage.toFixed(0), (transfer + storage).toFixed(0), (SERVICE + transfer + storage).toFixed(0)])); } console.log(`the service's own backup is ${SERVICE.toFixed(0)} units/month; the portable copy is ` + `${COPY_GB.toFixed(1)} GB x ${WEEKLY}/month (compression ${COMPRESSION}).`); console.log(`export ${(DATA_GB * 1024 / EXPORT_MBS / 3600).toFixed(2)} h (${EXPORT_MBS} MB/s); ` + `restoring from it ${(DATA_GB * 1024 / IMPORT_MBS / 3600).toFixed(2)} h (${IMPORT_MBS} MB/s, ` + `indexes are rebuilt) — ` + `${(DATA_GB * 1024 / IMPORT_MBS / restore(DATA_GB, 1).total).toFixed(1)}x the service's own restore.`); ``` ``` layout lost window continuous restore oldest point stored GB fee(unit/month) snapshot 24 h 24 h none 30 d 484 436 snapshot 12 h 12 h none 30 d 545 491 snapshot 6 h 6 h none 30 d 620 558 snapshot 3 h 3 h none 30 d 713 641 snapshot 1 h 1 h none 30 d 905 814 24 h + log 1 d 60 s 1 d 30 d 530 477 24 h + log 7 d 60 s 7 d 30 d 806 725 24 h + log 35 d 60 s 35 d 30 d 2094 1885 data (GB) instance (s) load (s) replay (s) verify (s) total (s) hours 55 210 593 736 217 1755 0.49 110 210 1186 1472 433 3301 0.92 220 210 2371 2944 866 6392 1.78 440 210 4743 5888 1733 12574 3.49 880 210 9485 11776 3466 24937 6.93 snapshot interval worst distance replay (s) total (h) fee(unit/month) 24 h 24 h 3925 2.05 436 12 h 12 h 1963 1.50 491 6 h 6 h 981 1.23 558 3 h 3 h 491 1.09 641 1 h 1 h 164 1.00 814 baseline (instance + load + verify, replay zero): 0.96 hours destination unit/GB transfer(mo) storage(mo) extra total service + extra same zone 0 0 301 301 1026 neighboring zone 1 334 301 635 1361 neighboring region 2 669 301 970 1695 to internet 9 3010 301 3311 4036 the service's own backup is 725 units/month; the portable copy is 83.6 GB x 4/month (compression 0.38). export 1.56 h (40 MB/s); restoring from it 2.41 h (26 MB/s, indexes are rebuilt) — 2.4x the service's own restore. ``` #### Window or Bill The first table's first five rows run on the snapshot alone. Pulling the frequency from twenty-four hours to one hour narrows the lost window twenty-fourfold and pushes storage from 484 GB to 905 GB — the bill rises from 436 to 814 units, an 87% increase. The growth in stored bytes looks backward: a more frequent snapshot carries fewer changed bytes each time. The number still grows, because as the interval shortens, the same blocks end up sitting in **more than one snapshot**. The cost of raising frequency is not request count, it is repetition. The last three rows turn on the continuous log and rewrite the table. Add seven days of log to the 24-hour snapshot and the lost window drops to **sixty seconds**, with the bill at 725 units; the hourly snapshot, by contrast, gives a one-hour window for 814 units. **The cheaper option narrows the window sixty times more.** Raising snapshot frequency is no substitute for a continuous log. The third and fourth columns show the limit. The continuous-restore window is the log's retention period: under the seven-day layout, you can return to **any second** of the last seven days. The oldest discrete point, though, is thirty days out — you can return to a snapshot from twenty days ago, not to that day's afternoon. The bill scales with the first: thirty-five days of log is 2,094 GB and 1,885 units, two and a half times the seven-day setup. #### The Floor of a Restore The second table splits the duration into four components. Opening a new instance is a fixed 210 seconds, and its share falls as the data grows: 12% of the total at 55 GB, 1% at 880 GB. Loading and verifying grow linearly with size. Replay grows twice over — as data grows, log output grows with it — and at 220 GB, at 2,944 seconds, it is **the largest component**. A design decision follows from this: what sets restore time is not just data size, it is **how far the target point sits from the last snapshot**. At a twenty-four-hour interval, the worst case is a 3,925-second replay and a 2.05-hour total; at a one-hour interval, replay drops to 164 seconds and the total to 1.00 hour. The price of halving that duration is in the first table: from 436 to 814 units. And the drop stops there: opening the instance plus loading plus verifying add up to **0.96 hours**, and no matter how close to zero the snapshot interval gets pushed, nothing goes below this floor. The continuous log brings down the lost window, not the restore time. **The two numbers are bought from two separate items.** #### The Delegated Decision The sentence "backups are automatic in a managed service" hides four items: **how** the backup is taken, which format it is kept in, where it is stored, and how it is verified all sit with the provider. The 95, 12, and 260 MB/s above are not a configuration line either — they are the number the service hands you. Three kept items directly set this lesson's tables: **retention period** (the first table's third and fourth columns), **who holds the encryption key**, and **whether it gets copied outside the region** (the last table's destination column). Four of seven items are delegated, three are kept; two of the three kept items directly write a transfer item. The encryption key is the quietest item: if it sits in the provider's store, the backup is readable but not portable; if it sits with you, a lost key erases the backup too. This item never gets written onto the bill at all. #### Whose Format the Backup Is In The last table measures this lesson's irreversible item. The backup the service takes is in its own format: it cannot be loaded anywhere else, cannot be handed to another engine. A portable copy has to be produced separately — a logical export, 1.56 hours of reading for 220 GB. The extra cost is two items. Four compressed 83.6 GB copies run 301 units a month in storage; the transfer item depends on the destination: zero in the same zone, 334 in a neighboring zone, 669 in a neighboring region, 3,010 to the internet. The service's own backup is 725 units; adding portability pushes the total to 1,026 units in the same zone (41%), to 1,695 in a neighboring region (134%). The third cost is time: restoring from the exported copy takes 2.41 hours, **2.4 times** the service's own restore. A portable backup is not just expensive, it is a slow backup. This item ties to M23/K01's exit column in one sentence: a portable backup is **the exit cost paid in installments.** When it is not paid, the saving is real, and the cost comes due all at once on the day you leave. #### Summary - Pulling snapshot frequency from twenty-four hours to one hour narrows the window twenty-fourfold and raises the bill from 436 to 814 units: hot blocks repeat across more than one snapshot. - A 24-hour snapshot plus seven days of continuous log narrows the window to sixty seconds and costs 725 units: both cheaper than the hourly snapshot and sixty times narrower. - The continuous-restore window is the log's retention period, the oldest discrete point is the snapshot's: you can return to any second up to seven days back, and only to specific moments up to thirty days back. - A restore has four components and a floor of 0.96 hours; replay is the largest component at 2,944 seconds for 220 GB, and it shortens only with the snapshot interval. - Four of seven decision items are delegated; two of the three kept items directly write a transfer fee. The backup format cannot travel: a portable copy adds between 301 and 3,311 units a month, and restoring from it is 2.4 times slower. #### Course Wrap-Up | Lesson | Delegated decision | Irreversible item | Egress or transfer cost | |---|---|---|---| | Object Storage | placement, replication, repair | key layout: 2,481,240 requests | 2.77 GiB → 0.055 / 0.236 units | | Storage Classes | durability, same across four classes | minimum-retention debt: reverting costs 0.338 units | 276.3 GiB retrieved → 23.48 units | | Versioning and Lifecycle Rules | enforcing the rule | versioning: 93.6 GiB remains after turning it off | 8,931,900 transitions → 239.75 units | | Block Storage | disks, replication, repair | size does not shrink: 1,056 units overpaid a year | 110 GiB snapshot → 2.20 / 9.35 units | | File Storage | metadata server, growth | access protocol: 30 of 54 call sites | 640 GiB → 12.80 / 54.40 units | | Archive Storage | storage medium, readability | 180-day debt: 4.5 months paid for nothing | 297.9 GiB → 25.32 units | | Access Control and Signed URLs | verification only | issued link: revoking it breaks 3,360,000 | leaked bucket: 184.8 GiB → 15.71 units | | Managed Relational Database | 8 of 28 items; 9 inaccessible | 4 of 11 kept items are one-way | 220 GB → 3.97 h, 1,980 units | | Read Replicas | replication mechanics | promoted replica: full 220 GB copy | 1,370 GB/month/replica → 2,741 units | | Managed Key-Value Service | partitioning and balancing | partition key: 8 h, 518.4K units | 62.2 GB export → 560 units | | Secondary Indexes and Streams | index build and update | missed window: 960K events at 30 h | 41.5 GB/month of events → 83 / 373 units | | Managed Cache Service | 7 of 17 items | 500 pending records; shard count fixed | other zone → 722 units/month | | Data Transfer and Egress Cost | 4 of 9 items: the price list | accumulated data: 33,600 GB → 67,200 units | 6,180 GB: 1,440 → 7,320 units | | Backup and Restore | taking and verifying the backup | backup format: export +301 units | 83.6 GB x 4 → 0 / 334 / 669 / 3,010 | Across the fourteen rows, the third column splits into three families. **Six rows are schema and format:** key layout, access protocol, collation choice, partition key, shard count, and backup format — none of them change without rewriting the data. **Five rows are prepaid or accumulated bytes:** the minimum-retention debt, the versioning backlog, the archive's one hundred eighty days, the volume that will not shrink, and the accumulated data itself; on these, the decision reverses and no fee is charged. **Three rows are a closed window of time:** the issued signed link, the broken replication link, and the missed event window. The fourth column points the same way across all fourteen rows: no row ever charged a fee for data coming in, every row charged for it going out, and the fee was measured by where it went. **That is this course's rule: data is cheap coming in, expensive going out — and in a rented data layer, what really binds you is not the price, it is the irreversible schema.** You can choose to pay the bill; you cannot pay your way out of the six-row schema family. Across three courses, a machine was rented, a network was rented, data was rented. Underneath all of it, the same thing still stands: **a server that keeps running.** Application instances are up at three in the morning too, the measurement network's dashboard waits even when no request arrives, and both get billed by the hour. The next course — **M23/K04 Serverless and Edge Computing** — asks this: can the continuously running server itself be rented away, or only the seconds it actually runs?