--- title: 'Scaling the Data Layer' source: 'https://academia.sh/en/courses/scaling-the-data' language: en updated: '2026-08-23T07:01:32+00:00' license: 'CC BY-SA 4.0' lessons: 18 outcomes: - 'Ties replication, sharding, and federation decisions to the access pattern' - 'Designs the cache layers with placement and invalidation' - 'Separates the read and write models where that is called for' --- # Scaling the Data Layer Removing the single-store assumption: tying the data store choice to the access pattern, replication, federation, and sharding decisions, partitioning strategies, denormalization and materialized views that shorten the read path, which layer the cache sits in and with which strategy, and separating the read and write models where that is called for; every decision ties back to the store load and stored data rows calculated in the introductory course. ## Data Distribution Decisions about where data sits: tying the choice between relational and non-relational to the access pattern, leader–follower and multi-leader replication models, splitting the database by function, key selection with rebalancing and hot spot problems, range, hash, and lookup-table partitioning strategies, denormalization that shortens the read path, precomputed results, and the use cases of key–value, document, wide-column, and graph stores. ### Data Store Selection Source: https://academia.sh/en/courses/scaling-the-data/data-store-selection Tying the store type to the access pattern: separating the request and record rates of three patterns, measuring the single-store and pattern-split layouts in a real engine by tables touched and records processed, drawing the decision axis from pattern and scale rather than the data model, and calculating what the split costs the introductory course's store load and stored data rows. The Application Layer and Service Interaction course made services stateless, let them discover each other, queued the work, and coordinated the workflow — all of it resting on one assumption: the data sat **somewhere**, and that somewhere was never questioned. The evidence is in the arithmetic: K01's `requests reaching store/s` (138.89) and `stored data GB` (712.48) rows never moved through that entire course. This course takes up those rows, and the first question is the store's **type**. What the relational model gives, what the non-relational families give up, and where schema flexibility actually moves to were established in the Data Modeling and Relational Theory course; data modeling is not taught here. The axis here is different: **access pattern and scale**. The same data, read through different questions, calls for a different store layout, and this lesson measures that link. #### Three Access Patterns A store decision cannot be defended by looking at the data alone — the defense rests on the **shape of the question**. The shipment tracking and pricing service carries three patterns, and all three rates come from K01's arithmetic. **P1 — single lookup by tracking number.** The key is known in advance, the response is small (V5: 480 bytes), and behind the edge cache it comes down to 41.67 requests per second. The query is known ahead of time and does not change. **P2 — state event write by carrier.** Carrier systems produce seven events per shipment (V4); the peak rate is 97.22 requests/s. Every event is a new record — it never overwrites an existing one. **P3 — periodic batch scan by seller.** The pricing job scans thirty days (V11) and reads 833.33 records per second inside a four-hour window. As a request count it is close to zero; as a record count it is the largest of the three. The first thing the three patterns say together: **request rate and record rate are separate measures.** P3 touches the store once a day yet reads more records than P1 does across its entire day, which makes sizing a store by request rate alone an incomplete calculation. #### The Trace Patterns Leave in the Store The measurement uses a real engine. Two layouts are set up in memory with `node:sqlite`: in the **single-store** layout, the shipment and the event live in separate tables; in the **pattern-split** layout, P1 also gets a document table holding the shipment's state and its last three route steps in a single row — the response V5 describes. What is measured is not the query language but the number of tables the engine touches and the number of records it processes, counted with a function the engine calls on every row. ```js // store/pattern.mjs — the tables touched and records processed by three access patterns // under two layouts. A real engine runs with node:sqlite; the query language itself is // not what is measured — that was set up in M17 and is not covered here. import { DatabaseSync } from "node:sqlite"; const SELLER = 20, DAY = 30, DAILY = 100, EVENTS = 7; // model scale: 3000 shipments per seller const db = new DatabaseSync(":memory:"); db.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INTEGER, day INTEGER); CREATE INDEX shipment_seller ON shipment(seller, day); CREATE TABLE event(trackingNo TEXT, seq INTEGER, state TEXT); CREATE INDEX event_trackingNo ON event(trackingNo); CREATE TABLE document(trackingNo TEXT PRIMARY KEY, body TEXT);`); let counter = 0; db.function("count", { deterministic: false }, (x) => { counter += 1; return x; }); const steps = (s) => db.prepare(`EXPLAIN QUERY PLAN ${s}`).all() .flatMap((r) => r.detail.match(/(?:SCAN|SEARCH) \w+/g) ?? []); const s = db.prepare("INSERT INTO shipment VALUES(?,?,?)"), e = db.prepare("INSERT INTO event VALUES(?,?,?)"); const d = db.prepare("INSERT INTO document VALUES(?,?)"); db.exec("BEGIN"); for (let v = 0; v < SELLER; v += 1) for (let day = 0; day < DAY; day += 1) for (let k = 0; k < DAILY; k += 1) { const trackingNo = `TR-${v}-${day}-${k}`; s.run(trackingNo, v, day); for (let i = 0; i < EVENTS; i += 1) e.run(trackingNo, i, `state${i}`); d.run(trackingNo, JSON.stringify({ state: `state${EVENTS - 1}`, steps: [EVENTS - 3, EVENTS - 2, EVENTS - 1] })); } db.exec("COMMIT"); console.log(`model: ${SELLER * DAY * DAILY} shipments, ${SELLER * DAY * DAILY * EVENTS} events, ` + `${SELLER} sellers, ${DAY} days (${DAY * DAILY} shipments per seller)`); const READS = [ ["P1 single lookup by tracking no", "single store", "TR-10-15-50", "SELECT count(s.trackingNo), e.state FROM shipment s JOIN event e ON e.trackingNo = s.trackingNo WHERE s.trackingNo = ? ORDER BY e.seq DESC LIMIT 3"], ["P1 single lookup by tracking no", "split", "TR-10-15-50", "SELECT count(body) FROM document WHERE trackingNo = ?"], ["P3 period scan by seller", "single store", 10, "SELECT count(e.trackingNo) FROM event e JOIN shipment s ON s.trackingNo = e.trackingNo WHERE s.seller = ?"], ["P3 period scan by seller", "split", 10, "SELECT count(trackingNo) FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29"], ]; console.log(`\n${"pattern".padEnd(33)}${"layout".padEnd(13)}${"table".padStart(6)}` + `${"processed".padStart(11)}${"returned".padStart(10)} engine steps`); for (const [name, layout, arg, sql] of READS) { counter = 0; const returned = db.prepare(sql).all(arg).length, st = steps(sql); console.log(`${name.padEnd(33)}${layout.padEnd(13)}${String(new Set(st).size).padStart(6)}` + `${String(counter).padStart(11)}${String(returned).padStart(10)} ${st.join(" + ")}`); } // P2: length of the write path — how many tables a state event touches const write = (layout) => { db.prepare("INSERT INTO event VALUES(?,?,?)").run("TR-0-0-0", EVENTS, "stateNew"); if (layout === "single store") return 1; db.prepare("UPDATE document SET body = ? WHERE trackingNo = ?") .run(JSON.stringify({ state: "stateNew", steps: [EVENTS - 2, EVENTS - 1, EVENTS] }), "TR-0-0-0"); return 2; }; console.log(`\nP2 write of a state event arriving from the carrier: single store ${write("single store")} table, ` + `split ${write("split")} tables`); ``` ``` model: 60000 shipments, 420000 events, 20 sellers, 30 days (3000 shipments per seller) pattern layout table processed returned engine steps P1 single lookup by tracking no single store 2 7 3 SEARCH s + SEARCH e P1 single lookup by tracking no split 1 1 1 SEARCH document P3 period scan by seller single store 2 21000 21000 SEARCH s + SEARCH e P3 period scan by seller split 1 3000 3000 SEARCH shipment P2 write of a state event arriving from the carrier: single store 1 table, split 2 tables ``` These numbers belong to the **measurement** class: they come from an engine running on this machine and are deterministic, but the plan the engine picks is its own decision, and it depends on the engine's version. The model's scale is also a choice — 3,000 shipments per seller is what thirty days works out to at K01's rate of 100 shipments/day. Three rows deserve a close read. **P1's work differs sevenfold between the two layouts.** The single store touches two tables, processes seven event records, and returns three; the split layout touches one table and processes one. The sevenfold ratio is not an indexing effect — both layouts index the same way. The difference is whether the data already **sits in the shape it is read in**. **P3's ratio is the same sevenfold, but the case runs the opposite way.** The pricing job wants the shipment's commercial fields, not event detail; in the single store the join drags it into 21,000 records, while the split layout holds it at 3,000. The same split that cheapened P1 does this. **P2 pays the cost.** A state event writes to one table in the single store; in the split layout it writes to the event table and updates the document, stretching the write path to two tables. Shortening the read path is not free — the write path collects the bill. #### Decision Axis The measurement reduces the choice to four questions. The questions concern **pattern and scale**, not the data model. **Is the key fixed?** P1 always arrives with the same key and wants one record — that is the entirety of a **key–value store**'s job: no declarative query, no join, no schema constraint. **Is the record read as a single unit?** P1's fields are read together and updated together, a boundary that matches a **document store**'s natural unit. P3's fields do not sit inside that boundary. **Is the query known in advance?** Pricing filters on a variable mix of period, seller, tariff, and contract, and its questions grow over time. An unknown-in-advance question calls for a declarative query, which is why the relational model wins here and stays the system of record. **Does the scale exceed a single engine?** Writes run at 97.22 requests/s, daily growth is 976 MB, stored data is 712.48 GB — all three still fit on a **single engine**, which is why this lesson's decision splits the read path by pattern, not the data itself. Splitting the data comes up once scale grows, and this topic's later lessons take that up. The four answers may not point to a single store. The non-relational families — key–value, document, **wide-column**, and **graph** stores — do not replace the system of record; they are secondary stores derived from it, and this topic's last lesson takes up their use cases on their own. The wide-column family appeared in the Data Modeling and Relational Theory course as the "column-family store"; the two are the same family, and that course's rule — the system of record stays singular — is not changed here. #### Back to the Arithmetic The split's cost shows up in K01's rows. The secondary store is only needed for shipments that are **still in transit**; the tracking page of a delivered shipment is opened rarely. **VD1 — the tracking document is retained for 30 days.** Rationale: once a shipment is delivered, tracking queries against it grow rare, while the record itself is kept for 730 days for contract disputes (V12). This assumption is not added to K01's table; its sensitivity is given below at 90 days. ```js // store/cost.mjs — the impact of the pattern-split layout on K01's calculated rows const READ = 41.67, WRITE = 97.22; // K01: read/s behind the cache, peak write requests/s const STORE = 138.89, RATIO = 2.33; // K01: requests/s reaching the store, store write/read ratio const SCAN = 833.33; // K01: batch job scan records/s const GROWTH = 976, STORED = 712.48; // K01: daily data growth MB, stored data GB const V3 = 400_000, V5 = 480; // K01: daily shipments, tracking response body (bytes) const VD1 = 30; // this lesson's assumption: tracking document retention (days) // Measured mechanism: P1 is 7 records / 2 tables in single store, 1 record / 1 table split; // P2 writes to 1 table in single store, 2 tables split; P3 is 21000 / 3000 records. const D = [["single store", 7, 1, 21000], ["split by pattern", 1, 2, 3000]]; const b = (x) => x.toFixed(2); console.log(`${"layout".padEnd(24)}${"read records/s".padStart(18)}${"store ops/s".padStart(14)}` + `${"write/read".padStart(13)}${"scan records/s".padStart(16)}`); for (const [name, records, target, scan] of D) { const readRecords = READ * records, ops = READ + WRITE * target; console.log(`${name.padEnd(24)}${b(readRecords).padStart(18)}${b(ops).padStart(14)}` + `${b((WRITE * target) / READ).padStart(13)}${b(SCAN * (scan / 3000)).padStart(16)}`); } console.log(`K01 baseline: requests/s reaching the store ${STORE}, write/read ratio ${RATIO}, ` + `batch scan ${SCAN} records/s`); const docDaily = (V3 * V5) / 1e6; // MB/day const docStored = (V3 * V5 * VD1) / 1e9; // GB console.log(`\n${"growth line".padEnd(26)}${"K01".padStart(10)}${"split".padStart(11)}${"factor".padStart(8)}`); console.log(`${"daily written MB".padEnd(26)}${b(GROWTH).padStart(10)}${b(GROWTH + docDaily).padStart(11)}` + `${b((GROWTH + docDaily) / GROWTH).padStart(8)}`); console.log(`${"stored data GB".padEnd(26)}${b(STORED).padStart(10)}${b(STORED + docStored).padStart(11)}` + `${b((STORED + docStored) / STORED).padStart(8)}`); console.log(`tracking document store is ${b(docStored)} GB = ` + `1/${(STORED / docStored).toFixed(0)} of stored data; if VD1 were 90 days, ${b((V3 * V5 * 90) / 1e9)} GB`); ``` ``` layout read records/s store ops/s write/read scan records/s single store 291.69 138.89 2.33 5833.31 split by pattern 41.67 236.11 4.67 833.33 K01 baseline: requests/s reaching the store 138.89, write/read ratio 2.33, batch scan 833.33 records/s growth line K01 split factor daily written MB 976.00 1168.00 1.20 stored data GB 712.48 718.24 1.01 tracking document store is 5.76 GB = 1/124 of stored data; if VD1 were 90 days, 17.28 GB ``` These rows belong to the **calculation** class; their inputs are K01's arithmetic, the measured record rates, and VD1. The **write/read ratio at store** climbs from 2.33 to 4.67, because every state event now writes to two places while the read side, shortened, does not grow. The store operation rate that replaces `requests reaching store/s` climbs from 138.89 to 236.11 — a **1.70×** factor. In exchange, the read side's processed record rate falls from 291.69 to 41.67, and the batch scan's falls from 5,833.31 to 833.33: the decision does not erase load, it **moves it from the read path to the write path.** The two growth lines split apart too. Daily written bytes rise from 976 to 1,168 MB (**1.20×**), but stored data moves only from 712.48 to 718.24 GB (**1.01×**), because the document drops off after thirty days. The payoff is striking: **the store serving all of P1's read load is 1/124 of stored data.** If VD1 tripled, it would be 17.28 GB and the ratio would still sit near one part in forty. The store type decision, for that reason, looks not at data volume but at **what share of it gets read.** #### Summary - A store decision is defended by access pattern: P1 splits out as 41.67 requests/s of single lookup, P2 as 97.22 requests/s of inserts, and P3 as 833.33 records/s of scanning, and request rate and record rate are separate measures. - The same read touches 2 tables and processes 7 records in the single store, but 1 table and 1 record in the pattern-split layout; the batch scan falls from 21,000 records to 3,000. - The cost sits on the write path: a state event now writes to two tables instead of one. - The choice is delivered by four questions — is the key fixed, is the record read as a single unit, is the query known in advance, does the scale exceed a single engine; the relational model stays the record's system of record because of questions not known in advance. - Back to K01: the store operation rate climbs from 138.89 to 236.11 operations/s (a 1.70× factor) and the write/read ratio climbs from 2.33 to 4.67; the read side's processed records fall from 291.69 to 41.67. - With VD1 = 30 days, the tracking document store is 5.76 GB, or 1/124 of stored data; daily written bytes climb from 976 to 1,168 MB while stored data climbs from 712.48 to 718.24 GB. #### Next Step The store type decision is made and the read path is shorter, but there is still a **single copy** on the table. That copy takes 236.11 operations per second, carries 718.24 GB, and when it goes down, neither the tracking query nor the pricing job gets an answer. The first way to spread read load across more than one machine is not to split the data but to **replicate** it — keep the same data in more than one place. The next lesson takes up that decision: how many copies the read splits across in a leader–follower layout, why the write does not split, what write/read ratio each copy ends up seeing, how much the stored data multiplies, and what number accepting writes in more than one place gives rise to. ### Replication Source: https://academia.sh/en/courses/scaling-the-data/replication What keeping the same data in more than one place costs at scale: splitting reads while writes repeat at every node in a leader–follower layout, the diminishing gain from adding replicas, stored data multiplying by the replica count, converting asynchronous lag into a stale tracking response, and counting two zones writing the same shipment in a multi-leader layout as permanent divergence. The previous lesson tied the store's type to the access pattern and shortened the read path. There is still a single copy on the table: one node taking 236.11 operations per second, carrying 718.24 GB, and leaving both the tracking query and the pricing job without an answer when it goes down. The first way to spread load across more than one machine is not to split the data but to **replicate** it: keep the same data in more than one place. The mechanics of replication — log streaming, replication lag, synchronous acknowledgment's cost, the majority rule, failover — were established and measured in the Relational Database Administration course and are not repeated here. This lesson's question is a **scaling decision**: which number does adding a replica bring down, which does it leave untouched, and what grows in exchange. The consistency models were also defined in the Introduction to System Design course; here they are only used. #### What Replication Splits In a **leader–follower** layout, one node accepts writes and the others apply its changes and serve reads. What splits is the read; the write does not, because every node must apply **every** write. The calculation below opens that asymmetry up by replica count, and the second table gives the read side's counterpart to asynchronous lag. **VD2 — a shipment stays in transit for 3 days.** Rationale: the time between acceptance and delivery; V4's seven state events spread across this span, and tracking queries arrive for shipments inside it. Its sensitivity is linear — doubling the span doubles the number of shipments in transit and halves the stale response ratio. This assumption is not added to K01's table. ```js // replication/leader-follower.mjs — reads split across replicas while writes repeat at // every node, and asynchronous lag converted into stale tracking responses. All arithmetic. const READ = 41.67; // K01: read/s behind the cache const WRITE = 194.44; // lesson 01: store write ops/s in the split layout const STORED = 718.24; // lesson 01: stored data GB const EVENTS = 97.22; // K01: peak write requests/s (state event) const V3 = 400_000; // K01: daily shipments const VD2 = 3; // this lesson's assumption: how long a shipment stays in transit (days) const INTRANSIT = V3 * VD2, DAY = 86_400; const b = (x, n = 2) => x.toFixed(n); console.log(`${"replicas".padStart(9)}${"read/s per replica".padStart(20)}${"write/s per node".padStart(19)}` + `${"node load".padStart(12)}${"write/read".padStart(13)}${"replica gain".padStart(14)}${"total GB".padStart(11)}`); let previous = null; for (const n of [1, 2, 3, 4, 5]) { const read = READ / n, load = WRITE + read; console.log(`${String(n).padStart(9)}${b(read).padStart(20)}${b(WRITE).padStart(19)}` + `${b(load).padStart(12)}${b(WRITE / read).padStart(13)}` + `${(previous === null ? "-" : b(previous - load)).padStart(14)}${b(STORED * (1 + n)).padStart(11)}`); previous = load; } console.log(`single node (lesson 01): load ${b(WRITE + READ)} ops/s, write/read ${b(WRITE / READ)}, ` + `${b(STORED)} GB`); console.log(`\nshipments in transit = ${INTRANSIT.toLocaleString("en-US")} (VD2 = ${VD2} days)`); console.log(`${"lag s".padStart(9)}${"changed shipments in window".padStart(29)}` + `${"stale read ratio".padStart(19)}${"stale responses/s".padStart(19)}${"stale responses/day".padStart(21)}`); for (const L of [0.5, 1, 5, 10]) { const changed = EVENTS * L, ratio = changed / INTRANSIT, stale = READ * ratio; console.log(`${b(L, 1).padStart(9)}${b(changed).padStart(29)}${ratio.toExponential(2).padStart(19)}` + `${b(stale, 5).padStart(19)}${b(stale * DAY, 1).padStart(21)}`); } ``` ``` replicas read/s per replica write/s per node node load write/read replica gain total GB 1 41.67 194.44 236.11 4.67 - 1436.48 2 20.84 194.44 215.28 9.33 20.84 2154.72 3 13.89 194.44 208.33 14.00 6.95 2872.96 4 10.42 194.44 204.86 18.66 3.47 3591.20 5 8.33 194.44 202.77 23.33 2.08 4309.44 single node (lesson 01): load 236.11 ops/s, write/read 4.67, 718.24 GB shipments in transit = 1,200,000 (VD2 = 3 days) lag s changed shipments in window stale read ratio stale responses/s stale responses/day 0.5 48.61 4.05e-5 0.00169 145.8 1.0 97.22 8.10e-5 0.00338 291.7 5.0 486.10 4.05e-4 0.01688 1458.4 10.0 972.20 8.10e-4 0.03376 2916.8 ``` These numbers belong to the **calculation** class; their inputs are K01's arithmetic, lesson 01's results, and VD2. #### Reading the Numbers **Adding a replica does not lower the write load at all** — the write-per-node column reads 194.44 ops/s across all five rows. This is a **floor**, not a ceiling: whatever the replica count, every node applies every state event. The only column that splits is the read, starting from 41.67. **The gain runs out quickly:** going from the first replica to the second drops node load 20.84 ops/s; second to third, 6.95; fourth to fifth, 2.08. The fifth replica delivers a tenth of what the first one did, because the only share that can shrink is the read, already 18 percent of the load. Leader–follower replication is, for that reason, a **read-heavy** tool; K01's arithmetic, where the store sees it, is write-heavy, and lesson 01's document update grew that weight further. The **write/read ratio at store** grows with every replica: 2.33 in K01, 4.67 after lesson 01, 14.00 at three replicas. The ratio is not degradation but a definition of the layout — as reads spread out, each node's mix skews toward writes, and that is where the decision to tune a node for writes gets read off. **`stored data GB` is multiplied outright:** 718.24 GB on a single node becomes 2,872.96 GB with three replicas and 4,309.44 GB with five. This is replication's bill, and the decision that moves K01's storage row most — the original 712.48 GB calculation can climb to six times its size. **Lag turns into a response count in the second table.** At one second of replication lag, changed shipments in the window number 97.22; against the 1,200,000 shipments in transit that is a ratio of 8.10e-5, and the payoff for the 41.67 reads reaching the store is 291.7 stale tracking responses a day — 1,458.4 at five seconds. The number is small because the divisor is large: the stale response ratio is the change rate over shipments in transit, and a short-lived, fast-changing data set would give a different result. Whether reading from a replica is acceptable is, for that reason, not a principle but a **result of this division**. A replica's second payoff is **service availability**: when the leader goes down, one of the replicas is promoted. Failover and the majority rule were measured in the Relational Database Administration course and are not repeated here. #### Multi-Leader and Collision As long as writes stay on one node, writes do not scale. A **multi-leader** layout lifts that constraint: more than one node accepts writes and applies the others' changes — at the cost of something the single leader gave for free, a **single order** for writes. The pattern makes this concrete. P2 is the state event write by carrier, and at the moment of transfer, a shipment sits **between two zones**: the departure zone writes "handed to transfer" and the arrival zone writes "received from transfer" a few seconds apart. **VD3 — a shipment's rate of near-simultaneous writes from two zones is 0.02.** Rationale: one of the seven events falls on the transfer moment, and both zones write at that moment. Its sensitivity is linear; it is given below at 0.04. Not added to K01's table. ```js // replication/multi-leader.mjs — two leaders writing the same shipment. This is a MODEL: a // round is an abstract step, propagation round is a parameter, not a measured duration. What // is counted is whether the two nodes show the same final state once propagation is done. const EVENTS = 20_000, SHIPMENTS = 4000, PROPAGATION = 3; // model parameters const VD3 = 0.02; // assumption: a shipment's rate of near-simultaneous writes from two zones let s = 20260730 % 2147483647; // fixed seed: same sequence on every run const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const events = []; for (let t = 0; t < EVENTS; t += 1) { const sh = Math.floor(rand() * SHIPMENTS), zone = rand() < 0.5 ? "A" : "B"; events.push({ t, sh, zone, label: `o${t}` }); if (rand() < VD3) { // transfer moment: the opposite zone writes too const delay = 1 + Math.floor(rand() * PROPAGATION); events.push({ t: t + delay, sh, zone: zone === "A" ? "B" : "A", label: `o${t}b` }); } } const shipmentEvents = new Map(); for (const e of events) (shipmentEvents.get(e.sh) ?? shipmentEvents.set(e.sh, []).get(e.sh)).push(e); let overlapping = 0; for (const list of shipmentEvents.values()) if (list.some((a) => list.some((c) => a.zone !== c.zone && Math.abs(a.t - c.t) <= PROPAGATION))) overlapping += 1; function run(rule) { const node = { A: new Map(), B: new Map() }; for (const d of ["A", "B"]) { const incoming = events.map((e) => ({ arrival: e.zone === d ? e.t : e.t + PROPAGATION, local: e.zone === d ? 1 : 0, e })); incoming.sort((x, y) => x.arrival - y.arrival || y.local - x.local); // arrival order, local first for (const { e } of incoming) { if (rule === "last-writer") { node[d].set(e.sh, e.label); continue; } const k = node[d].get(e.sh) ?? node[d].set(e.sh, []).get(e.sh); // append: event set k.push(e); } } const finalState = (d, sh) => { const v = node[d].get(sh); if (rule === "last-writer") return v; return v.slice().sort((x, y) => y.t - x.t || (x.label < y.label ? 1 : -1))[0].label; // deterministic rule }; let diverged = 0; for (const sh of node.A.keys()) if (finalState("A", sh) !== finalState("B", sh)) diverged += 1; const records = rule === "last-writer" ? node.A.size : [...node.A.values()].reduce((a, v) => a + v.length, 0); return { diverged, records, touched: node.A.size }; } console.log(`model: ${events.length} events, ${shipmentEvents.size} shipments touched, ` + `propagation ${PROPAGATION} rounds, VD3 = ${VD3}`); console.log(`shipments with near-simultaneous writes from two zones = ${overlapping} ` + `(${((100 * overlapping) / shipmentEvents.size).toFixed(2)}% of touched)`); console.log(`\n${"merge rule".padEnd(22)}${"diverged shipments".padStart(20)}` + `${"share of overlapping".padStart(22)}${"records kept at node".padStart(22)}`); for (const rule of ["last-writer", "append"]) { const r = run(rule); console.log(`${rule.padEnd(22)}${String(r.diverged).padStart(20)}` + `${(r.diverged / overlapping).toFixed(4).padStart(22)}${String(r.records).padStart(22)}`); } const DAILY_EVENTS = 2_800_000, V4 = 7; // K01: daily state events, events/shipment const ratio = run("last-writer").diverged / shipmentEvents.size; console.log(`\nscaled to K01: ${DAILY_EVENTS.toLocaleString("en-US")} events and ` + `${(DAILY_EVENTS / V4).toLocaleString("en-US")} shipments a day (V4 = ${V4})`); console.log(`under the last-writer rule, ${Math.round((DAILY_EVENTS / V4) * ratio).toLocaleString("en-US")} ` + `shipments a day show a different final state in the two zones; VD3 = 0.04 would be about twice that`); ``` ``` model: 20365 events, 3971 shipments touched, propagation 3 rounds, VD3 = 0.02 shipments with near-simultaneous writes from two zones = 363 (9.14% of touched) merge rule diverged shipments share of overlapping records kept at node last-writer 66 0.1818 3971 append 0 0.0000 20365 scaled to K01: 2,800,000 events and 400,000 shipments a day (V4 = 7) under the last-writer rule, 6,648 shipments a day show a different final state in the two zones; VD3 = 0.04 would be about twice that ``` The model says three things. First, **collision is not rare**: at an average of five events per touched shipment, a 2-percent-per-event transfer rate turns 9.14 percent of shipments into ones written from both zones. Second, not every colliding write produces divergence — only 66 of 363 shipments, one in five. **Divergence arises only when the two events reach the two nodes in a different order:** node A applies its local event first, node B applies the other one first, and once propagation finishes the two nodes hold two different final states that nothing corrects on its own. Scaled to K01, that is 6,648 shipments a day with a different final state in the two zones; doubling VD3 roughly doubles the number. Third, and the important point for design: **divergence is a consequence of the storage shape, not of the data.** Under the append rule the same event sequence produces zero divergence, because the merge is a set **union** and a union does not depend on order — the final state derives from the set by a deterministic rule, and the two nodes reach the same result because they hold the same set. The cost sits in the last column: records held at a node rise from 3,971 to 20,365, a **5.13×** factor. In this course's flow that cost is on paper only, since K01 already stores every event (V6, V12); what actually changes is that the final state stops being an overwritten field and becomes a **derived value**. P2's shape is, for that reason, decisive: a state event is an insert, so it fits a multi-leader layout, while an overwritten field would not. The multi-leader decision questions not the data but the **shape of the write**. Replicas converging once propagation finishes is what eventual consistency promises, defined in the Introduction to System Design course; this measurement shows the promise holds only when the merge rule is **order-independent**. #### Summary - In a leader–follower layout, the read splits and the write does not: write per node stays at 194.44 ops/s across all five replicas, while read per replica falls from 41.67 to 8.33. - The gain from adding a replica runs out quickly: node load drops 20.84 ops/s on the first addition and only 2.08 on the fourth; the write/read ratio at store climbs from 4.67 to 14.00 at three replicas. - `stored data GB` is multiplied outright: 718.24 GB becomes 2,872.96 GB with three replicas and 4,309.44 GB with five. - Lag converts into a response count: with VD2 = 3 days and 1,200,000 shipments in transit, one second of lag produces 291.7 stale tracking responses a day, and five seconds produces 1,458.4. - Under multi-leader with VD3 = 0.02, 9.14 percent of shipments are written from two zones; under the last-writer rule, one in five of those produces permanent divergence — 6,648 shipments a day at K01's scale. - Divergence arises from the storage shape: the append rule produces zero divergence, at the cost of records held at a node climbing from 3,971 to 20,365 (a 5.13× factor). #### Next Step Replication split the read but left the same 194.44 write on every node and multiplied stored data by the replica count. Both limits come from the same place: **all the data sits everywhere.** Yet the data the store carries is not a single whole — pricing's questions and delivery operations' questions touch different records, as lesson 01's scan measurement already showed. The next lesson splits the data not by key but by **function**: once the two contexts split into their own stores, what happens to the requests and data each carries, how many stores the write path stretches across, how many stores a cross-context query has to reach, and what duplicating the shared fields adds to stored data. ### Federation Source: https://academia.sh/en/courses/scaling-the-data/federation Splitting data by function rather than by key: separating the pricing and delivery operations contexts into their own stores, measuring in a real engine the stores touched and the keys carried between stores by a query that touches both contexts, calculating what duplicating shared fields adds to stored data, and taking the batch scan out of the online store. Replication split the read but left the same 194.44 write on every node and multiplied stored data by the replica count. Both limits come from the same place: **all the data sits everywhere.** Yet the data the store carries is not a single whole — pricing's questions and delivery operations' questions touch different records, as lesson 01's scan measurement already showed: the period scan dropped from 21,000 records to 3,000 with no need for event detail at all. **Federation** is splitting data by function rather than by key: each context gets its own store, and that store serves only that context's questions. The split has nothing to do with the key — the same tracking number can appear in both stores; what it has to do with is **which question** that number is asked with. #### Two Contexts, Two Stores The shipment tracking and pricing service has two contexts. **Delivery operations** carries P1 and P2: single lookup by tracking number and the state event write from the carrier. **Pricing** carries P3: periodic batch scan by seller, tariff, contract, and invoice line. The split is not clean, because the two contexts both need some of the same records. Pricing must know when a shipment was accepted and when it was delivered, but it does not need to know the route steps in between. **VD4 — the share of events that concern pricing is 2/7.** Rationale: the invoice depends on acceptance and delivery time; only two of V4's seven events affect the fee. Its sensitivity is given below at 3/7. **VD5 — when the shipment record splits across the two contexts, the shares are 500 and 600 bytes.** Rationale: in V7's 900-byte record, weight, volume, contract, and tariff go to pricing, and zone and route fields go to operations; the tracking number, seller ID, and period tag have to sit in both stores, so the total climbs to 1,100 bytes, and the duplicated share is 200 bytes. Its sensitivity is given below with 400 bytes of duplication. Neither assumption is added to K01's table. #### How Many Stores a Split Query Touches The setup below builds three stores in memory — the unfederated single store and two federated stores — and asks the same questions of both. What is measured is the number of stores touched, the records processed, and, in the federated layout, the number of keys carried between stores. ```js // federation/cross.mjs — the stores touched, records processed, and keys carried between stores // by the same queries in a single store and two federated stores. A real engine runs with node:sqlite. import { DatabaseSync } from "node:sqlite"; const SELLER = 20, DAY = 30, DAILY = 100, EVENTS = 7, LAST = EVENTS - 1; // model scale const [single, pricing, ops] = [0, 1, 2].map(() => new DatabaseSync(":memory:")); single.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, day INT, tariff INT, zone INT); CREATE INDEX s_seller ON shipment(seller, day); CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`); pricing.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, day INT, tariff INT); CREATE INDEX s_seller ON shipment(seller, day); CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`); ops.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, zone INT); CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`); const counter = { single: 0, pricing: 0, ops: 0 }; for (const [name, db] of [["single", single], ["pricing", pricing], ["ops", ops]]) db.function("count", { deterministic: false }, (x) => { counter[name] += 1; return x; }); const total = () => counter.single + counter.pricing + counter.ops; const sS = single.prepare("INSERT INTO shipment VALUES(?,?,?,?,?)"), eS = single.prepare("INSERT INTO event VALUES(?,?,?)"); const sP = pricing.prepare("INSERT INTO shipment VALUES(?,?,?,?)"), eP = pricing.prepare("INSERT INTO event VALUES(?,?,?)"); const sO = ops.prepare("INSERT INTO shipment VALUES(?,?,?)"), eO = ops.prepare("INSERT INTO event VALUES(?,?,?)"); for (const d of [single, pricing, ops]) d.exec("BEGIN"); for (let v = 0; v < SELLER; v += 1) for (let day = 0; day < DAY; day += 1) for (let k = 0; k < DAILY; k += 1) { const trackingNo = `TR-${v}-${day}-${k}`; sS.run(trackingNo, v, day, 100 + v, day % 5); sP.run(trackingNo, v, day, 100 + v); sO.run(trackingNo, v, day % 5); for (let i = 0; i < EVENTS; i += 1) { eS.run(trackingNo, i, `state${i}`); eO.run(trackingNo, i, `state${i}`); if (i === 0 || i === LAST) eP.run(trackingNo, i, `state${i}`); // VD4: acceptance and delivery only } } for (const d of [single, pricing, ops]) d.exec("COMMIT"); const count = (db, t) => db.prepare(`SELECT count(*) c FROM ${t}`).get().c; console.log(`model: ${count(single, "shipment")} shipments; event records -> single store ${count(single, "event")}, ` + `ops ${count(ops, "event")}, pricing ${count(pricing, "event")}`); const cross = () => { // carry keys to ask the second store const a = pricing.prepare("SELECT count(trackingNo) AS trackingNo FROM shipment WHERE seller = ?").all(10).map((r) => r.trackingNo); const q = ops.prepare(`SELECT count(trackingNo) FROM event WHERE trackingNo = ? AND seq = ${LAST}`); for (const k of a) q.all(k); return a.length; }; const JOBS = [ ["P1 single lookup / federated", 1, () => (ops.prepare("SELECT count(trackingNo) FROM event WHERE trackingNo = ? ORDER BY seq DESC LIMIT 3").all("TR-10-15-50"), 0)], ["P3 period scan / federated", 1, () => (pricing.prepare("SELECT count(trackingNo) FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29").all(10), 0)], ["Q4 last route step / single store", 1, () => (single.prepare( `SELECT count(s.trackingNo), e.state FROM shipment s JOIN event e ON e.trackingNo = s.trackingNo WHERE s.seller = ? AND e.seq = ${LAST}`).all(10), 0)], ["Q4 last route step / federated", 2, cross], ["Q5 fee by delivery date / federated", 1, () => (pricing.prepare( `SELECT count(s.trackingNo), s.tariff FROM shipment s JOIN event e ON e.trackingNo = s.trackingNo WHERE s.seller = ? AND e.seq = ${LAST}`).all(10), 0)], ["Q5 if the delivery event were not copied", 2, cross], ]; console.log(`\n${"query / layout".padEnd(40)}${"stores".padStart(7)}${"records processed".padStart(19)}${"keys carried".padStart(15)}`); for (const [name, stores, run] of JOBS) { counter.single = 0; counter.pricing = 0; counter.ops = 0; const keys = run(); console.log(`${name.padEnd(40)}${String(stores).padStart(7)}${String(total()).padStart(19)}${String(keys).padStart(15)}`); } const bytes = Buffer.byteLength("TR-10-15-50"); console.log(`\nkey is ${bytes} bytes: in the federated layout, Q4 carries ` + `${(3000 * bytes) / 1000} kB between stores; 0 in the single store`); ``` ``` model: 60000 shipments; event records -> single store 420000, ops 420000, pricing 120000 query / layout stores records processed keys carried P1 single lookup / federated 1 7 0 P3 period scan / federated 1 3000 0 Q4 last route step / single store 1 3000 0 Q4 last route step / federated 2 6000 3000 Q5 fee by delivery date / federated 1 3000 0 Q5 if the delivery event were not copied 2 6000 3000 key is 11 bytes: in the federated layout, Q4 carries 33 kB between stores; 0 in the single store ``` These numbers belong to the **measurement** class; they come from an engine running on this machine and are deterministic. The first two rows carry the good news: each context's own questions stay in its own store and cost exactly what they cost in the single store. Federation neither helps nor hurts these questions. The fourth and fifth rows carry the real information. **Q4 — the last route step of a seller's shipments over a period** sits at the intersection of both contexts: seller and period are pricing's field, route step is operations' field. In the single store this is one query; the engine performs the join internally and processes 3,000 records. In the federated layout the query splits in two: 3,000 keys are read from pricing, the application carries them and asks the delivery store, and total records processed climbs to 6,000. What is lost is the join itself; what replaces it is 3,000 keys carried between stores — **33 kilobytes**. This is the cost federation actually imposes on the application: work the engine did for free, application code now pays for. The last two rows measure VD4's rationale. **Q5 — fee by delivery date** stays in a single store and finishes at 3,000 records, because a copy of the delivery event sits in the pricing store. Without that copy, the same question would fall into the same shape as Q4: 2 stores, 6,000 records, 3,000 keys carried. In other words, **the duplicated 2/7 of events brings a cross-context query down to a single store.** That is the measure of whether a context boundary is drawn correctly: the boundary sitting in the right place means the number of cross-context questions stays small. #### Back to the Arithmetic Duplication's cost collects in two of K01's growth rows. ```js // federation/cost.mjs — what a store split into two contexts costs on K01's rows const V3 = 400_000, V4 = 7, V6 = 220, V7 = 900, V8 = 3, V12 = 730; // K01 assumptions const READ = 41.67, EVENT_WRITE = 97.22, SCAN = 833.33; // K01 arithmetic const GROWTH = 976, STORED = 712.48; // K01 arithmetic const DOCUMENT = 194.44; // lesson 01: state event writes to 2 targets on the delivery side const VD4 = 2 / 7; // assumption: share of events that concern pricing const VD5 = [500, 600]; // assumption: shares of the split shipment record (pricing, ops) bytes const SHIPMENT_WRITE = 13.89; // K01 derived: (V3 / 86,400) x V8 = shipment creation, peak records/s const b = (x, n = 2) => x.toFixed(n); const ops = SHIPMENT_WRITE + DOCUMENT + READ; const pricing = SHIPMENT_WRITE + EVENT_WRITE * VD4; console.log(`${"store".padEnd(24)}${"requests+ops/s".padStart(15)}${"scan records/s".padStart(16)}` + `${"daily MB".padStart(10)}${"stored GB".padStart(12)}`); const opsMB = (V3 * VD5[1] + V3 * V4 * V6) / 1e6, pricingMB = (V3 * VD5[0] + V3 * V4 * VD4 * V6) / 1e6; console.log(`${"delivery operations".padEnd(24)}${b(ops).padStart(15)}${"0".padStart(16)}` + `${b(opsMB).padStart(10)}${b((opsMB * V12) / 1000).padStart(12)}`); console.log(`${"pricing".padEnd(24)}${b(pricing).padStart(15)}${b(SCAN).padStart(16)}` + `${b(pricingMB).padStart(10)}${b((pricingMB * V12) / 1000).padStart(12)}`); console.log(`${"single store (unfederated)".padEnd(24)}${b(SHIPMENT_WRITE + DOCUMENT + READ).padStart(15)}` + `${b(SCAN).padStart(16)}${b(GROWTH).padStart(10)}${b(STORED).padStart(12)}`); const totalMB = opsMB + pricingMB; console.log(`\ntotal ops ${b(ops + pricing)} /s = ${b((ops + pricing) / (SHIPMENT_WRITE + DOCUMENT + READ))}x the single store`); console.log(`daily written ${b(totalMB)} MB (K01: ${GROWTH}) = ${b(totalMB / GROWTH)}x`); console.log(`stored ${b((totalMB * V12) / 1000)} GB (K01: ${STORED}) = ${b((totalMB * V12) / 1000 / STORED)}x`); const path = (SHIPMENT_WRITE * 2 + EVENT_WRITE * (1 + VD4)) / (SHIPMENT_WRITE + EVENT_WRITE); console.log(`length of the write path = ${b(path, 3)} stores (shipment 2, state event ${b(1 + VD4, 3)})`); const sensitivity = (vd4, dup) => ((V3 * (V7 + dup) + V3 * V4 * V6 * (1 + vd4)) / 1e6); console.log(`\nsensitivity: VD4 = 3/7 -> daily ${b(sensitivity(3 / 7, 200))} MB, ` + `stored ${b((sensitivity(3 / 7, 200) * V12) / 1000)} GB`); console.log(` duplicated identifier 400 bytes -> daily ${b(sensitivity(VD4, 400))} MB, ` + `stored ${b((sensitivity(VD4, 400) * V12) / 1000)} GB`); console.log(`all three together: lesson 01's document store adds 5.76 GB, with lesson 02's three replicas ` + `${b(3 * ((totalMB * V12) / 1000 + 5.76))} GB`); ``` ``` store requests+ops/s scan records/s daily MB stored GB delivery operations 250.00 0 856.00 624.88 pricing 41.67 833.33 376.00 274.48 single store (unfederated) 250.00 833.33 976.00 712.48 total ops 291.67 /s = 1.17x the single store daily written 1232.00 MB (K01: 976) = 1.26x stored 899.36 GB (K01: 712.48) = 1.26x length of the write path = 1.375 stores (shipment 2, state event 1.286) sensitivity: VD4 = 3/7 -> daily 1320.00 MB, stored 963.60 GB duplicated identifier 400 bytes -> daily 1312.00 MB, stored 957.76 GB all three together: lesson 01's document store adds 5.76 GB, with lesson 02's three replicas 2715.36 GB ``` These rows belong to the **calculation** class. **The batch scan comes out of the online store.** Delivery operations' scan column is zero; the four-hour job reading 833.33 records per second no longer shares a disk, memory, or lock manager with the tracking query. This is federation's scaling rationale, and it fits in a single number: **the record rate the online path sees falls from 833.33 to zero.** **Total ops climbs to a 1.17× factor.** The single store's 250.00 ops/s splits into 250.00 and 41.67; the extra 41.67 is pricing writing its own copies. The write path's length climbs to 1.375 stores: shipment creation writes to two stores, and a state event writes to an average of 1.286. A longer write path raises a durability question — what happens if one of the two stores refuses the write — but that question is not this course's subject. **`daily data growth MB` and `stored data GB` climb together to a 1.26× factor.** 976 MB becomes 1,232 MB, 712.48 GB becomes 899.36 GB. The growth has two sources and each can be read separately: duplicated identifier fields push the shipment record from 900 to 1,100 bytes, and duplicated acceptance and delivery events add 176 MB a day. If VD4 rose to a third, stored data would be 963.60 GB; if the duplicated identifier rose to 400 bytes, it would be 957.76 GB — the two assumptions carry about the same magnitude of effect. **The last row gives the combination of three decisions.** Applied together, lesson 01's document store, this lesson's federation, and lesson 02's three replicas bring stored data to 2,715.36 GB — a **3.81×** factor over K01's 712.48 GB. This is the common trait of data-layer decisions: all of them grow the same number, and their effects **multiply, not add.** #### Summary - Federation splits data by function rather than by key; the boundary is drawn by the context's questions, and the same tracking number can appear in both stores. - A context's own questions stay in its own store, but the Q4 question at the intersection of both contexts climbs from 1 store to 2, from 3,000 records to 6,000, and carries 3,000 keys (33 kB) between stores. - Duplicating 2/7 of events brings a cross-context question (Q5) down to a single store; the measure of a correct boundary is the number of cross-context questions. - The batch scan comes out of the online store: the scan rate delivery operations' store sees falls from 833.33 records/s to zero. - Total ops climbs from 250.00 to 291.67 ops/s (a 1.17× factor), and the write path stretches to 1.375 stores. - Duplication brings two of K01's growth rows to a 1.26× factor (976 to 1,232 MB, 712.48 to 899.36 GB); combined with lessons 01 and 02's decisions, stored data reaches 2,715.36 GB. #### Next Step Federation split the store's work, but each context's own data still sits whole on a single node. The delivery operations store carries 624.88 GB and takes 250.00 operations per second; these numbers do not shrink by splitting the context further, because there is no context left to split. The one path left is to split the data **within itself**: distribute the same context's records across independent nodes by a key. The next lesson takes up that decision — how the key is chosen, how the choice changes the number of nodes each of the three access patterns touches, how many times over the busiest node carries what the others do, and how many bytes change hands when the node count changes. ### Sharding Source: https://academia.sh/en/courses/scaling-the-data/sharding Distributing the same context's data across independent nodes by a key: comparing three candidate shard keys by data distribution, hot spot ratio, and the node count each access pattern touches, measuring the data and requests the busiest node carries, converting the fraction of keys that move on rebalancing into bytes moved, and choosing the key by the most frequent pattern. Federation split the store's work, but each context's own data still sits whole on a single node. The delivery operations store carries 624.88 GB and takes 250.00 operations per second; these numbers do not shrink by splitting the context further, because there is no context left to split. The one path left is to split the data **within itself**. **Sharding** is distributing the same context's records across independent nodes by a key. It should not be confused with the **partitioning** from the Relational Database Administration course: partitioning divides a table into pieces inside the same engine, while sharding distributes those pieces across separate engines and machines. That course counted the costs sharding imposes on the application — joins, uniqueness and transaction boundaries, the scatter-gather pattern breaking the tail, the mapping method setting the move ratio on rebalancing — and they are not retold here. This lesson's question is a **scaling decision**: which key is chosen, and what the choice does to the access patterns. #### Key Candidates and Their Distributions The shipment data has three candidates, and all three are the key of a real question: tracking number (P1's key), carrier (P2's key), and seller (P3's key). Two properties separate the candidates — how many distinct values they take, and how evenly those values are distributed. **VD6 — the distribution of the three candidate keys.** Tracking number is uniform and distinct for every shipment; carrier takes 12 distinct values and the largest carrier's share is 0.35; seller takes 4,000 distinct values and the largest 1-percent slice (40 sellers) produces 0.30 of shipments. Rationale: tracking number is a generated identifier; carrier traffic splits among a small number of large players; seller volume concentrates in a small number of large sellers. The seller count comes from K01's arithmetic (4,000 invoice lines a day = 4,000 sellers); the rest is this lesson's assumption and is not added to K01's table. Sensitivity is given below at a carrier share of 0.50. Mapping a key to a node uses **consistent hashing**. The rule itself, and the fraction of keys that move when the node count changes, were measured in the Traffic Layer course; here it is used only for data placement, and that measurement is not repeated. ```js // sharding/key.mjs — the data distribution, hot spot, touched nodes, and rebalancing cost of // three candidate shard keys. The placement rule is consistent hashing; the rule itself and the // move ratio were measured in M19/K02, used here only for data placement. THIS IS A MODEL. const SHIPMENTS = 100_000, SHARDS = 8, VIRTUAL = 200; // model parameters const CARRIER = 12, CARRIER_SHARE = 0.35; // VD6 const SELLER = 4000, BIG = 40, BIG_SHARE = 0.30; // VD6 let s = 20260731 % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; function hash(text) { // FNV-1a; the function was set up in M01/K03 let h = 2166136261; for (let i = 0; i < text.length; i += 1) h = Math.imul(h ^ text.charCodeAt(i), 16777619) >>> 0; h ^= h >>> 15; h = Math.imul(h, 2246822507) >>> 0; h ^= h >>> 13; return h >>> 0; } const ring = (n) => { const nodes = []; for (let d = 0; d < n; d += 1) for (let v = 0; v < VIRTUAL; v += 1) nodes.push([hash(`d${d}#${v}`), d]); return nodes.sort((a, b) => a[0] - b[0]); }; const place = (key, r) => { const c = hash(key); if (c > r[r.length - 1][0]) return r[0][1]; let lo = 0, hi = r.length - 1; while (lo < hi) { const m = (lo + hi) >> 1; if (r[m][0] < c) lo = m + 1; else hi = m; } return r[lo][1]; }; const records = []; for (let i = 0; i < SHIPMENTS; i += 1) records.push({ trackingNo: `TR-${1_000_000 + i}`, carrier: `T${rand() < CARRIER_SHARE ? 0 : 1 + Math.floor(rand() * (CARRIER - 1))}`, seller: `S${rand() < BIG_SHARE ? Math.floor(rand() * BIG) : BIG + Math.floor(rand() * (SELLER - BIG))}`, }); const CANDIDATES = ["trackingNo", "carrier", "seller"]; const ring8 = ring(SHARDS), ring9 = ring(SHARDS + 1); const READ = 41.67, WRITE = 97.22; // K01: requests/s reaching the store for P1 and P2 const GB = 624.88, DAILY_MB = 856; // lesson 03: delivery operations store const TOUCHED = { trackingNo: [1, 1, SHARDS], carrier: [SHARDS, 1, SHARDS], seller: [SHARDS, SHARDS, 1] }; const b = (x, n = 2) => x.toFixed(n); console.log(`model: ${SHIPMENTS.toLocaleString("en-US")} shipments, ${SHARDS} shards, ${VIRTUAL} virtual nodes`); console.log(`\n${"shard key".padEnd(16)}${"distinct keys".padStart(15)}${"hot spot".padStart(12)}` + `${"busiest GB".padStart(13)}${"busiest ops/s".padStart(16)}${"8->9 moved".padStart(13)}${"moved GB".padStart(11)}`); for (const a of CANDIDATES) { const y8 = records.map((k) => place(k[a], ring8)), y9 = records.map((k) => place(k[a], ring9)); const counts = new Array(SHARDS).fill(0); for (const d of y8) counts[d] += 1; const busiest = Math.max(...counts) / SHIPMENTS, hotspot = busiest * SHARDS; const moved = y8.filter((d, i) => d !== y9[i]).length / SHIPMENTS; console.log(`${a.padEnd(16)}${String(new Set(records.map((k) => k[a])).size).padStart(15)}` + `${b(hotspot, 3).padStart(12)}${b(GB * busiest).padStart(13)}` + `${b((READ + WRITE) * busiest).padStart(16)}${b(moved, 4).padStart(13)}${b(GB * moved).padStart(11)}`); } console.log(`perfect distribution has hot spot 1.000; at ${SHARDS} shards, per node ` + `${b(GB / SHARDS)} GB and ${b((READ + WRITE) / SHARDS)} ops/s`); { const y8 = records.map((k) => place(k.trackingNo, ring8)), y9 = records.map((k) => place(k.trackingNo, ring9)); const moved = y8.filter((d, i) => d !== y9[i]).length / SHIPMENTS, movedGB = GB * moved * 1000; console.log(`moved with tracking number as key: ${b(movedGB / 1000)} GB = ` + `${b(movedGB / DAILY_MB, 1)} days of the delivery store's daily growth (lesson 03: ${DAILY_MB} MB/day)`); } console.log(`\n${"shard key".padEnd(16)}${"P1 nodes".padStart(10)}${"P2 nodes".padStart(10)}` + `${"P3 nodes".padStart(10)}${"online touches/s".padStart(19)}${"baseline factor".padStart(17)}`); for (const a of CANDIDATES) { const [p1, p2, p3] = TOUCHED[a], touches = READ * p1 + WRITE * p2; console.log(`${a.padEnd(16)}${String(p1).padStart(10)}${String(p2).padStart(10)}${String(p3).padStart(10)}` + `${b(touches).padStart(19)}${b(touches / (READ + WRITE)).padStart(17)}`); } console.log(`online touches in the unsharded store ${b(READ + WRITE)} /s = K01's requests/s reaching the store`); // VD6 sensitivity: if the largest carrier's share were 0.50 instead of 0.35 (carrier field regenerated) { const counts = new Array(SHARDS).fill(0); for (const k of records) counts[place(`T${rand() < 0.5 ? 0 : 1 + Math.floor(rand() * (CARRIER - 1))}`, ring8)] += 1; const busiest = Math.max(...counts) / SHIPMENTS; console.log(`\nsensitivity: if the largest carrier's share were 0.50, hot spot would be ` + `${b(busiest * SHARDS, 3)} and the busiest node ${b(GB * busiest)} GB (in the table, at 0.35: ` + `${b(3.748, 3)} and ${b(292.78)} GB)`); } ``` ``` model: 100,000 shipments, 8 shards, 200 virtual nodes shard key distinct keys hot spot busiest GB busiest ops/s 8->9 moved moved GB trackingNo 100000 1.116 87.14 19.37 0.1193 74.54 carrier 12 3.748 292.78 65.07 0.1182 73.85 seller 4000 1.148 89.63 19.92 0.1100 68.74 perfect distribution has hot spot 1.000; at 8 shards, per node 78.11 GB and 17.36 ops/s moved with tracking number as key: 74.54 GB = 87.1 days of the delivery store's daily growth (lesson 03: 856 MB/day) shard key P1 nodes P2 nodes P3 nodes online touches/s baseline factor trackingNo 1 1 8 138.89 1.00 carrier 8 1 8 430.58 3.10 seller 8 8 1 1111.12 8.00 online touches in the unsharded store 138.89 /s = K01's requests/s reaching the store sensitivity: if the largest carrier's share were 0.50, hot spot would be 4.713 and the busiest node 368.15 GB (in the table, at 0.35: 3.748 and 292.78 GB) ``` These numbers belong to the **calculation** class: they are deterministic, come from a fixed-seed generator and a fixed placement rule, and do not depend on the machine. #### The Hot Spot Depends on How Many Values the Key Takes The first table's hot spot column separates the three candidates immediately. **Hot spot** is the busiest node's share divided by the perfect share; 1.000 is a perfect distribution. Tracking number gives 1.116 and seller gives 1.148; both are acceptable. Carrier gives 3.748: the busiest node carries almost four times the perfect share, 292.78 GB and 65.07 operations per second. In an eight-node cluster, 78.11 GB and 17.36 ops/s are expected per node, and one node takes four times that — for that node, sharding might as well not have happened. The reason is **how many distinct values the key takes.** Carrier takes only 12 values, and the largest one alone carries 0.35 of the load; no placement rule can split a single key across two nodes, because the key is the unit of indivisibility. Seller is skewed too — 40 sellers produce 0.30 of volume — but spread across 4,000 distinct values, the skew dissolves in placement: the largest seller's own share is 0.0075 and cannot capture a single node by itself. The rule is this: **what creates a hot spot is not skew, but skew concentrated in a small number of key values.** The sensitivity row confirms it — at a carrier share of 0.50, the hot spot would climb to 4.713 and the busiest node to 368.15 GB. #### The Number of Nodes Touched Changes with the Pattern The second table flips the choice around. If a request cannot filter on the shard key, it has to go to every node. With tracking number as the key, P1 and P2 each touch one node: the tracking query already knows the number, and the state event carries it too. The online touch rate stays at 138.89 node-touches/s — K01's `requests reaching store/s` number itself, since sharding adds no extra touch to the online path. With carrier as the key, P1 cannot tell the carrier from the number, so it asks all eight nodes at once, and the rate climbs to 430.58, a **3.10×** factor over the baseline. With seller as the key, both P1 and P2 go to all eight nodes: 1,111.12 touches/s, exactly **8.00×** the baseline. The third pattern completes the table. P3 stays on a single node only with seller as the key; with the others, it spreads across all eight. But P3 runs once a day with a four-hour window, so spreading across eight nodes is not a latency problem for it, only a coordination task. P1 and P2, by contrast, flow continuously at 138.89 requests a second, and tracking reads carry a 200-millisecond threshold. The decision comes out of where these two tables intersect, and it fits one sentence: **the shard key is the field the most frequent, tightest-threshold pattern filters on.** Tracking number wins on both tables — hot spot of 1.116, online touch at the baseline. Carrier keeps P2 on a single node but spreads P1 across eight and adds a 3.748 hot spot on top of that; seller saves P3 but multiplies the online path by eight. #### The Cost of Rebalancing The last two columns give what happens when a node is added. Going from eight shards to nine, 0.1193 of keys move under tracking number as the key — close to the theoretical lower bound of 1/9, or 0.1111. The ratio itself was what the Traffic Layer course measured; here it converts to bytes: **74.54 GB** out of a 624.88 GB store. The size alone means nothing; it needs a yardstick. The delivery operations store grows 856 MB a day (lesson 03), so 74.54 GB equals **87.1 days** of that store's growth — adding one node means moving roughly three months of growth over the network all over again. That is why rebalancing is a planned job done without downtime, claiming its own share of bandwidth — not an instant configuration change but a **data-moving job**. With carrier as the key, the ratio is similar (0.1182), but its meaning changes: when one of 12 keys moves, all of that key's data moves at once, and during the move two nodes together carry a large share of the load. That is the second cost of a key with few distinct values. #### Summary - Sharding distributes the same context's records across separate engines; table partitioning stays inside the same engine, and the two are separate decisions. - A hot spot is created not by skew but by skew concentrated in a small number of key values: a hot spot of 3.748 (292.78 GB, 65.07 ops/s) for the 12-valued carrier, 1.148 for the 4,000-valued seller, 1.116 for the uniform tracking number. - A pattern that cannot filter on the shard key goes to every node: online touch rate is 138.89/s (baseline) with tracking number as the key, 430.58 (3.10×) with carrier, 1,111.12 (8.00×) with seller. - The key is the field the most frequent, tightest-threshold pattern filters on; a batch scan with a four-hour window can absorb the scatter-gather cost, an online path running at 138.89 requests a second cannot. - At eight shards, per-node load is 78.11 GB and 17.36 ops/s; at VD6's carrier share of 0.50, the busiest node would climb to 368.15 GB. - Going from eight shards to nine moves 0.1193 of keys, equal to 74.54 GB — 87.1 days of the delivery operations store's growth. #### Next Step The sharding decision is made and the key is chosen: tracking number. But one question was left open, waved off with a single answer in this lesson — by what rule the key gets distributed across shards. **Consistent hashing** was used here because it had been measured in the Traffic Layer course, but it is not the only option. A key can be split by range, distributed by its hash, or mapped from key to shard through a separate lookup table. All three place the same data on the same nodes in different shapes, and all three affect rebalancing differently: under one, a range scan stays on a single node but new records keep landing on the same node; under another, the distribution evens out but the range scan spreads out too; under the third, the mapping can be changed freely but every lookup passes through a layer of indirection. The next lesson compares these three strategies. ### Partitioning Strategies Source: https://academia.sh/en/courses/scaling-the-data/partitioning-strategies Distributing the same key to partitions by range, hash, and lookup-table rules: measuring the hot spot ratio across three access patterns, the partition touched and the record count scanned by a periodic scan, and the store touches per access, plus the fraction of keys that move during rebalancing and the floor the largest unsplit group sets on the balance. The sharding decision was made, and the key choice was shown to determine the hot spot — but the decision was left half finished. Once a key is chosen, **which rule** distributes it to partitions is a separate question: the same tracking number can be split into ranges, hashed, or have its group's location kept in a separate table. The three rules produce three placements from the same key and affect rebalancing in three different ways. **Partitioning strategy** is the rule that converts a partition key into a partition number. The Table Partitioning lesson in the Relational Database Administration course built range, list, and hash mechanics inside a single engine; the Sharding Patterns lesson distributed those same partitions across separate stores. Neither is retold here — the question is selection: across the same three access patterns, which rule moves which number by how much. The terminology split holds in one sentence: partitioning splits a table into pieces inside a single store, sharding puts those pieces into separate stores; the rule is the same in both, only where the result appears differs. #### The Three Rules **Range partitioning** assigns the partition to the value range the key falls into. Because tracking numbers are issued in increasing creation order, the range axis is a time axis: consecutive numbers fall into the same partition. **Hash partitioning** derives the partition number from the key's hash. Neighboring keys land far apart from each other; placement cannot be chosen. The consistent hash the previous lesson used belongs to this family and behaves the same way. In a **lookup table**, the mapping from key group to partition number sits in a separate table. The rule is data, not arithmetic: placement can be chosen, changed for one group, or split in two when a group needs it. Here the group is the seller, and the tracking number carries the seller's prefix, so the single-record read and the periodic scan share the same mapping. The cost comes from the same place — every access reads the mapping first. Measurement needs three new assumptions, **not added** to the assumption table in the Introduction to System Design course; they belong to this lesson alone. | Code | Assumption | Value | Rationale | |---|---|---|---| | D1 | exponent of the seller volume distribution | 0.9 | the share of the 4,000 sellers (K01: daily invoice lines) decays as $i^{-0.9}$; a few large sellers, a long tail | | D2 | partition count | 8 | starting capacity; sensitivity measured with 16 partitions | | D3 | active window | 5 days | the shipment age that reads and writes fall into: a delivered shipment is neither queried nor updated | Three access patterns come from K01: a single-record read by tracking number, the carrier's state event write, and the seller's thirty-day period report. The first two fall into the same shipment set because of D3, so they share the same distribution. ```js // partition/strategy.mjs — the same key distributed to partitions by three separate rules. // MODEL: partitions are counters, not a real dataset; results are deterministic. export const DAILY = 400_000, RETENTION = 730, PERIOD = 30; // V3, V12, V11 export const SELLER = 4000, EXPONENT = 0.9, WINDOW = 5; // D1, D3 export const STORE_REQUEST = 138.89, STORED_GB = 712.48; // K01 computed values export function shares(exponent = EXPONENT, factor = 1) { const p = []; let H = 0; for (let i = 1; i <= SELLER; i++) { const a = Math.pow(i, -exponent) * (i === 1 ? factor : 1); p.push(a); H += a; } return p.map((a) => a / H); } export const hash = (n) => { let z = Math.imul(n ^ (n >>> 16), 0x21f0aaad) >>> 0; z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0; return (z ^ (z >>> 15)) >>> 0; }; export const ratio = (k) => Math.max(...k) / (k.reduce((a, b) => a + b, 0) / k.length); export function initialAssign(group, P) { // in decreasing order of volume, to the least-loaded partition const load = new Array(P).fill(0), placement = new Map(); for (const x of [...group].sort((a, b) => b.weight - a.weight)) { let least = 0; for (let p = 1; p < P; p++) if (load[p] < load[least]) least = p; placement.set(x.name, least); load[least] += x.weight; } return placement; } export const loadCalc = (group, placement, P) => { const y = new Array(P).fill(0); for (const x of group) y[placement.get(x.name)] += x.weight; return y; }; export function requestRange(P) { // partition = the number range the tracking number falls into const sliceSize = (RETENTION * DAILY) / P, k = new Array(P).fill(0); for (let g = RETENTION - WINDOW; g < RETENTION; g++) for (let s = 0; s < DAILY; s += 1000) k[Math.floor((g * DAILY + s) / sliceSize)] += 1000; return k; } export function requestHash(P) { // partition = the hash of the tracking number const k = new Array(P).fill(0); for (let g = RETENTION - WINDOW; g < RETENTION; g++) for (let s = 0; s < DAILY; s++) k[hash(g * DAILY + s) % P] += 1; return k; } export function table(P) { const group = shares().map((a, i) => ({ name: `${i}`, weight: a })); const D = loadCalc(group, initialAssign(group, P), P); const busiest = Math.max(...D) / D.reduce((a, b) => a + b, 0); return [ ["range", ratio(requestRange(P)), 1, PERIOD * DAILY, 1], ["hash", ratio(requestHash(P)), P, PERIOD * DAILY, 1], ["lookup table", ratio(D), 1, Math.round(PERIOD * DAILY * busiest), 2], ]; } if (import.meta.filename === process.argv[1]) { const pay = shares(); console.log(`D1: ${SELLER} sellers, largest share ${(pay[0] * 100).toFixed(2)}%, ` + `first eight ${(pay.slice(0, 8).reduce((a, b) => a + b, 0) * 100).toFixed(2)}%, ` + `smallest ${(DAILY * pay[SELLER - 1]).toFixed(1)} shipments/day`); for (const P of [8, 16]) { console.log(`\nP = ${P} partitions`); console.log("strategy request hot spot period report partition scanned records store touches"); for (const [name, i, b, t, d] of table(P)) console.log(name.padEnd(17) + i.toFixed(2).padStart(17) + String(b).padStart(20) + t.toLocaleString("en-US").padStart(15) + String(d).padStart(15)); } console.log(`\nK01 tie-back: requests reaching the store ${STORE_REQUEST} req/s, stored ${STORED_GB} GB, P = 8`); for (const [name, i, , , d] of table(8)) console.log(` ${name.padEnd(15)} busiest partition ${(STORE_REQUEST * i / 8).toFixed(2).padStart(6)} req/s` + ` and ${(STORED_GB / 8).toFixed(2)} GB | total reaching the store ${(STORE_REQUEST * d).toFixed(2)} req/s`); } ``` ``` D1: 4000 sellers, largest share 7.41%, first eight 21.92%, smallest 17.0 shipments/day P = 8 partitions strategy request hot spot period report partition scanned records store touches range 8.00 1 12,000,000 1 hash 1.00 8 12,000,000 1 lookup table 1.00 1 1,500,137 2 P = 16 partitions strategy request hot spot period report partition scanned records store touches range 16.00 1 12,000,000 1 hash 1.00 16 12,000,000 1 lookup table 1.19 1 889,566 2 K01 tie-back: requests reaching the store 138.89 req/s, stored 712.48 GB, P = 8 range busiest partition 138.89 req/s and 89.06 GB | total reaching the store 138.89 req/s hash busiest partition 17.42 req/s and 89.06 GB | total reaching the store 138.89 req/s lookup table busiest partition 17.36 req/s and 89.06 GB | total reaching the store 277.78 req/s ``` These numbers belong to the computed-value class: they follow arithmetically from D1–D3 and K01's assumptions, and none depends on the environment. #### No Strategy Wins Outright The hot spot column divides the busiest partition's load by the per-partition average: 1.00 is perfect balance, 8.00 means only one of eight partitions is doing any work. **Range partitioning splits nothing in a partitioned store.** The ratio is 8.00 — D3 is why: the shipments read and written are from the last five days, and since consecutive tracking numbers fall into the same range, they all land in the last partition. The other seven partitions carry 725 days of data and see almost no requests. The storage column confirms this: the stored 712.48 GB splits evenly, 89.06 GB per partition. **Range partitioning balances storage, not requests** — and the row K01 wants moved is the request row. **Hash partitioning balances requests, and spreads the query.** The ratio is 1.00: the hash of twelve million tracking numbers falls almost evenly across eight partitions. The cost shows up in the period report column: a seller's records spread across eight partitions, so the thirty-day report has to touch all eight. Total scanned records stay the same (12,000,000), but the query now falls into the scatter-gather behavior measured in the Sharding Patterns lesson. **The lookup table delivers both, at the cost of one extra touch.** Placing sellers in decreasing order of volume into the least-loaded partition brings the ratio to 1.00, and since a seller's records all stay in one partition, the period report touches one partition. Scanned records come out to 1,500,137 instead of 12,000,000: the window is still thirty days, but that partition only holds the sellers assigned to it. The eightfold reduction comes from there. The cost sits in the last column: every access reads the mapping, then goes to the data — two store touches per access. K01's `requests reaching the store` row climbs from 138.89 to 277.78. Read the two together: the busiest partition's load drops from 138.89 to 17.36 while total requests reaching the store doubled, because the lookup table itself sees the entirety of that request — a ninth hot spot built to balance the other eight. Whether this table stays in memory is a caching decision, the next topic's question. The P = 16 rows give the sensitivity. Range's ratio climbs to 16.00: adding partitions grows the hot spot. The lookup table's scanned records drop to 889,566 while its ratio climbs to 1.19, since the largest seller's 7.41 percent share exceeds a single partition's 6.25 percent share. #### Rebalancing Placement is not set up once and left alone. When a seller grows, the three strategies each do something different. ```js // partition/rebalance.mjs — the behavior of the three strategies when the largest seller triples. // MODEL: the rebalancer starts from the current placement and moves a single group at a time // from the busiest partition to the least busy; moved weight is summed. Results are deterministic. import { shares, initialAssign, loadCalc, ratio, requestRange, requestHash, SELLER, PERIOD } from "./strategy.mjs"; const P = 8, FACTOR = 3, TARGET = 1.20; function moveLeast(group, placement0, target) { const placement = new Map(placement0), y = loadCalc(group, placement, P); let moved = 0, steps = 0; while (ratio(y) > target) { let h = 0, c = 0; for (let p = 1; p < P; p++) { if (y[p] > y[h]) h = p; if (y[p] < y[c]) c = p; } const candidate = group.filter((x) => placement.get(x.name) === h && y[c] + x.weight < y[h]) .sort((a, b) => b.weight - a.weight)[0]; if (candidate === undefined) break; placement.set(candidate.name, c); y[h] -= candidate.weight; y[c] += candidate.weight; moved += candidate.weight; steps += 1; } return { placement, ratio: ratio(y), moved, steps }; } const format = (r) => `ratio ${r.ratio.toFixed(3)} | moved keys ${(r.moved * 100).toFixed(2)}%` + ` | ${r.steps} moves`; const before = shares(), after = shares(0.9, FACTOR); const gAfter = after.map((a, i) => ({ name: `${i}`, weight: a })); const placement0 = initialAssign(before.map((a, i) => ({ name: `${i}`, weight: a })), P); console.log(`largest seller share ${(before[0] * 100).toFixed(2)}% -> ${(after[0] * 100).toFixed(2)}%`); const [rR, rH] = [ratio(requestRange(P)), ratio(requestHash(P))]; // both independent of the seller console.log(`range: ratio ${rR.toFixed(2)} -> ${rR.toFixed(2)} | the move does not remove the write hot spot`); console.log(`hash: ratio ${rH.toFixed(2)} -> ${rH.toFixed(2)} | the partition comes from the tracking number`); console.log(`lookup table, untouched: ratio ${ratio(loadCalc(gAfter, placement0, P)).toFixed(3)}`); console.log(`lookup table, group = seller: ${format(moveLeast(gAfter, placement0, TARGET))}`); const gSplit = []; // only the hot seller is split into day groups for (let g = 0; g < PERIOD; g++) gSplit.push({ name: `0/${g}`, weight: after[0] / PERIOD }); for (let i = 1; i < SELLER; i++) gSplit.push({ name: `${i}`, weight: after[i] }); const placementSplit = new Map(placement0); for (let g = 0; g < PERIOD; g++) placementSplit.set(`0/${g}`, placement0.get("0")); const r = moveLeast(gSplit, placementSplit, TARGET); const spread = new Set(); for (let g = 0; g < PERIOD; g++) spread.add(r.placement.get(`0/${g}`)); console.log(`lookup table, group = seller-day: ${format(r)}`); console.log(` cost: the hot seller's period report now touches ${spread.size} partitions (others: 1),` + ` lookup rows ${SELLER} -> ${gSplit.length}`); console.log("\nD1 sensitivity: as the distribution exponent grows, the largest unsplit group sets a floor"); for (const exponent of [0.9, 1.1, 1.3]) { const p = shares(exponent); console.log(` exponent ${exponent.toFixed(1)}: largest seller share ${(p[0] * 100).toFixed(2)}%,` + ` best reachable ratio ${Math.max(1, p[0] * P).toFixed(2)}`); } ``` ``` largest seller share 7.41% -> 19.37% range: ratio 8.00 -> 8.00 | the move does not remove the write hot spot hash: ratio 1.00 -> 1.00 | the partition comes from the tracking number lookup table, untouched: ratio 1.904 lookup table, group = seller: ratio 1.549 | moved keys 4.43% | 468 moves lookup table, group = seller-day: ratio 1.181 | moved keys 9.04% | 14 moves cost: the hot seller's period report now touches 8 partitions (others: 1), lookup rows 4000 -> 4029 D1 sensitivity: as the distribution exponent grows, the largest unsplit group sets a floor exponent 0.9: largest seller share 7.41%, best reachable ratio 1.00 exponent 1.1: largest seller share 16.07%, best reachable ratio 1.29 exponent 1.3: largest seller share 27.36%, best reachable ratio 2.19 ``` In the hash row, the ratio stays at 1.00 even as the seller triples, because the partition comes from the tracking number, not the seller. Hash partitioning produces no seller-driven hot spots — but it can also never bring the period report down to a single partition. In the range row, moving anything is pointless: the hot spot comes from the time axis, not a specific seller. The lookup table's three rows are the real measurement. Left untouched, the ratio is 1.904. Moving the largest seller whole only brings it to 1.549, after 468 moves and 4.43 percent of the keys, then it stops: that partition now holds one seller worth 19.37 percent, and unsplit it cannot fall below 1.549 times the eight partitions' average. **The largest unsplit group is the floor on the balance**; the sensitivity rows repeat this: at exponent 1.3, the largest seller alone holds 27.36 percent, and the best reachable ratio is 2.19. The only way to lower the floor is to redefine the group. Splitting the hot seller into seller-day groups brings the ratio to 1.181 in 14 moves — 9.04 percent of keys moved, twice the first attempt, but reaching a ratio the first could not. The cost: the hot seller's period report now touches eight partitions, so the lookup table's entire gain there was given back, while the other 3,999 sellers still touch a single partition. The decision: **the lookup table builds balance selectively; the cost falls only on the split group's query.** The Sharding Patterns lesson measured the moved fraction when the shard count changes; here the question differs — fixing the placement while the partition count stays fixed. #### Summary - Partitioning strategy converts a key into a partition; it is not key selection, and the same key produces three different placements under three rules. - Range partitioning distributed 712.48 GB evenly across eight partitions but never split requests: the hot spot ratio is 8.00, and the busiest partition sees all of K01's 138.89 req/s. - Hash partitioning brought the ratio to 1.00 (17.42 req/s per partition) but spread the seller's period report across all eight partitions. - The lookup table brought the ratio to 1.00, kept the report in a single partition, and dropped scanned records from 12,000,000 to 1,500,137; in exchange it required two store touches per access and pushed `requests reaching the store` from 138.89 to 277.78. - The largest unsplit group sets a floor on the balance: when the largest seller tripled, the best seller-level ratio stayed at 1.549; splitting the group into seller-days moved 9.04 percent of the keys, brought the ratio to 1.181, and spread that seller's report across eight partitions. #### Next Step The three strategies share a common silence: all of them settle **where** the data goes, none asks **what shape** it stands in. The 1,500,137 records the lookup table scans is the count read to produce the seller's report, and part of that comes from the schema itself: completing a report means pulling tariff, zone, and contract records alongside the shipment record, so the read path spans multiple tables even within a single partition. The next lesson shortens that path: what gets copied alongside the record being read, how many tables and records the copy drops from the read, how many steps that stretches the write path by, and how much the stored 712.48 GB and the daily 976 MB growth grow because of it. ### Denormalization Source: https://academia.sh/en/courses/scaling-the-data/denormalization Shortening the read path as a scaling decision: how the narrow and wide copy drive secondary-table records toward zero, why the number of tables touched is a misleading measure, measuring the bytes the copy adds to a shipment record in a real engine, and how far the scope the copy has to track the source over pushes the store's write/read ratio past 2.33. Partitioning strategy settled where data goes, not what shape it stands in. The 1,500,137 records the lookup table scanned was the count read to produce an answer, and not all of it came from the shipment table. The seller's invoice line multiplies weight by tariff, tariff by zone factor, and the result by contract discount; the three factors sit in three separate tables. The read path spans more than one table even within a single partition, and crosses the partition boundary too when partitions are spread across separate stores. **Denormalization** is deliberately reintroducing redundancy into a normalized schema for read performance. The definition, the normal forms, and the consistency rule for a derived column were established in the Denormalization lesson of the Data Modeling and Relational Theory course, which showed that every write path touching the source must fill the copy and that an audit job recomputing from the source is required. None of that is retold here. This lesson's question is a scaling question: **how many records** the copy drops from the read, **how many bytes** it adds to the shipment record, and **how far** it pushes K01's write/read ratio once it must track the source. #### Narrow Copy and Wide Copy The copy's width is a separate decision. The **narrow copy** carries only the numbers the calculation needs: tariff rate, zone factor, contract discount. The **wide copy** also carries the names the report displays: tariff name and zone name. Both target the same read, but what they store and must track differ. The measurement runs on a real engine, at lesson 01's model scale — 20 sellers, 30 days, 100 shipments a day — so the numbers compare with that lesson's. Three things are measured: the query's **records pulled from secondary tables**, the shipment table's **bytes occupied**, and the **rows rewritten** when a source field changes. ```js // schema/copy.mjs — the records the same invoice query pulls, the bytes it occupies, and // the rows rewritten when a source changes, across three schemas. A real engine runs with // node:sqlite; SQL is not taught, what is measured is record count and bytes. import { DatabaseSync } from "node:sqlite"; const SELLER = 20, DAY = 30, DAILY = 100, TARIFF = 40, ZONE = 12; // lesson 01's model scale const tariffName = (i) => `TARIFF-${String(i).padStart(3, "0")} domestic express`; const zoneName = (i) => `ZONE-${String(i).padStart(2, "0")} west anatolia`; const EXTRA = { normal: "", narrow: ", rate REAL, factor REAL, discount REAL", wide: ", rate REAL, factor REAL, discount REAL, tariff_name TEXT, zone_name TEXT" }; function setup(schema) { const db = new DatabaseSync(":memory:"); db.exec(`CREATE TABLE tariff(id INTEGER PRIMARY KEY, name TEXT, rate REAL); CREATE TABLE zone(id INTEGER PRIMARY KEY, name TEXT, factor REAL); CREATE TABLE contract(seller INTEGER PRIMARY KEY, discount REAL); CREATE TABLE shipment(id TEXT PRIMARY KEY, seller INTEGER, day INTEGER, tariff_id INTEGER, zone_id INTEGER, weight REAL${EXTRA[schema]}); CREATE INDEX shipment_seller ON shipment(seller, day);`); db.exec("BEGIN"); for (let i = 0; i < TARIFF; i += 1) db.prepare("INSERT INTO tariff VALUES(?,?,?)").run(i, tariffName(i), 12 + (i % 7) * 0.5); for (let i = 0; i < ZONE; i += 1) db.prepare("INSERT INTO zone VALUES(?,?,?)").run(i, zoneName(i), 1 + (i % 5) * 0.1); for (let i = 0; i < SELLER; i += 1) db.prepare("INSERT INTO contract VALUES(?,?)").run(i, (i % 9) * 0.01); const columns = { normal: 6, narrow: 9, wide: 11 }[schema]; const g = db.prepare(`INSERT INTO shipment VALUES(${"?,".repeat(columns - 1)}?)`); for (let s = 0; s < SELLER; s += 1) for (let d = 0; d < DAY; d += 1) for (let k = 0; k < DAILY; k += 1) { const ti = (s * 7 + d + k) % TARIFF, bi = (s + k) % ZONE; const t = [`TR-${s}-${d}-${k}`, s, d, ti, bi, 1 + (k % 30)]; const n = [12 + (ti % 7) * 0.5, 1 + (bi % 5) * 0.1, (s % 9) * 0.01]; g.run(...(schema === "normal" ? t : schema === "narrow" ? [...t, ...n] : [...t, ...n, tariffName(ti), zoneName(bi)])); } db.exec("COMMIT"); return db; } const QUERY = { // a seller's 30-day invoice line, broken down by tariff normal: `SELECT t.name, sum(g.weight * touch(t.rate) * touch(b.factor) * (1 - touch(s.discount))) FROM shipment g JOIN tariff t ON t.id = g.tariff_id JOIN zone b ON b.id = g.zone_id JOIN contract s ON s.seller = g.seller WHERE g.seller = ? AND g.day BETWEEN 0 AND 29 GROUP BY t.name`, narrow: `SELECT touch(t.name), x.amount FROM (SELECT tariff_id, sum(weight * rate * factor * (1 - discount)) AS amount FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29 GROUP BY tariff_id) x JOIN tariff t ON t.id = x.tariff_id`, wide: `SELECT tariff_name, sum(weight * rate * factor * (1 - discount)) FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29 GROUP BY tariff_name`, }; console.log(`model: ${SELLER * DAY * DAILY} shipments, ${SELLER} sellers, ${DAY} days, ` + `${TARIFF} tariffs, ${ZONE} zones; invoice line is broken down by tariff`); const measure = {}; let base = 0; for (const schema of ["normal", "narrow", "wide"]) { const db = setup(schema); let touches = 0; db.function("touch", (x) => { touches += 1; return x; }); db.prepare(QUERY[schema]).all(10); const plan = db.prepare(`EXPLAIN QUERY PLAN ${QUERY[schema]}`).all() .flatMap((r) => r.detail.match(/(?:SCAN|SEARCH) \w+/g) ?? []); const bytes = db.prepare("SELECT sum(pgsize) AS b FROM dbstat WHERE name LIKE 'shipment%'").get().b; if (base === 0) base = bytes; const changes = (sql, ...a) => db.prepare(sql).run(...a).changes; const display = changes("UPDATE tariff SET name = 'TARIFF-003 new name' WHERE id = 3") + (schema === "wide" ? changes("UPDATE shipment SET tariff_name = 'TARIFF-003 new name' WHERE tariff_id = 3") : 0); const computed = changes("UPDATE contract SET discount = 0.05 WHERE seller = 10") + (schema === "normal" ? 0 : changes("UPDATE shipment SET discount = 0.05 WHERE seller = 10")); measure[schema] = { tables: new Set(plan).size, secondary: touches, bytes, extra: (bytes - base) / (SELLER * DAY * DAILY), display, computed }; } const report = (title, fields) => { console.log(`\n${title.padEnd(26)}${["normal", "narrow", "wide"].map((s) => s.padStart(12)).join("")}`); for (const [name, f, b] of fields) console.log(name.padEnd(26) + ["normal", "narrow", "wide"] .map((s) => (b ? b(measure[s][f]) : String(measure[s][f])).padStart(12)).join("")); }; report("read path", [["tables touched", "tables"], ["secondary-table records", "secondary"], ["shipment table bytes", "bytes", (x) => x.toLocaleString("en-US")], ["extra bytes per shipment", "extra", (x) => x.toFixed(2)]]); report("write path (rows)", [["tariff name change", "display"], ["contract discount change", "computed"]]); console.log(`\nrecords read from the main table are ${DAY * DAILY} in all three schemas`); ``` ``` model: 60000 shipments, 20 sellers, 30 days, 40 tariffs, 12 zones; invoice line is broken down by tariff read path normal narrow wide tables touched 4 3 1 secondary-table records 9000 40 0 shipment table bytes 2,437,120 3,641,344 6,709,248 extra bytes per shipment 0.00 20.07 71.20 write path (rows) normal narrow wide tariff name change 1 1 1498 contract discount change 1 3001 3001 records read from the main table are 3000 in all three schemas ``` These numbers belong to the measurement class: they come from an engine running on this machine, they are deterministic, but page fill factor and plan selection are the engine's own decisions. #### Tables Touched Is the Wrong Measure Read alone, the first row makes the narrow copy look useless: tables touched drops from four to three, and three is still more than one. The second row corrects that. In the normalized schema, the engine pulls **9000 records** from secondary tables — for each of three thousand shipments, the tariff, zone, and contract rows are looked up separately. In the narrow copy, the same number is **40**: the calculation finishes from a single table, and tariff names are looked up **after** summing, once per breakdown. So the narrow copy lowered tables touched by one, and records pulled by 225 times. The result is a reading rule: **a read's cost is measured not by the number of tables it touches, but by the number of records it pulls from secondary tables.** Looking once per breakdown at a small, rarely changing table is not the same thing as looking once per row. This distinction changes character in a partitioned layer. The Partitioning Strategies lesson spread partitions across separate stores; in the normalized schema, each of the 9000 lookups is a cross-partition access unless the tariff and zone tables were copied to every partition. The narrow copy drives those crossings to zero, and the remaining 40 are served from a small table copied once per partition. The wide copy drops records pulled from 40 to 0. The gain is 40 records; its cost sits two rows below. #### Where the Write Path Stretches The table below separates two kinds of source change, and they are not the same thing. **A tariff name change** is a display field. In the normalized and narrow schemas, a single row is updated. In the wide copy, 1498 rows are updated, because the name was copied into the relevant breakdown of sixty thousand shipments. This is the cost of the wide copy's 40-record read gain. **A contract discount change** is a computed field, and in both copy schemas, 3001 rows are updated. The difference comes not from the schema but from the **contract**: whether the discount applies retroactively. In pricing, a past invoice is frozen — an issued invoice does not change afterward — so the copy can be defined as a **snapshot**, and under that definition propagation drops to zero. If the same copy must track the source instead, propagation runs for as long as the record is retained. Which one a copy is cannot be read off the schema; it is a written decision. #### Back to the Numbers The measured extra bytes and propagation figures are converted into K01's rows. Propagation's scope needs an assumption, and that assumption is **not added** to K01's table. **D5 — daily source field changes: 1 tariff, 5 contract.** Rationale: tariff and contract fields are tied to commercial agreements and change rarely within a day. Sensitivity is given at ten times. ```js // schema/cost.mjs — the effect of the measured extra bytes and write propagation on K01's rows const V3 = 400_000, V4 = 7, V6 = 220, V7 = 900, V12 = 730; // K01 assumptions const GROWTH = 976, STORED = 712.48, RATIO = 2.33; // K01 computed values const DAY_SECONDS = 86_400; const READ = (V3 * 30 * 0.1) / DAY_SECONDS; // K01: reads behind cache, average const WRITE = (V3 * V4) / DAY_SECONDS; // K01: state event write, average const EXTRA = { normal: 0, narrow: 20.07, wide: 71.20 }; // schema/copy.mjs measurement const TARIFF_SHARE = 1497 / 60_000; // schema/copy.mjs: rate bound to one tariff const SELLER_DAILY = V3 / 4000; // K01: daily invoice lines 4000 const D5 = { tariff: 1, contract: 5 }; // this lesson's assumption const growth = (extra) => (V3 * (V7 + extra) + V3 * V4 * V6) / 1e6; console.log(`average store reads ${READ.toFixed(2)}/s, writes ${WRITE.toFixed(2)}/s, ` + `ratio ${(WRITE / READ).toFixed(2)} (K01: ${RATIO})`); console.log(`\n${"schema".padEnd(8)}${"record bytes".padStart(14)}${"daily MB".padStart(12)}` + `${"x".padStart(7)}${"stored GB".padStart(13)}${"x".padStart(7)}`); for (const [name, extra] of Object.entries(EXTRA)) { const a = growth(extra), s = (a * V12) / 1000; console.log(name.padEnd(8) + (V7 + extra).toFixed(2).padStart(14) + a.toFixed(2).padStart(12) + (a / GROWTH).toFixed(3).padStart(7) + s.toFixed(2).padStart(13) + (s / STORED).toFixed(3).padStart(7)); } console.log(`\npropagation of a source change in the wide copy (D5: ${D5.tariff} tariff field, ` + `${D5.contract} contract fields per day)`); console.log(`${"propagation scope".padEnd(20)}${"tariff records".padStart(16)}${"contract records".padStart(18)}` + `${"rewrite/s".padStart(17)}${"write/read".padStart(13)}`); for (const [name, g] of [["history, 730 days", V12], ["period, 30 days", 30], ["open shipment, 5 days", 5]]) { const t = g * V3 * TARIFF_SHARE, s = g * SELLER_DAILY; const rewrite = (D5.tariff * t + D5.contract * s) / DAY_SECONDS; console.log(name.padEnd(20) + Math.round(t).toLocaleString("en-US").padStart(16) + Math.round(s).toLocaleString("en-US").padStart(18) + rewrite.toFixed(2).padStart(17) + ((WRITE + rewrite) / READ).toFixed(2).padStart(13)); } const scope = 30, t = scope * V3 * TARIFF_SHARE, s = scope * SELLER_DAILY; for (const factor of [1, 10]) { const y = factor * (D5.tariff * t + D5.contract * s) / DAY_SECONDS; console.log(`D5 sensitivity (30 days, ${factor}x): rewrite ${y.toFixed(2)}/s, ` + `write/read ${((WRITE + y) / READ).toFixed(2)}`); } ``` ``` average store reads 13.89/s, writes 32.41/s, ratio 2.33 (K01: 2.33) schema record bytes daily MB x stored GB x normal 900.00 976.00 1.000 712.48 1.000 narrow 920.07 984.03 1.008 718.34 1.008 wide 971.20 1004.48 1.029 733.27 1.029 propagation of a source change in the wide copy (D5: 1 tariff field, 5 contract fields per day) propagation scope tariff records contract records rewrite/s write/read history, 730 days 7,285,400 73,000 88.55 8.71 period, 30 days 299,400 3,000 3.64 2.60 open shipment, 5 days 49,900 500 0.61 2.38 D5 sensitivity (30 days, 1x): rewrite 3.64/s, write/read 2.60 D5 sensitivity (30 days, 10x): rewrite 36.39/s, write/read 4.95 ``` The first line is a check: the write/read ratio from the average rates is 2.33, the same ratio K01 derives from peak rates. Because both rates carry the same peak factor, the ratio does not change, and this shows the following rows are comparable with K01. **The storage cost is small.** The narrow copy raises the shipment record from 900 bytes to 920.07 bytes: daily growth from 976 MB to 984.03 MB, stored data from 712.48 GB to 718.34 GB. Even in the wide copy, the ratio is 1.029. Denormalization's expense is not disk. **The write cost depends on scope, and the range is wide.** If the copy propagates back through history, a tariff change rewrites 7,285,400 records; under D5 the daily rewrite rate is 88.55 records/s, and `write/read ratio at store` climbs from 2.33 to **8.71**. Tracking only period records gives 2.60; only open shipments, 2.38. **The copy's width determines the read gain, its scope determines the write cost, and the two are separate decisions.** The sensitivity rows say scope matters more than frequency: when D5 goes to ten times, the ratio in the thirty-day scope moves from 2.60 to 4.95, but the scope that propagates through history is already at 8.71 at a single multiple. Misjudging frequency is cheaper than choosing the wrong scope. #### Summary - Denormalization is a scaling decision here: its measure is the records the copy drops from the read, the bytes it adds to the record, and the scope over which it tracks the source. - A read's cost is not measured by tables touched: the narrow copy dropped that from four to three while dropping records pulled from secondary tables from 9000 to 40. - In a partitioned layer, those 9000 lookups are cross-partition accesses; the narrow copy zeroes them out, and the remaining 40 lookups are served from a small table copied per partition. - The storage cost is small: the shipment record climbs from 900 bytes to 920.07 (narrow) and 971.20 bytes (wide), stored data from 712.48 GB to 718.34 and 733.27 GB. - The real cost is in propagation's scope: `write/read ratio at store` climbs from 2.33 to 8.71 propagating through history, to 2.60 in the period scope, to 2.38 in the open-shipment scope. - The wide copy writes 1498 rows when a display field changes; in a copy defined as a snapshot, the same change stays a single row. #### Next Step The copy dropped the number of records the read pulls, but never touched the number the read **scans**: the seller's invoice line still reads and sums three thousand shipments one by one, which at K01's scale is 833.33 records/s within a four-hour window. The sum is redone on every request, even though the same period's sum does not change during the day. The next lesson takes on this repetition: precomputing and storing the sum, how far the scanned-record count drops, what refreshing the stored result costs, and how the refresh interval sets the accepted staleness window. ### Materialized Views Source: https://academia.sh/en/courses/scaling-the-data/materialized-views Precomputing and storing a repeated sum: how the result's grain determines the record count scanned, incremental refresh's cost against full recompute, why the storage cost stays negligible, and the trade-off the refresh interval sets up between the staleness window and the store's write/read ratio. The copy dropped the number of records the read pulls but never touched the number scanned. The seller's invoice line still reads and sums three thousand shipments one by one; at K01's scale that is 833.33 records/s within a four-hour window. The sum is redone on every request, even though a closed day's sum never changes again. The same arithmetic can be done once a day and stored. A **materialized view** is a query's result, computed and stored. The view built in the Views lesson of the SQL Fundamentals course holds no data — it stores the query and reruns it on every call; a materialized view holds the **result**, so reads are cheap while accuracy depends on time. This lesson does not teach query language; what it measures is the record count a query scans, the cost of refreshing, and how far the stored result can fall behind. #### What the Grain Determines The result to be stored requires a choice: at what level to aggregate. When the **grain** is coarse, row count drops, scanned records shrink, but the result answers fewer questions. Two grains are measured: seller-day-tariff and seller-day. The source is at shipment grain. The model scale is again lesson 01's scale. A seller has 100 shipments a day across eight distinct tariffs; the number eight is a model choice, not a real measurement. ```js // view/refresh.mjs — the same report read from the source and from two materialized-view // grains. A real engine runs with node:sqlite; what is measured is records scanned, // bytes stored, and the rows a refresh scans and writes. import { DatabaseSync } from "node:sqlite"; const SELLER = 20, DAY = 30, DAILY = 100, TARIFF = 40, TARIFF_DAY = 8; // lesson 01's scale const db = new DatabaseSync(":memory:"); db.exec(`CREATE TABLE shipment(id TEXT PRIMARY KEY, seller INTEGER, day INTEGER, tariff_id INTEGER, amount REAL); CREATE INDEX shipment_seller ON shipment(seller, day); CREATE INDEX shipment_day ON shipment(day); CREATE TABLE summary_day_tariff(seller INTEGER, day INTEGER, tariff_id INTEGER, record INTEGER, amount REAL, PRIMARY KEY(seller, day, tariff_id)); CREATE TABLE summary_day(seller INTEGER, day INTEGER, record INTEGER, amount REAL, PRIMARY KEY(seller, day));`); db.exec("BEGIN"); const g = db.prepare("INSERT INTO shipment VALUES(?,?,?,?,?)"); for (let s = 0; s < SELLER; s += 1) for (let d = 0; d < DAY; d += 1) for (let k = 0; k < DAILY; k += 1) g.run(`TR-${s}-${d}-${k}`, s, d, (s * 7 + d * 3 + (k % TARIFF_DAY)) % TARIFF, 12 + (k % 30) * 0.5); db.exec("COMMIT"); let touches = 0; db.function("touch", (x) => { touches += 1; return x; }); const measure = (sql, ...a) => { touches = 0; const r = db.prepare(sql).all(...a); return [touches, r.length]; }; const REFRESH = { // incremental refresh: only the given day's records summary_day_tariff: `INSERT OR REPLACE INTO summary_day_tariff SELECT seller, day, tariff_id, count(*), sum(amount) FROM shipment WHERE day = ? GROUP BY seller, day, tariff_id`, summary_day: `INSERT OR REPLACE INTO summary_day SELECT seller, day, count(*), sum(amount) FROM shipment WHERE day = ? GROUP BY seller, day`, }; db.exec("BEGIN"); for (let d = 0; d < DAY; d += 1) for (const s of Object.values(REFRESH)) db.prepare(s).run(d); db.exec("COMMIT"); const REPORT = { shipment: ["shipment | grain: shipment", `SELECT tariff_id, sum(touch(amount)) FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29 GROUP BY tariff_id`], summary_day_tariff: ["summary | grain: seller-day-tariff", `SELECT tariff_id, sum(touch(amount)) FROM summary_day_tariff WHERE seller = ? AND day BETWEEN 0 AND 29 GROUP BY tariff_id`], summary_day: ["summary | grain: seller-day", `SELECT sum(touch(amount)) FROM summary_day WHERE seller = ? AND day BETWEEN 0 AND 29`], }; const bytesOf = (t) => db.prepare("SELECT sum(pgsize) AS b FROM dbstat WHERE name = ?").get(t).b; const rowCount = (t) => db.prepare(`SELECT count(*) AS n FROM ${t}`).get().n; console.log(`model: ${SELLER * DAY * DAILY} shipments, ${SELLER} sellers, ${DAY} days, ` + `${TARIFF} tariffs, ${TARIFF_DAY} tariffs per seller-day`); console.log(`\n${"table read".padEnd(34)}${"rows".padStart(8)}${"scanned".padStart(9)}` + `${"returned".padStart(9)}${"x".padStart(8)}${"bytes".padStart(10)}${"bytes/row".padStart(12)}`); let base = 0; for (const [table, [label, sql]] of Object.entries(REPORT)) { const [scanned, returned] = measure(sql, 10); if (base === 0) base = scanned; const n = rowCount(table), b = bytesOf(table); console.log(label.padEnd(34) + n.toLocaleString("en-US").padStart(8) + String(scanned).padStart(9) + String(returned).padStart(9) + (base / scanned).toFixed(2).padStart(8) + b.toLocaleString("en-US").padStart(10) + (b / n).toFixed(1).padStart(12)); } console.log(`\n${"refresh".padEnd(18)}${"incremental scanned".padStart(20)}${"incremental written".padStart(20)}` + `${"full scanned".padStart(15)}${"full written".padStart(15)}`); for (const [table, sql] of Object.entries(REFRESH)) { const written = db.prepare(sql).run(DAY - 1).changes; console.log(table.padEnd(18) + String(SELLER * DAILY).padStart(20) + String(written).padStart(20) + String(SELLER * DAY * DAILY).padStart(15) + String(rowCount(table)).padStart(15)); } ``` ``` model: 60000 shipments, 20 sellers, 30 days, 40 tariffs, 8 tariffs per seller-day table read rows scanned returned x bytes bytes/row shipment | grain: shipment 60,000 3000 40 1.00 1,761,280 29.4 summary | grain: seller-day-tariff 4,800 240 40 12.50 94,208 19.6 summary | grain: seller-day 600 30 1 100.00 16,384 27.3 refresh incremental scanned incremental written full scanned full written summary_day_tariff 2000 160 60000 4800 summary_day 2000 20 60000 600 ``` These numbers belong to the measurement class; the bytes column depends on page fill factor and looks large when row count is small because bytes are spread over few rows. The three rows give a gradient. Reading the report from the source scans 3000 records; at the seller-day-tariff grain, 240; at the seller-day grain, 30. As the grain coarsens, scanned records drop by 12.5 and 100 times. The fourth column states the cost. The seller-day-tariff grain preserves the report's breakdown: the same 40 rows return as the source. At the seller-day grain, returned rows drop to 1 — that summary does not **carry** the tariff breakdown, so the question "how much from which tariff" cannot be asked of it. **Grain selection is not a performance tweak; it is a design decision that narrows the set of answerable questions.** The table below separates two forms of refresh. **Incremental refresh** scans only the changed day's records: it reads 2000 records, writes 160 rows for seller-day-tariff, 20 for seller-day. **Full recompute** scans all 60,000 records every time and rewrites the entire summary. The ratio between them is thirty times on both the scanned and written sides, and it grows with retention length, because incremental refresh's scanned count does not depend on the number of days. #### Back to the Numbers The measured grain ratios and bytes per row are converted into K01's rows. The scaling is direct: the seller-day grain's period row count is 4,000 sellers times 30 days, the same 120,000 rows K01 already computed as `period seller-days`. ```js // view/window.mjs — the effect of the measured grain ratios on K01's rows and the trade-off // between the refresh interval's staleness window and write load. All of it is arithmetic. const V3 = 400_000, V10 = 4, V12 = 730, PERIOD = 30; // K01 assumptions const SELLER = 4000, TARIFF_DAY = 8; // K01: daily invoice lines; model measurement const SCAN = 833.33, READ_GB = 10.80, STORED = 712.48; // K01 computed values const DAY_SECONDS = 86_400, WINDOW = V10 * 3600; const READ = (V3 * PERIOD * 0.1) / DAY_SECONDS, WRITE = (V3 * 7) / DAY_SECONDS; // K01 averages const BYTES = { "seller-day-tariff": 19.6, "seller-day": 27.3 }; // view/refresh.mjs measurement const period = { shipment: PERIOD * V3, "seller-day-tariff": SELLER * PERIOD * TARIFF_DAY, "seller-day": SELLER * PERIOD }; console.log(`${"table read".padEnd(20)}${"period rows".padStart(14)}${"scan records/s".padStart(16)}` + `${"x".padStart(8)}${"read MB".padStart(12)}${"storage GB".padStart(12)}${"x".padStart(8)}`); for (const [label, n] of Object.entries(period)) { const bytes = BYTES[label] ?? (READ_GB * 1e9) / period.shipment; const stored = label === "shipment" ? STORED : STORED + (n / PERIOD) * V12 * bytes / 1e9; console.log(label.padEnd(20) + n.toLocaleString("en-US").padStart(14) + (n / WINDOW).toFixed(2).padStart(16) + (period.shipment / n).toFixed(2).padStart(8) + ((n * bytes) / 1e6).toFixed(2).padStart(12) + stored.toFixed(2).padStart(12) + (stored / STORED).toFixed(4).padStart(8)); } console.log(`K01 baseline: batch job scan ${SCAN} records/s, read ${READ_GB} GB, ` + `stored ${STORED} GB`); console.log(`\nrefresh interval against the staleness window and write load (incremental refresh)`); console.log(`${"interval".padEnd(12)}${"staleness".padStart(11)}` + `${"seller-day writes/s".padStart(21)}${"ratio".padStart(7)}` + `${"seller-day-tariff writes/s".padStart(28)}${"ratio".padStart(7)}`); for (const [label, T] of [["once a day", DAY_SECONDS], ["once an hour", 3600], ["5 minutes", 300], ["1 minute", 60]]) { const cell = [SELLER, SELLER * TARIFF_DAY].map((rows) => { const y = rows / T; return [y.toFixed(2), ((WRITE + y) / READ).toFixed(2)]; }); const duration = T >= 3600 ? `${T / 3600} h` : `${T / 60} min`; console.log(label.padEnd(12) + duration.padStart(11) + cell[0][0].padStart(21) + cell[0][1].padStart(7) + cell[1][0].padStart(28) + cell[1][1].padStart(7)); } console.log(`K01 baseline: average write ${WRITE.toFixed(2)}/s, read ${READ.toFixed(2)}/s, ratio ` + `${(WRITE / READ).toFixed(2)}`); for (const T of [DAY_SECONDS, 300]) { const full = (DAY_SECONDS / T) * period.shipment; console.log(` ${T === DAY_SECONDS ? "once a day" : "5 minutes"} refresh, daily scanned records: ` + `incremental ${V3.toLocaleString("en-US")}, full recompute ${full.toLocaleString("en-US")}`); } ``` ``` table read period rows scan records/s x read MB storage GB x shipment 12,000,000 833.33 1.00 10800.00 712.48 1.0000 seller-day-tariff 960,000 66.67 12.50 18.82 712.94 1.0006 seller-day 120,000 8.33 100.00 3.28 712.56 1.0001 K01 baseline: batch job scan 833.33 records/s, read 10.8 GB, stored 712.48 GB refresh interval against the staleness window and write load (incremental refresh) interval staleness seller-day writes/s ratio seller-day-tariff writes/s ratio once a day 24 h 0.05 2.34 0.37 2.36 once an hour 1 h 1.11 2.41 8.89 2.97 5 minutes 5 min 13.33 3.29 106.67 10.01 1 minute 1 min 66.67 7.13 533.33 40.73 K01 baseline: average write 32.41/s, read 13.89/s, ratio 2.33 once a day refresh, daily scanned records: incremental 400,000, full recompute 12,000,000 5 minutes refresh, daily scanned records: incremental 400,000, full recompute 3,456,000,000 ``` The upper table breaks down K01's largest read item. The end-of-day job scans 833.33 records/s and reads 10,800 MB when reading from the source; reading from the seller-day summary, it scans 8.33 records/s and reads 3.28 MB. The bandwidth side reverses K01's most striking finding: there, the end-of-day job demanded 3.75 times the peak read egress in bandwidth; here that item drops to three-thousandths of it. The storage column shows a cost close to nothing. The seller-day summary, kept for seven hundred thirty days, raises stored data from 712.48 GB to 712.56 GB; the seller-day-tariff summary, to 712.94 GB. Both are under one part in a thousand. **The cost of a precomputed result is not on disk.** #### The Staleness Window The table below says where the cost is. The stored result falls behind until it is refreshed, and this lag is called the **staleness window**; under incremental refresh, the window is the refresh interval itself. As the interval shortens, the window shrinks and write load grows. At the seller-day grain, refreshing once a day raises `write/read ratio at store` from 2.33 to 2.34 — a cost too small to measure. Refreshing every five minutes gives a ratio of 3.29; every minute, 7.13. The same curve is steeper at a finer grain: the seller-day-tariff summary's ratio is 10.01 refreshed every five minutes, 40.73 refreshed every minute. **A fine grain gains little on reads and makes refreshing far more expensive.** The decision follows from one of K01's assumptions. V10 sets the end-of-day job's window at four hours: the seller's report is requested in the morning. A twenty-four-hour staleness window meets that requirement, since the report already covers closed days. For pricing, the right interval is once a day, at the cost of the ratio moving from 2.33 to 2.34. If the same summary is wanted **live** on a seller dashboard instead, the window drops to minutes and the cost is the ratio climbing three to seven times; that is a second, separate use of the summary, and one interval cannot serve both. The last two lines mark the limit of the refresh style. Incremental refresh scans 400,000 records a day regardless of interval — each shipment once. Full recompute scans 12,000,000 running once a day, 3,456,000,000 running every five minutes. **Shortening the interval is only an option if the refresh is incremental.** #### Summary - A materialized view stores the result, not the query: a stored query reruns on every call, a stored result lags until refreshed. - Grain determines the answerable questions: the seller-day-tariff summary dropped scanned records from 3000 to 240 while preserving the breakdown; the seller-day summary dropped it to 30 but can no longer answer the tariff question. - Back to K01: the end-of-day job's scan drops from 833.33 records/s and 10,800 MB to 8.33 records/s and 3.28 MB — a hundred times. - The storage cost is negligible: the 730-day summary raises stored data from 712.48 GB to 712.56 GB. - The cost is in the refresh: `write/read ratio at store` climbs from 2.33 to 2.34 refreshing once a day, to 3.29 every five minutes, to 7.13 every minute; the same intervals give 2.36, 10.01, and 40.73 at the finer grain. - Shortening the interval is only an option under incremental refresh: incremental refresh scans 400,000 records a day, while full recompute scans 3,456,000,000 records at a five-minute interval. #### Next Step Every decision in this topic has kept one assumption fixed: the data sits in a table with rows and columns. The three access patterns measured were all served by that same store shape — a single-record read by tracking number, the state event write, and the periodic scan. Yet these three patterns want very different things: one wants a single record by a single key, one wants continuous appends, one wants a sum over a wide range. The next lesson measures the same three patterns across four store types: in a key-value, document, wide-column, and graph store, what record count is touched and what query count is required for each pattern, and which pattern the structure that cheapens one makes more expensive. ### Store Types Source: https://academia.sh/en/courses/scaling-the-data/store-types Measuring the key-value, document, wide-column, and graph store against the same access patterns: the query count and record count each type requires for a pattern, which pattern the structure that cheapens one makes expensive, and the cost the choice charges to the introductory course's store request rate and write/read ratio. Every decision up to this point was made on a single store shape with rows and columns. Yet the measured patterns want very different things: the tracking query wants a single record by a single key, the state event does continuous appends, the period scan sums over a wide range. When a store type writes one job into its structure, it leaves the others to the application, and this lesson counts that trade-off. The families' definitions — what the key-value, document, wide-column, and graph store give up and what they give — were established in the Data Modeling and Relational Theory course; the relational-versus-non-relational choice and the rule that the system of record stays singular were measured in this topic's first lesson. Neither is retold. The question here is narrow: **how many queries each type needs per pattern, and how many records it touches.** #### What Separates the Types A store type is defined by the access shape it can answer natively. Every shape it cannot answer falls to the application as a separate store request. A **key-value store** knows only the primary key. The inside of the value is meaningless to it, so secondary access runs through a key list the application keeps, and each record on that list becomes a separate request. A **document store** sees the value's fields and can build a secondary index on one. A secondary filter finishes in a single request. A **wide-column store** groups rows by partition key and reads a sorted range within the group in one request. Filtering by a different key requires writing the same data again under a second partition key. A **graph store** makes nodes and edges first-class objects; a traversal runs inside the store and appears to the client as one request. For measurement, a fourth pattern joins the first three, since none of the first three lands where the graph store wins. **P4 — route traversal:** when a transfer hub goes down, finding the shipments passing through it, deriving their next stops, and advancing three steps to the shipments passing through those. It is a question the delivery operation asks, concerning not a record itself but the links between records. ```js // type/access.mjs — the queries and records touched that the same access patterns require // across four store types. MODEL: stores are objects, counters are explicit; not a product, // a model of access capabilities. Results are deterministic. const SHIPMENT = 60_000, SELLER = 20, HUB = 8; // lesson 01's scale: 3000 per seller const shipments = Array.from({ length: SHIPMENT }, (_, i) => ({ id: `TR-${i}`, seller: i % SELLER, route: [0, 1, 2].map((j) => (i * 3 + j) % HUB), })); const sellerIndex = new Map(), hubIndex = new Map(), routeNext = new Map(); for (const g of shipments) { if (!sellerIndex.has(g.seller)) sellerIndex.set(g.seller, []); sellerIndex.get(g.seller).push(g); for (let j = 0; j < g.route.length; j += 1) { const m = g.route[j]; if (!hubIndex.has(m)) hubIndex.set(m, []); hubIndex.get(m).push(g); if (j + 1 < g.route.length) { if (!routeNext.has(m)) routeNext.set(m, new Set()); routeNext.get(m).add(g.route[j + 1]); } } } class Store { // every call is one store request; touched records are counted constructor() { this.query = 0; this.record = 0; } request(n) { this.query += 1; this.record += n; return n; } } // P1: single-record read by tracking number (state + last three route steps) // P2: writing a state event from the carrier // P3: a seller's period scan // P4: route traversal starting from a hub and advancing H steps const H = 3; // traversal depth const PATTERN = { "key-value": { // only the primary key; secondary index in the application P1: (d) => d.request(1), // the value comes back whole P2: (d) => { d.request(1); d.request(1); }, // read, then write back the whole value P3: (d) => { const n = sellerIndex.get(3).length; d.request(1); // the index key first for (let i = 0; i < n; i += 1) d.request(1); }, // then one request per record P4: (d, depth) => { let m = new Set([0]); for (let h = 0; h < depth; h += 1) { const s = new Set(); for (const x of m) { for (const g of hubIndex.get(x)) d.request(1); for (const y of routeNext.get(x) ?? []) s.add(y); d.request(1); } m = s; } }, }, "document": { // secondary index on document fields P1: (d) => d.request(1), P2: (d) => d.request(1), // appending to the document's event array P3: (d) => d.request(sellerIndex.get(3).length), P4: (d, depth) => { let m = new Set([0]); for (let h = 0; h < depth; h += 1) { const s = new Set(); let k = 0; for (const x of m) { k += hubIndex.get(x).length; for (const y of routeNext.get(x) ?? []) s.add(y); } d.request(k); m = s; } }, }, "wide-column": { // partition key + sorted range within the partition P1: (d) => d.request(3), // last three event rows P2: (d) => { d.request(1); d.request(1); }, // event table + seller table P3: (d) => d.request(sellerIndex.get(3).length), P4: (d, depth) => { let m = new Set([0]); for (let h = 0; h < depth; h += 1) { const s = new Set(); let k = 0; for (const x of m) { k += hubIndex.get(x).length; for (const y of routeNext.get(x) ?? []) s.add(y); } d.request(k); m = s; } }, }, "graph": { // nodes and edges; the traversal runs inside the store P1: (d) => d.request(1 + 3), // shipment node + last three edges P2: (d) => d.request(2), // event node + edge P3: (d) => d.request(1 + 2 * sellerIndex.get(3).length), // seller node, edge, shipment P4: (d, depth) => { let m = new Set([0]), k = 0; for (let h = 0; h < depth; h += 1) { const s = new Set(); for (const x of m) { k += 1 + hubIndex.get(x).length; for (const y of routeNext.get(x) ?? []) s.add(y); } m = s; } d.request(k); }, }, }; console.log(`model: ${SHIPMENT} shipments, ${SELLER} sellers, ${HUB} hubs, ` + `route 3 steps, traversal ${H} steps; numbers are query/record`); const labels = ["P1 single read", "P2 event write", "P3 period scan", "P4 route traversal"]; console.log(`\n${"store type".padEnd(16)}${labels.map((a) => a.padStart(21)).join("")}`); for (const [type, o] of Object.entries(PATTERN)) { const cell = ["P1", "P2", "P3", "P4"].map((p) => { const d = new Store(); o[p](d, H); return `${d.query}/${d.record}`; }); console.log(type.padEnd(16) + cell.map((h) => h.padStart(21)).join("")); } console.log(`\nP4 traversal depth and query count`); console.log(`${"store type".padEnd(16)}${[1, 2, 3].map((h) => `${h} step(s)`.padStart(10)).join("")}`); for (const [type, o] of Object.entries(PATTERN)) console.log(type.padEnd(16) + [1, 2, 3].map((h) => { const d = new Store(); o.P4(d, h); return String(d.query).padStart(10); }).join("")); ``` ``` model: 60000 shipments, 20 sellers, 8 hubs, route 3 steps, traversal 3 steps; numbers are query/record store type P1 single read P2 event write P3 period scan P4 route traversal key-value 1/1 2/2 3001/3001 67503/67503 document 1/1 1/1 1/3000 3/67500 wide-column 1/3 2/2 1/3000 3/67500 graph 1/4 1/2 1/6001 1/67503 P4 traversal depth and query count store type 1 step(s) 2 step(s) 3 step(s) key-value 22501 45002 67503 document 1 2 3 wide-column 1 2 3 graph 1 1 1 ``` These numbers belong to the computed-value class: they come from the model's counters, are deterministic, and are independent of the machine. The model is not a product; it is an abstraction of access capabilities. #### No Column Wins Start to Finish **P1 does not separate any type.** All four types have a query count of 1. Records touched differ as 1, 1, 3, and 4, but fetching a single record by a single key is a job every type does. Choosing a design by the tracking query alone cannot tell the four options apart. **P2 separates two types.** The document and graph store append an event in one request. The key-value store needs two: it cannot reach into the value, so it reads the whole value and writes it whole. The wide-column store also needs two, but for a different reason — serving P3 means writing the same data a second time under a second partition key. **Two types produce the same number for two separate reasons, and the reason determines how it grows**: the key-value store's second request scales with record size, the wide-column store's with the number of supported queries. **P3 eliminates the key-value store.** The document and wide-column store read 3000 records in a single request; the graph store touches 6001, since from the seller node each shipment is reached through an edge and then a node. The key-value store needs 3001 requests: the secondary index lives in the application, so whatever is on the list is pulled one at a time. Records touched are the same order of magnitude, **request count differs by three thousand times.** **P4 flips the table.** Records touched are of the same order across all four types: between 67,500 and 67,503. The separation is in the query column: document and wide-column spend one request per step, the graph store finishes in a single request regardless of depth, the key-value store needs 67,503. The table below makes this clear: as depth goes from one to three, document and wide-column's request count advances 1, 2, 3, while the graph store's stays at 1. **The graph store's win is not records touched, but a request count independent of traversal depth.** A traversal of unknown depth becomes, in the other types, a job of unknown request count. #### Back to the Numbers Query counts convert directly into K01's rows: each pattern's request rate is multiplied by the query count that pattern requires. ```js // type/cost.mjs — the effect of the measured query and record counts on K01's rows. // All of it is arithmetic; the input is the output of the type/access.mjs run. const READ = 41.67, WRITE = 97.22, STORE_REQUEST = 138.89, RATIO = 2.33; // K01 computed values const SCAN = 833.33, SELLER = 4000, WINDOW = 4 * 3600; // K01 computed values and V10 const MEASURE = { // type/access.mjs: [query, record] "key-value": { P1: [1, 1], P2: [2, 2], P3: [3001, 3001] }, "document": { P1: [1, 1], P2: [1, 1], P3: [1, 3000] }, "wide-column": { P1: [1, 3], P2: [2, 2], P3: [1, 3000] }, "graph": { P1: [1, 4], P2: [1, 2], P3: [1, 6001] }, }; console.log(`${"store type".padEnd(16)}${"store req/s".padStart(16)}${"x".padStart(7)}` + `${"write/read".padStart(13)}${"batch req/s".padStart(15)}${"batch rec/s".padStart(15)}` + `${"x".padStart(7)}`); for (const [type, o] of Object.entries(MEASURE)) { const read = READ * o.P1[0], write = WRITE * o.P2[0]; const request = (o.P3[0] * SELLER) / WINDOW, record = (o.P3[1] * SELLER) / WINDOW; console.log(type.padEnd(16) + (read + write).toFixed(2).padStart(16) + ((read + write) / STORE_REQUEST).toFixed(2).padStart(7) + (write / read).toFixed(2).padStart(13) + request.toFixed(2).padStart(15) + record.toFixed(2).padStart(15) + (record / SCAN).toFixed(2).padStart(7)); } console.log(`K01 baseline: requests reaching the store ${STORE_REQUEST} req/s, write/read ${RATIO}, ` + `batch scan ${SCAN} records/s (batch req/s at a single store ${(SELLER / WINDOW).toFixed(2)})`); ``` ``` store type store req/s x write/read batch req/s batch rec/s x key-value 236.11 1.70 4.67 833.61 833.61 1.00 document 138.89 1.00 2.33 0.28 833.33 1.00 wide-column 236.11 1.70 4.67 0.28 833.33 1.00 graph 138.89 1.00 2.33 0.28 1666.94 2.00 K01 baseline: requests reaching the store 138.89 req/s, write/read 2.33, batch scan 833.33 records/s (batch req/s at a single store 0.28) ``` Two types grow `requests reaching the store` from 138.89 to 236.11, 1.70 times, and push `write/read ratio at store` from 2.33 to 4.67. The reason is the same in both: P2 needs two requests. This topic's first lesson produced the same arithmetic — stretching the write path to two targets gave the same 236.11. **The store type changes, but the number comes from the same place: how many targets the write path touches.** The batch column carries the real warning. In the document and wide-column store, the end-of-day job produces 0.28 requests/s in the four-hour window, exactly repeating K01's computed 833.33 records/s scan. In the key-value store, the same job produces **833.61 requests/s**: the store sees six times the 138.89 req/s built up over the whole day. A single architectural decision moves K01's largest item into a different column — record count did not change, request count did. The graph column is the mirror image: request count matches the best types, but its batch record rate climbs from 833.33 to 1666.94, doubling, since every shipment is reached through an edge. The structure that cheapens traversal makes aggregation more expensive. Read together, the four columns yield one sentence: **no type cheapens all four patterns at once.** The choice is made against the most expensive pattern, and the first lesson's rule holds — the system of record stays singular, and the other types are secondary stores derived from it. #### Summary - A store type is defined by the access shape it answers natively; every shape it cannot answer falls to the application as a separate store request. - The single-record read does not distinguish the four types: all take one request, between 1 and 4 records. - The key-value store needs 3001 requests for the period scan, the document and wide-column store need 1; records touched are the same order, request count differs by three thousand times. - The graph store's gain is in request count, not records touched: in a three-step traversal, document and wide-column spend 3 requests while the graph store stays at 1, unchanged as depth grows. - Back to K01: the key-value and wide-column store push `requests reaching the store` from 138.89 to 236.11 (1.70 times) and `write/read ratio at store` from 2.33 to 4.67. - The end-of-day job produces 833.61 requests/s in the key-value store and pushes the scan rate to 1666.94 records/s in the graph store, from 833.33; no type cheapens all four patterns at once. #### Next Step Eight lessons designed **where** and **in what shape** data stands: type was chosen against the access pattern, data was replicated, federated, and sharded; shards were placed under three separate rules; the read path was shortened by denormalization, and a repeated sum was precomputed. The gain is real: the period scan dropped from 833.33 records/s to 8.33, a read's secondary records from 9000 to 40. The one number that did not gain is the read side of `requests reaching the store`. Even in this lesson's best column, the tracking query still knocks on the store's door at 41.67 requests/s, because none of the eight lessons kept the read from **reaching the store** — each only arranged what it would find once there. The next topic takes on meeting the read **in front of** the store, and its first question is which layer the copy stands in. ## Cache Architecture Which layer a cache sits in with which strategy: the placement decision across the client, edge, web, application, and database layers, application-controlled cache-aside, write-through and write-behind where the write path is wired to the cache, refresh-ahead that renews an entry before its lifetime expires, the cases where a cache is not really needed, and settling on an acceptable staleness window. ### Cache Placement Source: https://academia.sh/en/courses/scaling-the-data/cache-placement Treating which layer a copy sits in as a design decision: separating the client, edge, web, application, and database layers by scope and capacity, measuring the per-layer hit ratio and miss penalty of a four-layer chain, turning the introductory course's 0.90 cache hit assumption into a result of the design, and showing that a single correctly sized cache delivers the same ratio with fewer touches. The previous topic designed where and in what form the data sits: the store type matched the access pattern, the data was replicated, sharded, and the read path shortened with denormalization. The path really did shorten, but one thing did not change: every read still goes to a store. The next question is meeting the read **in front of** the store — which layer should the copy sit in. This lesson turns that question into a placement decision. The Caching, Queues and Asynchronous Processing course built and measured cache layers, strategies, and invalidation, none of it retold here — what changes is scale itself. In the Introduction to System Design course's back-of-the-envelope computation, the cache was not a strategy but a single number: V9, the fraction of tracking queries served from cache, was taken as 0.90 and brought `reads behind cache/s` down to 41.67 and `requests reaching the store/s` down to 138.89. That number was an **assumption**; this lesson's job is to turn it into a **result** — which placement produces 0.90, and what producing it costs. #### What Defines a Layer Three things define a cache layer. **Scope** is how many requests the same copy can serve: the client cache serves one user, the edge cache every user in a region, the shared web-layer response cache every request, and the in-process application cache only requests landing on its own instance. As scope grows, warming one copy pays off for more requests. **Capacity** is how many entries the layer can hold. When capacity falls under the working set, the layer evicts its own entries and leaks the overflow onward. **Revocability** is whether a placed copy can be removed, and it runs opposite to scope: an application entry is deleted instantly by code, an edge entry needs a separate operation, and a client copy has no removal path at all — it stays until the lifetime given at placement runs out. This measure does not appear in the numbers here, but it decides the outcome in the last section. There is a fifth layer, different from the rest: the **database cache** sits inside the source itself, lowering a query's cost rather than the **number** of queries, and no line of the computation depends on it. This was established and measured in the Caching, Queues and Asynchronous Processing course, not repeated here — why the model below has four layers. #### The Chain Model The model is in-process: layers are objects, counters are explicit, and capacity and access pattern are parameters. No real cache product or cloud environment is set up; what is measured is a **count**, not time, so it does not change from run to run. The access pattern is the course's tracking query: a single-key read by tracking number. A shipment's queries arrive together — the user opens the page, refreshes it, and checks back after a while. This bursting is this lesson's assumption, not added to K01's table. **B1 (this lesson's assumption): a shipment's queries arrive in bursts of length 10, and 1000 shipments are queried at once.** Rationale: the Traffic Layer course's Content Delivery Networks lesson used the same burst length and measured an edge hit rate of 0.8998; the addition here is a concurrent shipment count that defines a **working set** and makes capacity meaningful. Sensitivity: at burst 5 the unique key count doubles and every layer's hit ratio falls; at working set 2000, fixed-capacity layers leak more overflow onward. ```js // cache/layer.mjs — the in-process model of the layer chain. Layers are objects, // counters are explicit. Capacity, burst length, and working set are the model's parameters. export function cache(capacity) { const box = new Map(); const o = { eviction: 0, get(a) { const d = box.get(a); if (d !== undefined) { box.delete(a); box.set(a, d); } return d; }, put(a, d) { box.delete(a); box.set(a, d); if (box.size > capacity) { box.delete(box.keys().next().value); o.eviction += 1; } }, }; return o; } // Access pattern: a single-key read by tracking number. A shipment's queries arrive in // bursts of length `burst`; `workingSet` shipments are queried at once. An `ownerShare` // share of the queries come from the user who started the shipment, the rest from another. export function stream({ requests, workingSet, burst, users, ownerShare = 0.7 }) { let seed = 20260730; const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); const spawn = (id) => ({ id, remaining: burst, owner: Math.floor(rand() * users) + 1 }); let next = 1; const active = Array.from({ length: workingSet }, () => spawn(next++)); const output = []; for (let i = 0; i < requests; i++) { const j = Math.floor(rand() * active.length); const item = active[j]; const requester = rand() < ownerShare ? item.owner : Math.floor(rand() * users) + 1; output.push([requester, `track:${item.id}`]); if ((item.remaining -= 1) === 0) active[j] = spawn(next++); } return output; } // Chain: each request polls the layers in order. A miss counts one poll per layer, one // access at the store, and one write touch back to each layer on the way back. export function chain(requests, layers) { const boxes = layers.map(() => new Map()); const counter = layers.map(() => ({ hit: 0, eviction: 0 })); const s = { store: 0, touch: 0, miss: 0 }; const getBox = (i, user, key) => { const scope = layers[i].scope; const id = scope === "user" ? user : scope === "instance" ? (key.length + Number(key.slice(6))) % layers[i].instance : 0; let c = boxes[i].get(id); if (c === undefined) { c = cache(layers[i].capacity); boxes[i].set(id, c); } return c; }; for (const [user, key] of requests) { let found = -1; for (let i = 0; i < layers.length; i++) { s.touch += 1; if (getBox(i, user, key).get(key) !== undefined) { found = i; break; } } if (found === -1) { s.store += 1; s.miss += 1; s.touch += 1; } else counter[found].hit += 1; const last = found === -1 ? layers.length - 1 : found - 1; for (let i = 0; i <= last; i++) { s.touch += 1; getBox(i, user, key).put(key, key); } } for (let i = 0; i < layers.length; i++) for (const c of boxes[i].values()) counter[i].eviction += c.eviction; return { counter, ...s }; } // M19/K01 back-of-the-envelope inputs (V1-V4, V8): peak read and peak write. export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, peakWrite: (400_000 * 7 / 86_400) * 3, }; ``` The window is 480 seconds; at K01's peak read rate of 416.67 req/s, that window holds 200,000 tracking queries. A **touch** is the model's single unit of cost: one layer poll, one store access, or one write-back. Touches are not weighted — a store access and a memory poll count the same. The placement decision itself assigns the weight; the count only says how much work was done. #### Measurement ```js // cache/placement.mjs — measuring the four-layer chain and returning to the K01 computation import { stream, chain, K01 } from "./layer.mjs"; const REQUESTS = 200_000, BURST = 10, WORKING = 1000, USERS = 40_000; const requests = stream({ requests: REQUESTS, workingSet: WORKING, burst: BURST, users: USERS }); const LAYERS = [ { name: "client", scope: "user", capacity: 4 }, { name: "edge", scope: "shared", capacity: 250 }, { name: "web", scope: "shared", capacity: 600 }, { name: "app", scope: "instance", capacity: 400, instance: 4 }, ]; const unique = new Set(requests.map(([, a]) => a)).size; const r = chain(requests, LAYERS); console.log(`requests ${REQUESTS} (480 s window), unique tracking numbers ${unique}, burst ${BURST}, concurrent shipments ${WORKING}`); console.log(); console.log("layer scope capacity hit hit ratio eviction"); console.log("---------- ----------- --------- -------- ------------ --------"); for (let i = 0; i < LAYERS.length; i++) { const k = LAYERS[i], c = r.counter[i]; const cap = k.scope === "instance" ? `${k.capacity}x${k.instance}` : String(k.capacity); console.log(`${k.name.padEnd(10)} ${k.scope.padEnd(11)} ${cap.padStart(9)} ` + `${String(c.hit).padStart(8)} ${(c.hit / REQUESTS).toFixed(4).padStart(12)} ${String(c.eviction).padStart(8)}`); } console.log(`${"store".padEnd(10)} ${"-".padEnd(11)} ${"-".padStart(9)} ${String(r.store).padStart(8)} ` + `${(r.store / REQUESTS).toFixed(4).padStart(12)} ${"-".padStart(8)}`); const h = 1 - r.store / REQUESTS; const behind = K01.peakRead * (1 - h); console.log(); console.log(`combined hit ratio = ${h.toFixed(4)} (V9 assumption 0.90)`); console.log(`touches per miss = ${2 * LAYERS.length + 1} (${LAYERS.length} polls + 1 store + ${LAYERS.length} write-backs)`); console.log(`touch / request = ${(r.touch / REQUESTS).toFixed(3)}`); console.log(); console.log(`back to K01 (peak read ${K01.peakRead.toFixed(2)}, peak write ${K01.peakWrite.toFixed(2)} req/s)`); console.log(`reads never leaving the client = ${(K01.peakRead * r.counter[0].hit / REQUESTS).toFixed(2)}`); console.log(`reads reaching the app = ${(K01.peakRead * (r.counter[3].hit + r.store) / REQUESTS).toFixed(2)}`); console.log(`reads behind cache/s = ${behind.toFixed(2)} (V9 gives 41.67)`); console.log(`requests reaching the store/s = ${(behind + K01.peakWrite).toFixed(2)} (V9 gives 138.89)`); console.log(`write/read at the store = ${(K01.peakWrite / behind).toFixed(2)} (V9 gives 2.33)`); ``` ``` requests 200000 (480 s window), unique tracking numbers 20454, burst 10, concurrent shipments 1000 layer scope capacity hit hit ratio eviction ---------- ----------- --------- -------- ------------ -------- client user 4 140045 0.7002 34810 edge shared 250 16520 0.0826 43185 web shared 600 14390 0.0720 28445 app instance 400x4 8394 0.0420 19051 store - - 20651 0.1033 - combined hit ratio = 0.8967 (V9 assumption 0.90) touches per miss = 9 (4 polls + 1 store + 4 write-backs) touch / request = 2.531 back to K01 (peak read 416.67, peak write 97.22 req/s) reads never leaving the client = 291.76 reads reaching the app = 60.51 reads behind cache/s = 43.02 (V9 gives 41.67) requests reaching the store/s = 140.25 (V9 gives 138.89) write/read at the store = 2.26 (V9 gives 2.33) ``` All these numbers belong to the **computed** class: counted over a deterministic stream, with no time measured, so they do not change from run to run. The first result is about V9. The four layers' combined hit ratio came out to 0.8967, slightly under the assumed 0.90. The difference looks small but shows up in the computation: `reads behind cache/s` 43.02 instead of 41.67, `requests reaching the store/s` 140.25 instead of 138.89, `write/read at the store` 2.26 instead of 2.33. The assumption is now defensible, because the placement producing it has been written down, and these lines get recalculated whenever capacity or the working set changes. The second result is about the **distribution** of the hit ratio. 0.7002 of requests stopped at the client layer, in a cache of only four entries, since most queries inside a burst come from the same user — so 291.76 reads per second never reach the network. The remaining three layers stopped 0.1966 combined, and reads reaching the app fell from 416.67 to 60.51 req/s. The Content Delivery Networks lesson in the Traffic Layer course measured the same line at 41.76; its application cache could hold every key in the window, while these layers' capacity sits under the working set, as the eviction column shows. The third result is about where the cost sits: the **miss penalty**. A miss makes nine touches — four polls, one store access, four write-backs on the way back — averaging 2.531 touches/request. As the chain lengthens the hit ratio rises, but **every** miss walks a longer path. #### The Limit of Adding a Layer A layer's value is the ratio of requests stopped to touches added. The run below builds the chain from scratch to isolate each layer's own share, and shows what it takes to match the same hit ratio with a single cache. ```js // cache/layer-count.mjs — what is gained and paid as layers are added to the chain import { stream, chain, K01 } from "./layer.mjs"; const REQUESTS = 200_000; const requests = stream({ requests: REQUESTS, workingSet: 1000, burst: 10, users: 40_000 }); const FULL = [ { name: "client", scope: "user", capacity: 4 }, { name: "edge", scope: "shared", capacity: 250 }, { name: "web", scope: "shared", capacity: 600 }, { name: "app", scope: "instance", capacity: 400, instance: 4 }, ]; console.log("chain hit ratio store/s touch/request stopped added touches touches/stopped"); console.log("---------------------------- --------- ------- -------------- ----------- ---------------- ------------------"); let prev = { touch: REQUESTS, store: REQUESTS }; // cacheless: one store access per request for (let n = 0; n <= FULL.length; n++) { const r = n === 0 ? { store: REQUESTS, touch: REQUESTS } : chain(requests, FULL.slice(0, n)); const h = 1 - r.store / REQUESTS; const storeRate = K01.peakRead * (1 - h) + K01.peakWrite; const stopped = prev.store - r.store; const added = r.touch - prev.touch; const label = n === 0 ? "(cacheless)" : "+" + FULL[n - 1].name; console.log(`${label.padEnd(28)} ${h.toFixed(4).padStart(9)} ${storeRate.toFixed(2).padStart(7)} ` + `${(r.touch / REQUESTS).toFixed(3).padStart(14)} ${(n === 0 ? "-" : String(stopped)).padStart(11)} ` + `${(n === 0 ? "-" : String(added)).padStart(16)} ${(n === 0 ? "-" : (added / stopped).toFixed(2)).padStart(18)}`); prev = r; } const full = chain(requests, FULL); const h = 1 - full.store / REQUESTS; console.log(); console.log(`four-layer combined hit ratio = ${h.toFixed(4)}`); console.log(`capacity needed for the same hit ratio with a single layer:`); for (const cap of [250, 600, 1000, 1200, 2000]) { const t = chain(requests, [{ name: "single", scope: "shared", capacity: cap }]); const th = 1 - t.store / REQUESTS; console.log(` capacity ${String(cap).padStart(4)} -> hit ratio ${th.toFixed(4)} store/s ${(K01.peakRead * (1 - th) + K01.peakWrite).toFixed(2).padStart(6)} touch/request ${(t.touch / REQUESTS).toFixed(3)}`); } ``` ``` chain hit ratio store/s touch/request stopped added touches touches/stopped ---------------------------- --------- ------- -------------- ----------- ---------------- ------------------ (cacheless) 0.0000 513.89 1.000 - - - +client 0.7002 222.13 1.600 140045 119910 0.86 +edge 0.7828 187.71 2.034 16520 86870 5.26 +web 0.8548 157.73 2.324 14390 58090 4.04 +app 0.8967 140.25 2.531 8394 41302 4.92 four-layer combined hit ratio = 0.8967 capacity needed for the same hit ratio with a single layer: capacity 250 -> hit ratio 0.2321 store/s 417.20 touch/request 2.536 capacity 600 -> hit ratio 0.5236 store/s 295.71 touch/request 1.953 capacity 1000 -> hit ratio 0.7915 store/s 184.11 touch/request 1.417 capacity 1200 -> hit ratio 0.8646 store/s 153.63 touch/request 1.271 capacity 2000 -> hit ratio 0.8975 store/s 139.95 touch/request 1.205 ``` The first table gives diminishing returns in numbers. In the cacheless design, requests reaching the store run at 513.89 req/s — the whole of K01's peak load at the edge. The first layer brings that to 222.13, adding only 0.86 touches per request stopped, since a stopped request already cancels a store access. The next three layers' ratios are 5.26, 4.04, and 4.92: each charges close to five extra touches per request saved. The fourth layer stops 8394 requests, adds 41,302 touches, and lowers the store load by only 17.49 req/s. The second table is more uncomfortable. A **single** shared cache with capacity 2000 gives a 0.8975 hit ratio — above the four-layer chain's 0.8967 — with 1.205 touches per request instead of 2.531. The chain's justification is not the combined hit ratio: a single, right-sized layer delivers the same ratio more cheaply. The chain's justification is **where** the hit ratio gets served. In a single shared cache, every hit costs a network round trip and all pass through one component; in the four-layer placement, 291.76 reads per second never reach the network. In exchange, those reads are served at the layer with the lowest revocability: once placed, the client copy cannot be removed. The placement decision sits between these two sentences, and both are consistent with the same hit ratio. #### Summary - Scope, capacity, and revocability define a cache layer; the database cache sits outside this ranking — it lowers a query's cost, not its count. - The four-layer chain's combined hit ratio measured 0.8967, turning K01's V9 = 0.90 assumption into a design result: `reads behind cache/s` 43.02 instead of 41.67, `requests reaching the store/s` 140.25 instead of 138.89, `write/read at the store` 2.26 instead of 2.33. - 0.7002 of the hit ratio stopped at the four-entry client cache; 291.76 reads per second never reached the network, and reads reaching the app fell from 416.67 to 60.51 req/s. - As the chain lengthens, every miss walks a longer path: at four layers a miss makes nine touches, averaging 2.531 touches/request, with each layer's extra touches per request stopped running 0.86, 5.26, 4.04, and 4.92. - A single shared cache with capacity 2000 delivers a 0.8975 hit ratio with 1.205 touches; the chain's justification is not the hit ratio but which layer serves it. #### Next Step In this lesson every layer worked the same way: check, ask the next one down if missing, put it back on the way up — a single loop in the code, and it has a name, and it is not the only option. A layer's hit ratio depends not only on capacity but on **when an entry drops out**, and what drops an entry is usually a write. K01's computation described a system that is write-heavy where the store sees it: a write/read ratio of 2.33. The next lesson names this application-controlled placement and turns the hit ratio from a fixed number into a function of write frequency, one that says where 0.90 holds and where it does not. ### Cache-Aside Source: https://academia.sh/en/courses/scaling-the-data/cache-aside Treating an application-controlled cache as a scaling tool: turning the hit ratio into a function of the write/read ratio, showing the introductory course's 0.90 assumption fall to 0.7287 at its 0.2333 ratio, showing that every invalidation causes a store read, and measuring whether the application can go to the store on its own when the cache layer partly fails. The previous lesson measured four layers as a chain, all behaving the same way: check, ask the next one down if missing, put it back on the way up. That behavior is called **cache-aside**, and a layer's hit ratio does not depend on capacity alone — a second force, a write, drops entries too. The back-of-the-envelope computation described a system write-heavy where the store sees it: `write/read at the store` 2.33. This lesson turns the hit ratio from a fixed number into a function of write frequency. Cache-aside's mechanics were built and measured in the Caching, Queues and Asynchronous Processing course and are not retold here. The questions here differ: which layer this strategy belongs to, and what it gives at K01's write rate. #### What Application-Controlled Means Cache-aside's defining trait is that the cache sits **beside** the read path: fetching responsibility lies with the application, not the cache. The same read path has two implementations — application code checking by hand, or the read passing through the cache layer — and both go by this name. The difference is not in the mechanics but in where the responsibility sits, which is the only thing the last section measures. This strategy is the only option everywhere the cache **cannot be wired to the write path**. In the shipment tracking service, state events come not from the application's read path but from the carrier, processed through the queue designed in the Application Layer and Service Interaction course and written to the store from there. The read-path cache only learns of the write through an invalidation message — which is why deleting the entry, not updating it, is all it can do. Wiring a write to the cache is the next lesson's subject. The layer choice follows from this too. In the previous lesson's measurement, the application layer's shared cache stopped only a small share of the hit ratio, but had the highest revocability: a delete message applies there instantly. Cache-aside is therefore placed at the application layer; edge and client layers cannot run the same strategy, since they have no delete, only a lifetime. #### The Effect of Writes on the Hit Ratio The model is in-process and single-layer: since the previous lesson's measurement showed that a single shared cache with capacity 2000 covers the working set, capacity is removed as a binding constraint here, leaving write ratio as the only variable. **B1 (this lesson's assumption): state events land on the shipments being queried at that moment.** Rationale: a shipment produces events and gets queried for the same reason — it is in transit now. This is the direction where invalidation is strongest; sensitivity: if half the events land on shipments not being queried, the deletion count and its extra reads fall by half. Not added to K01's table. ```js // cache/aside.mjs — the in-process model of cache-aside. Cache, store, and event stream are // each their own module; counters are explicit. Write/read ratio, capacity, and failure share are parameters. export function openBox(capacity) { const m = new Map(); const o = { eviction: 0, has: (a) => m.has(a), get(a) { const d = m.get(a); if (d !== undefined) { m.delete(a); m.set(a, d); } return d; }, put(a, d) { m.delete(a); m.set(a, d); if (m.size > capacity) { m.delete(m.keys().next().value); o.eviction += 1; } }, remove: (a) => m.delete(a), }; return o; } // Stream: reads and writes land on the same active shipment set (B1). `ratio` is the write/read ratio. export function stream({ reads, ratio, workingSet, burst }) { let seed = 20260730; const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); let next = 1; const active = Array.from({ length: workingSet }, () => ({ id: next++, remaining: burst })); const output = []; let debt = 0; for (let i = 0; i < reads; i++) { const j = Math.floor(rand() * active.length); const item = active[j]; output.push(["read", `track:${item.id}`]); if ((item.remaining -= 1) === 0) active[j] = { id: next++, remaining: burst }; for (debt += ratio; debt >= 1; debt -= 1) output.push(["write", `track:${active[Math.floor(rand() * active.length)].id}`]); } return output; } // Cache-aside: on read, check-fetch-put; on write, write to the store and delete the entry. // invalidate=false: a write does not delete the entry. failure: the share of cache ops that fail. // canBypass=false: the read passes through the cache; the application cannot go to the store itself. export function run(ops, { capacity, invalidate = true, failure = 0, canBypass = true }) { const c = openBox(capacity); let seed = 991; const failed = () => failure > 0 && ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648) < failure; const s = { hit: 0, miss: 0, storeRead: 0, storeWrite: 0, deletion: 0, stale: 0, touch: 0, unserved: 0 }; const truth = new Map(); let version = 0; for (const [type, key] of ops) { if (!truth.has(key)) truth.set(key, 0); if (type === "read") { s.touch += 1; if (failed()) { if (!canBypass) { s.unserved += 1; continue; } s.miss += 1; s.storeRead += 1; s.touch += 1; continue; } if (c.has(key)) { s.hit += 1; if (c.get(key) !== truth.get(key)) s.stale += 1; continue; } s.miss += 1; s.storeRead += 1; s.touch += 2; c.put(key, truth.get(key)); } else { truth.set(key, (version += 1)); s.storeWrite += 1; s.touch += 1; if (!invalidate) continue; s.touch += 1; if (failed()) continue; // the delete is lost: the entry stays stale if (c.remove(key)) s.deletion += 1; } } return { ...s, eviction: c.eviction }; } // M19/K01 back-of-the-envelope inputs (V1-V4, V8). export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, peakWrite: (400_000 * 7 / 86_400) * 3, }; ``` ```js // cache/ratio.mjs — measuring the hit ratio as a function of the write/read ratio import { stream, run, K01 } from "./aside.mjs"; const READS = 200_000, CAPACITY = 2000, WORKING = 1000, BURST = 10; const K01_RATIO = K01.peakWrite / K01.peakRead; console.log(`reads ${READS}, capacity ${CAPACITY}, concurrent shipments ${WORKING}, burst ${BURST}`); console.log(`K01 write/read ratio = ${K01_RATIO.toFixed(4)} (peak write ${K01.peakWrite.toFixed(2)} / peak read ${K01.peakRead.toFixed(2)})`); console.log(); console.log(" write/read write hit store read behind cache/s reaching store/s write/read at store"); console.log("----------- ------- -------- ----------- ------------------ ---------------- -------------------"); for (const ratio of [0, 0.05, 0.1167, K01_RATIO, 0.4667, 1]) { const ops = stream({ reads: READS, ratio, workingSet: WORKING, burst: BURST }); const r = run(ops, { capacity: CAPACITY }); const h = r.hit / READS; const behind = K01.peakRead * (1 - h); const label = Math.abs(ratio - K01_RATIO) < 1e-9 ? `${ratio.toFixed(4)}*` : ratio.toFixed(4); console.log(`${label.padStart(11)} ${String(r.storeWrite).padStart(7)} ${h.toFixed(4).padStart(8)} ` + `${String(r.storeRead).padStart(11)} ${behind.toFixed(2).padStart(18)} ` + `${(behind + K01.peakWrite).toFixed(2).padStart(16)} ${(K01.peakWrite / behind).toFixed(2).padStart(19)}`); } console.log("* the ratio implied by K01's peak rates"); console.log(); const ops = stream({ reads: READS, ratio: K01_RATIO, workingSet: WORKING, burst: BURST }); const withInv = run(ops, { capacity: CAPACITY }); const without = run(ops, { capacity: CAPACITY, invalidate: false }); console.log(`K01 ratio, with invalidation : hit ${(withInv.hit / READS).toFixed(4)} store read ${withInv.storeRead} deletions ${withInv.deletion} stale reads ${withInv.stale}`); console.log(`K01 ratio, without invalidation: hit ${(without.hit / READS).toFixed(4)} store read ${without.storeRead} deletions ${without.deletion} stale reads ${without.stale}`); console.log(`extra store read per deletion = ${((withInv.storeRead - without.storeRead) / withInv.deletion).toFixed(2)}`); console.log(`extra work per write = ${((withInv.touch - without.touch) / withInv.storeWrite).toFixed(2)} touches + ${((withInv.storeRead - without.storeRead) / withInv.storeWrite).toFixed(2)} store reads`); ``` ``` reads 200000, capacity 2000, concurrent shipments 1000, burst 10 K01 write/read ratio = 0.2333 (peak write 97.22 / peak read 416.67) write/read write hit store read behind cache/s reaching store/s write/read at store ----------- ------- -------- ----------- ------------------ ---------------- ------------------- 0.0000 0 0.8976 20483 42.67 139.90 2.28 0.0500 10000 0.8551 28976 60.37 157.59 1.61 0.1167 23340 0.8039 39212 81.69 178.91 1.19 0.2333* 46666 0.7287 54257 113.04 210.26 0.86 0.4667 93340 0.6134 77312 161.07 258.29 0.60 1.0000 200000 0.4523 109549 228.23 325.45 0.43 * the ratio implied by K01's peak rates K01 ratio, with invalidation : hit 0.7287 store read 54257 deletions 33957 stale reads 0 K01 ratio, without invalidation: hit 0.8977 store read 20451 deletions 0 stale reads 105794 extra store read per deletion = 1.00 extra work per write = 2.45 touches + 0.72 store reads ``` All the numbers belong to the **computed** class; they were counted over a deterministic stream, and no time was measured. The first row repeats the previous lesson's result with no writes: hit ratio 0.8976, `requests reaching the store/s` 139.90 — K01's assumption holds here, but not at K01's own write rate. At a write/read ratio of 0.2333, the hit ratio drops to **0.7287**, `reads behind cache/s` climbs from 41.67 to 113.04, and `requests reaching the store/s` from 138.89 to **210.26**. V9 = 0.90 implicitly assumed "nothing ever gets written to the cache"; at its own write rate the system falls nineteen percent short. The `write/read at the store` column is even more interesting. K01 computed this ratio at 2.33 and called the read-heavy system write-heavy where the store sees it. At cache-aside's real hit ratio it falls to **0.86**: invalidation inflates the store's read load enough to turn it read-heavy again. The write count never changed; what changed is the reads each write causes. The last three rows count that causation. At K01's ratio, 33,957 entries were deleted and store reads rose from 20,451 to 54,257: **1.00 extra store read per deletion** — a deleted entry is almost always read again, since the reason it was deleted is exactly that the shipment is active. A write's cost to the cache is thus not "one deletion" but "one deletion plus 0.72 store reads." Switching invalidation off returns the hit ratio to 0.8977, and the bill shows up in a different column: 105,794 stale reads, more than half of all reads. Cache-aside's trade-off sits between these two columns, and the middle ground between them — the acceptable staleness window — is the subject of this topic's last lesson. #### Whether the Application Can Go to the Store on Its Own The difference between the two implementation forms becomes measurable when the cache layer fails. The model makes a share of cache operations fail; the application-controlled form counts this as a miss and goes to the store, while the pass-through form cannot serve the request. ```js // cache/failure.mjs — how the two forms behave when the cache layer partly fails import { stream, run, K01 } from "./aside.mjs"; const READS = 200_000, CAPACITY = 2000; const RATIO = K01.peakWrite / K01.peakRead; const ops = stream({ reads: READS, ratio: RATIO, workingSet: 1000, burst: 10 }); console.log("failure form served hit store read reaching store/s stale reads"); console.log("------- ----------------------- ---------- ------- ----------- ---------------- -----------"); for (const failure of [0, 0.1, 0.5, 1]) { for (const [label, canBypass] of [["application-controlled", true], ["pass-through", false]]) { const r = run(ops, { capacity: CAPACITY, failure, canBypass }); const served = READS - r.unserved; const h = r.hit / READS; const storeRate = K01.peakRead * (r.storeRead / READS) + K01.peakWrite; console.log(`${failure.toFixed(2).padStart(7)} ${label.padEnd(23)} ${(served / READS).toFixed(4).padStart(10)} ` + `${h.toFixed(4).padStart(7)} ${String(r.storeRead).padStart(11)} ${storeRate.toFixed(2).padStart(16)} ${String(r.stale).padStart(11)}`); } } ``` ``` failure form served hit store read reaching store/s stale reads ------- ----------------------- ---------- ------- ----------- ---------------- ----------- 0.00 application-controlled 1.0000 0.7287 54257 210.26 0 0.00 pass-through 1.0000 0.7287 54257 210.26 0 0.10 application-controlled 1.0000 0.6468 70650 244.41 7473 0.10 pass-through 0.8974 0.6468 50126 201.65 7473 0.50 application-controlled 1.0000 0.3209 135818 380.18 18780 0.50 pass-through 0.4876 0.3209 33334 166.67 18780 1.00 application-controlled 1.0000 0.0000 200000 513.89 0 1.00 pass-through 0.0000 0.0000 0 97.22 0 ``` In the healthy state both forms produce the same row; the difference appears only under failure. When the cache fails completely, the application-controlled form keeps serving every request, and requests reaching the store climb to 513.89 req/s — the whole of K01's peak load at the edge, a design requirement: choosing cache-aside requires the store to carry this load briefly. In the pass-through form, served requests drop to zero, and the service's availability becomes tied to the cache layer staying up. The requests-reaching-the-store column misleads here, and is deliberately placed side by side: at a 50 percent failure rate, the pass-through form's store sees **less** load than in the healthy state — 166.67 req/s. While the store load looks low, half of all requests go unanswered — one metric improving cannot be read on its own. The third column carries an unexpected result: at a 10 percent failure rate there are 7473 stale reads, where the healthy run had none. What gets lost is not reads but **deletions**; a failed delete leaves a stale entry without showing up as an error in any counter, so cache-aside's correctness depends on the operation that fails in the quietest way. #### Summary - Cache-aside is the read-path cache where fetching responsibility lies with the application, not the cache; it is the only option when a write cannot be wired to the cache, and deleting the entry is all it can do. - The hit ratio is a function of the write/read ratio: 0.8976 with no writes, **0.7287** at K01's ratio of 0.2333. The V9 = 0.90 assumption implicitly assumed a system with no writes. - Back to K01: `reads behind cache/s` climbs from 41.67 to 113.04, `requests reaching the store/s` from 138.89 to 210.26; `write/read at the store` falls from 2.33 to 0.86, meaning the store turns read-heavy again. - Each deletion causes 1.00 extra store read; a write's cost to the cache is 2.45 touches and 0.72 store reads. Turning invalidation off returns the hit ratio to 0.8977, but produces 105,794 stale reads. - When the cache fails completely, the application-controlled form serves every request and the store takes on 513.89 req/s; in the pass-through form, served requests drop to zero. Under partial failure, lost deletions leave 7473 stale reads. #### Next Step This lesson's entire cost grew from one constraint: the cache was not where the write happened, so it could only delete the old value, never put in the new one, and every deletion caused a store read. That constraint does not always hold. When the path writing shipment state to the store passes through the application's own code, nothing stops it from also writing the new value to the cache — the write goes to both places, and no deletion happens. The same link can go one step further, sending the write to the cache first and the store later in a batch, this time changing a number not on the read side but in the numerator of `write/read at the store`. The next lesson measures these two arrangements: how many places the write path stretches to, how much extra work each write costs, and how many events remain in memory when the process stops. ### Write-Through and Write-Behind Source: https://academia.sh/en/courses/scaling-the-data/write-through-and-write-behind Two arrangements for wiring the write path to the cache: write-through removing the delete and raising the hit ratio from 0.7285 to 0.9170, the second write target added by denormalization stretching the path to two stores, write-behind lowering store writes per event from 2.00 to 1.20 through batching, and measuring the 200 events pending in memory in exchange as a durability window. The previous lesson's entire cost grew from one constraint: the cache was not where the write happened, so it could only delete the old value, never put in the new one, and every deletion caused a store read. That constraint lifts once the path writing shipment state to the store passes through the application's own code: the write goes to both places and no deletion happens; or the link goes one step further, writing to the cache first and the store later in a batch. The first is **write-through**, the second **write-behind** — this lesson measures both. Both strategies' mechanics were built and measured in the Caching, Queues and Asynchronous Processing course and are not retold here. The question here is scale: how many places the write path stretches to, how much extra work each event costs, and which of K01's lines moves. #### The Second Write Target A number this topic established changes in this lesson. The previous topic applied denormalization to shorten the read path: a zone's shipment count is kept in a separate counter instead of being counted at query time. That paid off on the read side but left a silent debt on the write side — every state event now updates **two** places: the shipment's tracking record and its zone's counter. K01's table had only one write; `peak write req/s` 97.22 was computed assuming one event writes to one record. When the write target doubles, the event rate stays the same, but the store write rate does not. This lesson's first measure is therefore **write path length**: how many places one event touches. **B1 (this lesson's assumption): the carrier network is divided into 40 distribution zones.** Rationale: route steps are reported at zone level, and V5's tracking response carries the zone as a field. Sensitivity is in the last measurement: as the zone count grows, the coalescing ratio in batching falls. Not added to K01's table. ```js // cache/write.mjs — the in-process model of the arrangements where the write path is // wired to the cache. Cache, store, and event stream are each their own module; counters // are explicit. Batch window and zone count are parameters; no time is measured, only ops. export function openBox(capacity) { const m = new Map(); const o = { eviction: 0, has: (a) => m.has(a), get: (a) => { const d = m.get(a); if (d !== undefined) { m.delete(a); m.set(a, d); } return d; }, put: (a, d) => { m.delete(a); m.set(a, d); if (m.size > capacity) { m.delete(m.keys().next().value); o.eviction += 1; } }, remove: (a) => m.delete(a), }; return o; } // Every state event updates two places: the shipment's tracking record and the zone's distribution counter. export function stream({ reads, ratio, workingSet, burst, zones }) { let seed = 20260730; const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); let next = 1; const active = Array.from({ length: workingSet }, () => ({ id: next++, remaining: burst })); const output = []; let debt = 0; for (let i = 0; i < reads; i++) { const j = Math.floor(rand() * active.length); const item = active[j]; output.push(["read", `track:${item.id}`]); if ((item.remaining -= 1) === 0) active[j] = { id: next++, remaining: burst }; for (debt += ratio; debt >= 1; debt -= 1) { const id = active[Math.floor(rand() * active.length)].id; output.push(["write", `track:${id}`, `zone:${Math.floor(rand() * zones) + 1}`]); } } return output; } // trackWrite: "delete" (cache-aside) or "update" (write-through). // zoneWrite: "through" or "behind"; with write-behind, the batch window is `batchWindow` writes. export function run(ops, { capacity, trackWrite, zoneWrite, batchWindow = 200 }) { const c = openBox(capacity); const zoneCount = new Map(); const pending = new Map(); // with write-behind, zone counters not yet sent to the store const s = { hit: 0, storeRead: 0, storeWrite: 0, touch: 0, writeTouch: 0, stale: 0, pendingEvents: 0 }; const truth = new Map(); let count = 0, eventDebt = 0, maxPendingEvents = 0; for (const [type, key, zone] of ops) { if (!truth.has(key)) truth.set(key, 0); if (type === "read") { s.touch += 1; if (c.has(key)) { s.hit += 1; if (c.get(key) !== truth.get(key)) s.stale += 1; continue; } s.storeRead += 1; s.touch += 2; c.put(key, truth.get(key)); continue; } const value = truth.get(key) + 1; truth.set(key, value); s.storeWrite += 1; s.writeTouch += 1; // the tracking record always goes to the store s.writeTouch += 1; if (trackWrite === "delete") c.remove(key); else c.put(key, value); zoneCount.set(zone, (zoneCount.get(zone) ?? 0) + 1); // the zone counter lands in the cache first s.writeTouch += 1; if (zoneWrite === "through") { s.storeWrite += 1; s.writeTouch += 1; continue; } pending.set(zone, zoneCount.get(zone)); eventDebt += 1; maxPendingEvents = Math.max(maxPendingEvents, eventDebt); if ((count += 1) % batchWindow === 0) { s.storeWrite += pending.size; s.writeTouch += pending.size; pending.clear(); eventDebt = 0; } } s.pendingEvents = eventDebt; return { ...s, pendingEntries: pending.size, maxPendingEvents }; } // M19/K01 back-of-the-envelope inputs (V1-V4, V8). export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, peakWrite: (400_000 * 7 / 86_400) * 3, }; ``` The model is in-process. No time is measured, only operations counted; the batch triggers on a write count, not a duration, since a duration would be machine-dependent and not a count. #### Three Arrangements ```js // cache/arrangement.mjs — comparing the three write arrangements on the same stream import { stream, run, K01 } from "./write.mjs"; const READS = 200_000, CAPACITY = 2000, ZONES = 40, WINDOW = 200; const RATIO = K01.peakWrite / K01.peakRead; const ops = stream({ reads: READS, ratio: RATIO, workingSet: 1000, burst: 10, zones: ZONES }); const EVENTS = ops.filter(([t]) => t === "write").length; console.log(`reads ${READS}, state events ${EVENTS}, zones ${ZONES}, batch window ${WINDOW} writes`); console.log(`K01: peak read ${K01.peakRead.toFixed(2)}/s, peak write ${K01.peakWrite.toFixed(2)}/s\n`); const ARRANGEMENTS = [ ["cache-aside + zone through", { trackWrite: "delete", zoneWrite: "through" }], ["write-through + zone through", { trackWrite: "update", zoneWrite: "through" }], ["write-through + zone behind", { trackWrite: "update", zoneWrite: "behind" }], ]; console.log("arrangement hit store read store write write/event path length reaching store/s write/read at store pending events"); console.log("------------------------------ ------ ----------- ----------- ----------- ------------- ---------------- ------------------- --------------"); for (const [label, choice] of ARRANGEMENTS) { const r = run(ops, { capacity: CAPACITY, batchWindow: WINDOW, ...choice }); const h = r.hit / READS; const behind = K01.peakRead * (1 - h); const writeRate = K01.peakWrite * (r.storeWrite / EVENTS); console.log(`${label.padEnd(30)} ${h.toFixed(4).padStart(6)} ${String(r.storeRead).padStart(11)} ` + `${String(r.storeWrite).padStart(11)} ${(r.storeWrite / EVENTS).toFixed(2).padStart(11)} ` + `${(r.writeTouch / EVENTS).toFixed(2).padStart(13)} ` + `${(behind + writeRate).toFixed(2).padStart(16)} ${(writeRate / behind).toFixed(2).padStart(19)} ` + `${String(r.maxPendingEvents).padStart(14)}`); } const writeBehind = run(ops, { capacity: CAPACITY, batchWindow: WINDOW, trackWrite: "update", zoneWrite: "behind" }); console.log(`\nloss window if the same rule applied to the tracking record too:`); console.log(` events pending in memory = ${writeBehind.maxPendingEvents}`); console.log(` event record 220 bytes (V6) -> ${writeBehind.maxPendingEvents * 220} bytes unwritten`); console.log(` peak write ${K01.peakWrite.toFixed(2)}/s -> ${(writeBehind.maxPendingEvents / K01.peakWrite).toFixed(2)} seconds of event flow`); console.log("\nwrites landing on the same key inside the batch window are the coalescing ratio:"); console.log("zone count store write write/event reaching store/s"); for (const zones of [40, 200, 1000]) { const a = stream({ reads: READS, ratio: RATIO, workingSet: 1000, burst: 10, zones }); const r = run(a, { capacity: CAPACITY, batchWindow: WINDOW, trackWrite: "update", zoneWrite: "behind" }); const behind = K01.peakRead * (1 - r.hit / READS); const writeRate = K01.peakWrite * (r.storeWrite / EVENTS); console.log(`${String(zones).padStart(12)} ${String(r.storeWrite).padStart(11)} ` + `${(r.storeWrite / EVENTS).toFixed(2).padStart(11)} ${(behind + writeRate).toFixed(2).padStart(16)}`); } ``` ``` reads 200000, state events 46666, zones 40, batch window 200 writes K01: peak read 416.67/s, peak write 97.22/s arrangement hit store read store write write/event path length reaching store/s write/read at store pending events ------------------------------ ------ ----------- ----------- ----------- ------------- ---------------- ------------------- -------------- cache-aside + zone through 0.7285 54308 93332 2.00 4.00 307.59 1.72 0 write-through + zone through 0.9170 16592 93332 2.00 4.00 229.01 5.63 0 write-through + zone behind 0.9170 16592 55908 1.20 3.20 151.04 3.37 200 loss window if the same rule applied to the tracking record too: events pending in memory = 200 event record 220 bytes (V6) -> 44000 bytes unwritten peak write 97.22/s -> 2.06 seconds of event flow writes landing on the same key inside the batch window are the coalescing ratio: zone count store write write/event reaching store/s 40 55908 1.20 151.04 200 75983 1.63 192.87 1000 88862 1.90 219.70 ``` All the numbers belong to the **computed** class: they were counted over a deterministic stream, and no time was measured. #### Reading the Numbers **Write-through fixes the read side.** With the delete removed, the hit ratio rose from 0.7285 to 0.9170, and store reads fell from 54,308 to 16,592. Notably, 0.9170 sits even **above** the no-write run's 0.8976: a write not only stops dropping the entry, it warms it — the moment a shipment produces an event it enters the cache, and the next query does not miss. V9's 0.90 is exceeded here for the first time, and the reason is not an assumption but the write path being wired. **The write path now stretches to two stores.** In the first two arrangements, store writes per event are 2.00; path length is 4.00 touches — two store writes, one tracking-cache update, one zone-counter update, a cost K01 never saw: the table assumed one event writes to one record. The second write target is the price of the denormalization decision, pulling `requests reaching the store/s` upward. **Write-behind fixes the write side.** Holding the zone counter in the cache first and batching it to the store every 200 writes brings store writes per event down from 2.00 to **1.20**: consecutive increments landing on the same zone coalesce into a single update. `requests reaching the store/s` drops from 229.01 to **151.04**, and `write/read at the store` falls from 5.63 to 3.37. The three arrangements' `requests reaching the store/s` column reads together: 307.59 → 229.01 → 151.04. K01's 138.89 equals none of them, since that computation knew neither invalidation nor the second write target. 151.04 is that line's real value once the design is finished. **The coalescing ratio depends on the key space.** The last table gives the rule: whatever number of writes land on the same key inside the window is what gets gained. At 40 zones, a 200-write window leaves 1.20 store writes per event; at 1000 zones, nearly every write lands on a separate key, and the ratio climbs to 1.90, `requests reaching the store/s` to 219.70. Write-behind pays off on a small, hot key space; on a wide one it only adds latency. #### Which Data Goes With Which Arrangement Write-behind's cost is written in the last three lines. Since the batch window is 200 writes, up to 200 events' worth of unwritten changes sit in memory at any moment — 2.06 seconds of event flow at peak write rate. If the process stopped then, these changes would be lost. For the zone counter, that loss is tolerable, since the counter is **derived**: the event records sit in the store, and the counter can be recomputed from them. For the tracking record and the event record itself, it is not. Per V6, an event record is 220 bytes, kept 730 days per V12 for contract disputes. The 200 lost events are 44,000 bytes, with no second copy anywhere. The rule is therefore written by data, not by layer: **data whose source of truth sits elsewhere** can use write-behind; **data that is itself the source of truth** cannot. Two keys in the same cache layer running under two different arrangements is not an inconsistency but a placement decision. #### Summary - The zone counter added by denormalization stretched the write path to two stores: 2.00 store writes per event, path length 4.00 touches. K01's table assumed one event writes to one record. - Removing the delete, write-through raised the hit ratio from 0.7285 to 0.9170 — above V9's 0.90 — since a write warms the entry rather than dropping it; store reads fell from 54,308 to 16,592. - Write-behind lowered store writes per event on the zone counter to 1.20; `requests reaching the store/s` fell from 229.01 to 151.04, `write/read at the store` from 5.63 to 3.37. - The coalescing ratio is how many writes in the batch window land on the same key: 1.20 store writes per event at 40 zones, 1.90 at 1000, and `requests reaching the store/s` 219.70. - Write-behind's loss window is 200 events, 44,000 bytes, and 2.06 seconds of flow; acceptable for derived data whose source sits elsewhere, not for the event record that is itself the source of truth. #### Next Step In this lesson, writes kept an entry in the cache alive; but an entry for a key that never receives a write drops for one reason only: its lifetime expires. The moment it does, a request for that key misses, goes to the store, and waits — and if the key is hot, many requests do the same thing at once. Waiting for expiry is not mandatory: while still valid and nearing the end of its lifetime, an entry can be refreshed in the background, and the reader never sees a miss. That refresh is not free; some refreshed entries are never read again, and those refreshes are pure load written to the store for nothing. The next lesson turns the threshold into a parameter and puts two numbers face to face: misses saved against refreshes wasted. ### Refresh-Ahead Source: https://academia.sh/en/courses/scaling-the-data/refresh-ahead Measuring an entry's refresh in the background before it expires: the lifetime dropping the hit ratio from 0.9170 to 0.7109, waiting requests falling and wasted refreshes rising as the threshold ratio grows, the store load at threshold 1 equaling the cacheless design's 513.89 req/s while the hit ratio still reads 0.8286, and showing that the real lever is lifetime, not threshold. The previous lesson had writes keep a cache entry alive. A key that never sees a write loses its entry for exactly one reason: it expires. In this design, lifetime is not an optional setting but a safety net — this topic's second lesson measured a lost invalidation message leaving a stale entry with no counter showing it. That is why an entry gets an upper-bound lifetime. That bound has a cost — this lesson's first number. The moment an entry expires, the request asking for that key misses, goes to the store, and **waits**. There is a way to remove the wait: while still valid, the entry is refreshed in the background near the end of its lifetime. This is called **refresh-ahead**. This lesson sweeps the threshold as a parameter and sets two numbers against each other: recovered waiting against wasted refresh. #### The Clock and the Lifetime The model is in-process and represents time with a counter: each read advances the clock one tick. Converting a tick to seconds uses K01's peak read rate — at 416.67 req/s, one second is 416.67 ticks. Real time is never measured; everything measured is a count that never changes between runs. This lesson takes the lifetime as 5 seconds. **B1 (this lesson's assumption): the upper-bound lifetime given to a tracking record's cache entry is 5 seconds.** Rationale: this duration bounds how long a lost invalidation can last, and what it should be is this topic's last lesson's subject; it is held fixed here so threshold is the only variable. Sensitivity is given in the last measurement. Not added to K01's table. ```js // cache/refresh.mjs — the in-process model of refresh-ahead. The clock is a counter: each read // advances one tick, converted to seconds using K01's peak read rate. Lifetime, threshold ratio, // and the selection rule are parameters; time is not measured, operations are counted. export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, peakWrite: (400_000 * 7 / 86_400) * 3, }; export function stream({ reads, ratio, workingSet, burst }) { let seed = 20260730; const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); let next = 1; const active = Array.from({ length: workingSet }, () => ({ id: next++, remaining: burst })); const output = []; let debt = 0; for (let i = 0; i < reads; i++) { const j = Math.floor(rand() * active.length); const item = active[j]; output.push(["read", `track:${item.id}`]); if ((item.remaining -= 1) === 0) active[j] = { id: next++, remaining: burst }; for (debt += ratio; debt >= 1; debt -= 1) output.push(["write", `track:${active[Math.floor(rand() * active.length)].id}`]); } return output; } // threshold: if an entry's remaining lifetime falls below this ratio of the lifetime, it is // refreshed from the store on read. threshold = 0 means no refresh-ahead. // minSightings: only keys read at least this many times are refreshed (selective refresh). export function run(ops, { capacity, lifetime, threshold, minSightings = 0 }) { const box = new Map(); // key -> { expires, refreshed } const s = { hit: 0, waiting: 0, refresh: 0, wastedRefresh: 0, storeRead: 0, storeWrite: 0 }; const sightings = new Map(); let clock = 0; const evict = (key) => { const g = box.get(key); if (g && g.refreshed) s.wastedRefresh += 1; box.delete(key); }; const put = (key, refreshed) => { box.delete(key); box.set(key, { expires: clock + lifetime, refreshed }); if (box.size > capacity) evict(box.keys().next().value); }; for (const [type, key] of ops) { if (type === "write") { s.storeWrite += 1; if (box.has(key)) put(key, false); continue; } clock += 1; sightings.set(key, (sightings.get(key) ?? 0) + 1); const g = box.get(key); if (g === undefined || g.expires <= clock) { if (g !== undefined) evict(key); // expired entry: wasted if it had been refreshed s.waiting += 1; s.storeRead += 1; put(key, false); continue; } s.hit += 1; if (g.refreshed) g.refreshed = false; // refreshed value read: not wasted if (threshold > 0 && g.expires - clock <= lifetime * threshold && sightings.get(key) >= minSightings) { s.refresh += 1; s.storeRead += 1; put(key, true); } else { box.delete(key); box.set(key, g); // moved to the most-recently-used end } } for (const key of [...box.keys()]) evict(key); return s; } ``` **Wasted refresh** is a refreshed entry dropping without ever being read again. The model counts this with a flag: if the refreshed entry is read, the flag is cleared; if the entry expires or is evicted without being read, the wasted-refresh counter increments. The write path carries over the previous lesson's decision — write-through — so a write does not drop the entry, it restarts its lifetime. #### Sweeping the Threshold ```js // cache/threshold.mjs — sweeping the threshold ratio: recovered waiting versus wasted refresh import { stream, run, K01 } from "./refresh.mjs"; const READS = 200_000, CAPACITY = 2000; const RATIO = K01.peakWrite / K01.peakRead; const LIFETIME_S = 5, LIFETIME = Math.round(LIFETIME_S * K01.peakRead); // lifetime in read ticks const ops = stream({ reads: READS, ratio: RATIO, workingSet: 1000, burst: 10 }); console.log(`reads ${READS}, capacity ${CAPACITY}, lifetime ${LIFETIME_S} s = ${LIFETIME} read ticks`); console.log(`one tick = 1/${K01.peakRead.toFixed(2)} s (K01 peak read rate)\n`); console.log("threshold hit waiting refresh wasted refresh wasted ratio store read reads behind cache/s requests reaching the store/s"); console.log("--------- ------ ------- ------- -------------- ------------ ---------- -------------------- ------------------------------"); for (const threshold of [0, 0.1, 0.25, 0.5, 1]) { const r = run(ops, { capacity: CAPACITY, lifetime: LIFETIME, threshold }); const behind = K01.peakRead * (r.storeRead / READS); console.log(`${threshold.toFixed(2).padStart(9)} ${(r.hit / READS).toFixed(4).padStart(6)} ` + `${String(r.waiting).padStart(7)} ${String(r.refresh).padStart(7)} ${String(r.wastedRefresh).padStart(14)} ` + `${(r.refresh === 0 ? 0 : r.wastedRefresh / r.refresh).toFixed(4).padStart(12)} ` + `${String(r.storeRead).padStart(10)} ${behind.toFixed(2).padStart(20)} ${(behind + K01.peakWrite).toFixed(2).padStart(30)}`); } console.log("\nwasted refresh does not drop by excluding heavily read keys (threshold 0.25):"); console.log("min sightings refresh wasted refresh wasted ratio"); for (const minSeen of [0, 3, 5, 8]) { const r = run(ops, { capacity: CAPACITY, lifetime: LIFETIME, threshold: 0.25, minSightings: minSeen }); console.log(`${String(minSeen).padStart(13)} ${String(r.refresh).padStart(8)} ${String(r.wastedRefresh).padStart(15)} ` + `${(r.wastedRefresh / r.refresh).toFixed(4).padStart(13)}`); } console.log("\nthe real lever is lifetime (threshold 0 next to threshold 0.25):"); console.log("lifetime (s) thr 0 hit thr 0 store thr 0.25 hit thr 0.25 store thr 0.25 waiting requests reaching the store/s"); for (const sec of [2, 5, 15, 60]) { const l = Math.round(sec * K01.peakRead); const a = run(ops, { capacity: CAPACITY, lifetime: l, threshold: 0 }); const b = run(ops, { capacity: CAPACITY, lifetime: l, threshold: 0.25 }); const behind = K01.peakRead * (b.storeRead / READS); console.log(`${String(sec).padStart(12)} ${(a.hit / READS).toFixed(4).padStart(10)} ${String(a.storeRead).padStart(12)} ` + `${(b.hit / READS).toFixed(4).padStart(13)} ${String(b.storeRead).padStart(15)} ${String(b.waiting).padStart(17)} ` + `${(behind + K01.peakWrite).toFixed(2).padStart(29)}`); } ``` ``` reads 200000, capacity 2000, lifetime 5 s = 2083 read ticks one tick = 1/416.67 s (K01 peak read rate) threshold hit waiting refresh wasted refresh wasted ratio store read reads behind cache/s requests reaching the store/s --------- ------ ------- ------- -------------- ------------ ---------- -------------------- ------------------------------ 0.00 0.7109 57815 0 0 0.0000 57815 120.45 217.67 0.10 0.7363 52731 9780 1862 0.1904 62511 130.23 227.45 0.25 0.7660 46806 23922 4120 0.1722 70728 147.35 244.57 0.50 0.7994 40128 50540 8728 0.1727 90668 188.89 286.11 1.00 0.8286 34270 165730 27825 0.1679 200000 416.67 513.89 wasted refresh does not drop by excluding heavily read keys (threshold 0.25): min sightings refresh wasted refresh wasted ratio 0 23922 4120 0.1722 3 22693 4035 0.1778 5 17116 3662 0.2140 8 8524 3075 0.3607 the real lever is lifetime (threshold 0 next to threshold 0.25): lifetime (s) thr 0 hit thr 0 store thr 0.25 hit thr 0.25 store thr 0.25 waiting requests reaching the store/s 2 0.4884 102320 0.5263 116205 94735 339.32 5 0.7109 57815 0.7660 70728 46806 244.57 15 0.8686 26288 0.8884 31562 22315 162.98 60 0.8976 20482 0.8977 20510 20461 139.95 ``` All the numbers belong to the **computed** class: they were counted over a deterministic stream. #### Reading the Numbers The first row hands over the safety net's bill. The previous lesson measured a hit ratio of 0.9170 with write-through; once a five-second upper-bound lifetime is placed on top of that, the hit ratio drops to **0.7109** and `requests reaching the store/s` climbs from 151.04 to 217.67. The bound placed against lost invalidations costs one-fifth of the hit ratio. As the threshold grows, two columns move in opposite directions. Waiting requests fall from 57,815 to 34,270; against that, the refresh count climbs from zero to 165,730, and 27,825 of those go to waste. **The wasted ratio is nearly independent of the threshold**: in four of the five rows it sits between 0.17 and 0.19. The reason is structural — a refresh going to waste means that key is never read again, and this is unavoidable on a burst's last query. No counter here can tell in advance that a query is the last one. The second table confirms this. The rule "do not refresh lightly read keys" should intuitively have cut wasted refresh; the measurement says the opposite. Once the threshold is pulled to eight, the refresh count drops from 23,922 to 8,524, but the wasted ratio climbs from 0.1722 to **0.3607**. The excluded refreshes were the ones already paying off: a heavily read key sits closer to the end of its burst. The waste sits at the end of access, not the beginning, and cannot be separated out by view count. The last row carries this lesson's harshest number. At threshold 1 — meaning every read triggers a refresh — the hit ratio sits at a respectable-looking **0.8286**, but store reads number 200,000, the entirety of all requests. `requests reaching the store/s` is **513.89**: the same number this topic's first lesson computed with no cache at all. The cache is running, the hit counter is filling up, the dashboard looks healthy, and the store is carrying the full load. **A high hit ratio, on its own, says nothing about store load.** #### The Real Lever The third table sets threshold next to lifetime and reverses the ranking. As lifetime grows from 2 seconds to 60 seconds, the threshold-free hit ratio climbs from 0.4884 to 0.8976, while `requests reaching the store/s` drops from 339.32 to 139.95. Over the same range, the threshold's contribution keeps shrinking: it raises the hit ratio by 0.0379 at 2 seconds, and by only 0.0001 at 60 seconds. At a 60-second lifetime, refresh-ahead makes 28 extra store reads and cuts waiting requests from 20,482 to 20,461 — a measurable but meaningless gain. The rule follows from this: **refresh-ahead cuts waiting only when the lifetime stays short relative to the access burst.** If the lifetime is longer than the burst, the entry is already read again before it expires, and there is nothing left to refresh. If the lifetime is short, refreshing pays off, but every waiting request it saves loads an extra read onto the store, and roughly one in six of those reads goes to waste. This makes refresh-ahead a **latency** tool, not a load-reduction tool. K01's `requests reaching the store/s` line went **up** every time in this lesson. A design decision does not have to improve the computation; it has to be measured, its trade-off written down. #### Summary - The five-second upper-bound lifetime placed against lost invalidations dropped the hit ratio from 0.9170 to 0.7109 and raised `requests reaching the store/s` from 151.04 to 217.67. - As the threshold rose from 0 to 1, waiting requests fell from 57,815 to 34,270, refresh climbed to 165,730 with 27,825 of it wasted; the wasted ratio stayed near 0.17 independent of the threshold. - Excluding lightly read keys does not cut wasted refresh — it raises the ratio from 0.1722 to 0.3607: the waste sits at the end of access and cannot be separated out by view count. - At threshold 1, the hit ratio reads 0.8286 while store reads number 200,000 and `requests reaching the store/s` reaches 513.89 — the cacheless design's number. A high hit ratio says nothing about store load. - The real lever is lifetime: as it rises from 2 seconds to 60 seconds, the hit ratio climbs from 0.4884 to 0.8976, `requests reaching the store/s` drops from 339.32 to 139.95, and refresh-ahead's contribution falls to 0.0001. #### Next Step This lesson's last number leaves an uncomfortable question. At a hit ratio of 0.8286, the store was carrying 513.89 req/s, as if no cache existed at all. A running cache, then, can produce the same load as no cache at all. That points to a question this topic has not asked yet, from the opposite direction: is a cache actually needed. Every lesson up to here assumed a cache exists and argued only over how to arrange it. The next lesson removes that assumption: which access patterns leave `requests reaching the store/s` unchanged by adding a cache, which patterns **raise** it, and where the boundary starts at which a cacheless design is a genuine anti-pattern. ### Cacheless Design Anti-Pattern Source: https://academia.sh/en/courses/scaling-the-data/cacheless-design-anti-pattern Deciding whether a cache is needed from the access pattern: a cacheless design loading 513.89 req/s onto the store for the tracking query, the cache missing entirely on the period scan while still costing 2.00 extra touches per request, a 5000x repeat rate in hot writes dropping to a 0.3001 hit ratio, and passing the bulk scan through the same cache cutting the tracking hit ratio by 0.4263 while raising store access by 85,252. The previous lesson's last number left an uncomfortable question. At a hit ratio of 0.8286, the store was carrying 513.89 req/s, as if no cache existed at all. Every lesson up to here assumed a cache exists and argued only over how to arrange it. This lesson removes that assumption and asks the question from the opposite direction: is a cache actually needed. The question has two wrong answers, both from the same flaw. The first is the **cacheless design anti-pattern**: every read going to the store even though the access pattern is full of repetition. The second is its mirror image: putting a cache on a path that carries no repetition. Both are the result of never measuring the access pattern, so this lesson measures it. #### Four Patterns The model is in-process: the same cache runs against four separate streams, counters left explicit. All four streams are derived from K01's three flows — the tracking query, the state event write, and the end-of-day billing scan — with rates from K01's computation: peak read 416.67 req/s, bulk scan 833.33 records/s. The window is again 480 seconds. ```js // cache/patterns.mjs — the same cache run against four access patterns. Cache and store are // modules of their own; counters are explicit. Patterns are derived from K01's three streams. export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, // 416.67 req/s peakWrite: (400_000 * 7 / 86_400) * 3, // 97.22 req/s bulkScanRate: (30 * 400_000) / (4 * 3600), // 833.33 records/s }; export const WINDOW = 480; // seconds function randomGenerator(seed) { return () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); } // Tracking query: one shipment's queries arrive in bursts of `burst`. export function trackingStream(requests, workingSet = 1000, burst = 10) { const rand = randomGenerator(20260730); let next = 1; const active = Array.from({ length: workingSet }, () => ({ id: next++, remaining: burst })); const output = []; for (let i = 0; i < requests; i++) { const j = Math.floor(rand() * active.length); output.push(["read", `track:${active[j].id}`]); if ((active[j].remaining -= 1) === 0) active[j] = { id: next++, remaining: burst }; } return output; } // Billing scan: every seller-day record is read exactly once, in order. export function scanStream(records) { return Array.from({ length: records }, (_, i) => ["read", `period:${i + 1}`]); } // Hot write: a small key space takes far more writes than reads. export function hotWriteStream(ops, keys = 40, writeShare = 0.7) { const rand = randomGenerator(4242); return Array.from({ length: ops }, () => { const a = `zone:${Math.floor(rand() * keys) + 1}`; return [rand() < writeShare ? "write" : "read", a]; }); } // Interleaves two streams: `everyN` tracking requests, then `thenN` scan requests. export function interleave(a, b, everyN, thenN) { const output = []; let i = 0, j = 0; while (i < a.length || j < b.length) { for (let k = 0; k < everyN && i < a.length; k++) output.push(a[i++]); for (let k = 0; k < thenN && j < b.length; k++) output.push(b[j++]); } return output; } // skip: keys in this namespace never touch the cache at all. export function run(ops, { capacity, skip = null }) { const m = new Map(); const s = { request: 0, read: 0, hit: 0, storeRead: 0, storeWrite: 0, touch: 0, trackingRequest: 0, trackingHit: 0 }; for (const [type, key] of ops) { s.request += 1; const tracking = key.startsWith("track:"); if (tracking) s.trackingRequest += 1; if (skip !== null && key.startsWith(skip)) { if (type === "read") s.storeRead += 1; else s.storeWrite += 1; s.touch += 1; continue; } if (type === "write") { s.storeWrite += 1; s.touch += 2; m.delete(key); continue; } s.read += 1; s.touch += 1; if (m.has(key)) { s.hit += 1; if (tracking) s.trackingHit += 1; const d = m.get(key); m.delete(key); m.set(key, d); continue; } s.storeRead += 1; s.touch += 2; m.set(key, 1); if (m.size > capacity) m.delete(m.keys().next().value); } return s; } ``` ```js // cache/decision.mjs — what the cache gains and costs across four patterns import { K01, WINDOW, trackingStream, scanStream, hotWriteStream, interleave, run } from "./patterns.mjs"; const CAPACITY = 2000; const READS = Math.round(K01.peakRead * WINDOW); const SCAN = Math.round(K01.bulkScanRate * WINDOW); const tracking = trackingStream(READS); const scan = scanStream(SCAN); const hot = hotWriteStream(READS); console.log(`window ${WINDOW} s: tracking query ${READS}, period scan ${SCAN} records, capacity ${CAPACITY}`); console.log(`(K01: peak read ${K01.peakRead.toFixed(2)}/s, bulk scan ${K01.bulkScanRate.toFixed(2)} records/s)\n`); const PATTERNS = [ ["tracking query", tracking], ["period scan", scan], ["hot write", hot], ["tracking + scan", interleave(tracking, scan, 1, 2)], ]; console.log("pattern request read unique keys repeat hit store access cacheless diff extra touch/req"); console.log("--------------- ------- ------ ------------- ------- ------ ------------- ---------- ------ ---------------"); for (const [name, stream] of PATTERNS) { const r = run(stream, { capacity: CAPACITY }); const unique = new Set(stream.map(([, a]) => a)).size; const store = r.storeRead + r.storeWrite; console.log(`${name.padEnd(15)} ${String(r.request).padStart(7)} ${String(r.read).padStart(6)} ${String(unique).padStart(13)} ` + `${(r.request / unique).toFixed(2).padStart(7)} ${(r.hit / r.read).toFixed(4).padStart(6)} ` + `${String(store).padStart(13)} ${String(r.request).padStart(10)} ${(store - r.request).toString().padStart(6)} ` + `${((r.touch - r.request) / r.request).toFixed(2).padStart(15)}`); } console.log("\nwhat happens to the tracking query if the period scan passes through the same cache:"); const together = interleave(tracking, scan, 1, 2); const passing = run(together, { capacity: CAPACITY }); const bypassing = run(together, { capacity: CAPACITY, skip: "period:" }); const alone = run(tracking, { capacity: CAPACITY }); const row = (name, r) => { const store = r.storeRead + r.storeWrite; const trackingRate = K01.peakRead * (1 - r.trackingHit / r.trackingRequest); console.log(`${name.padEnd(28)} ${(r.trackingHit / r.trackingRequest).toFixed(4).padStart(14)} ` + `${String(store).padStart(13)} ${(store / WINDOW).toFixed(2).padStart(15)} ${(trackingRate + K01.peakWrite).toFixed(2).padStart(16)}`); }; console.log("state tracking hit store access store access/s requests reaching the store/s"); console.log("---------------------------- --------------- ------------- --------------- ----------------"); row("no scan", alone); row("scan passes through cache", passing); row("scan bypasses cache", bypassing); console.log(`\ncost of the scan passing through the cache = ${(passing.storeRead + passing.storeWrite) - (bypassing.storeRead + bypassing.storeWrite)} extra store accesses`); console.log(`loss in tracking hit ratio = ${((bypassing.trackingHit / bypassing.trackingRequest) - (passing.trackingHit / passing.trackingRequest)).toFixed(4)}`); ``` ``` window 480 s: tracking query 200000, period scan 400000 records, capacity 2000 (K01: peak read 416.67/s, bulk scan 833.33 records/s) pattern request read unique keys repeat hit store access cacheless diff extra touch/req --------------- ------- ------ ------------- ------- ------ ------------- ---------- ------ --------------- tracking query 200000 200000 20477 9.77 0.8976 20483 200000 -179517 0.20 period scan 400000 400000 400000 1.00 0.0000 400000 400000 0 2.00 hot write 200000 60078 40 5000.00 0.3001 181968 200000 -18032 1.12 tracking + scan 600000 600000 420477 1.43 0.1571 505735 600000 -94265 1.69 what happens to the tracking query if the period scan passes through the same cache: state tracking hit store access store access/s requests reaching the store/s ---------------------------- --------------- ------------- --------------- ---------------- no scan 0.8976 20483 42.67 139.90 scan passes through cache 0.4713 505735 1053.61 317.50 scan bypasses cache 0.8976 420483 876.01 139.90 cost of the scan passing through the cache = 85252 extra store accesses loss in tracking hit ratio = 0.4263 ``` All the numbers belong to the **computed** class: they were counted over deterministic streams. #### The Cost of the Anti-Pattern The first row defines the anti-pattern. The tracking query's repeat rate is 9.77: each tracking number is queried about ten times on average within the window. In a cacheless design, the store sees 200,000 accesses; with a cache, 20,483 — a difference of 179,517 accesses, nine-tenths of it unnecessary. In K01's computation, this is the gap in `requests reaching the store/s` between 513.89 and 139.90: **a 3.67x factor**. A cacheless design requires the store sized for that factor; avoiding it costs 0.20 extra touches per request. The second row is the mirror's other face. The period scan reads 400,000 seller-day records, all of them distinct; the repeat rate is exactly 1.00, the hit ratio 0.0000. Store access is equal with and without a cache — a zero difference. With nothing gained, **2.00 extra touches** are paid per request: one lookup and one placement. In this pattern, a cacheless design is not an anti-pattern — it is correct. The third row shows that repetition alone is not enough. In the hot-write pattern, forty keys see 200,000 operations; the repeat rate is 5000, the highest value in this table. Against that, the hit ratio is only **0.3001**. The reason: 140,000 of those operations are writes, each dropping the entry so the next read misses. A high repeat rate does not justify a cache; what justifies it is repetition that **fits between two writes**. #### One Pattern Breaks Another The fourth row and the second table show the same fact from two angles. When the tracking query and the period scan pass through the same cache, the combined hit ratio drops to 0.1571, and the tracking query's own hit ratio drops from 0.8976 to **0.4713**. The scan's 400,000 distinct keys keep entering and leaving the cache, capacity 2000, pushing tracking entries out on the way in. Here the number changes sign. When the scan bypasses the cache, total store access is 420,483; when it passes through, **505,735**: the 85,252-access gap is load the cache **adds**. `requests reaching the store/s` climbs from 139.90 to 317.50. A cache, opened up to the wrong pattern, breaks the very thing it was protecting. The fix is not to remove the cache but to keep the scan stream from passing through it. The last row measures this: once the `period:` namespace is skipped, the tracking hit ratio stays at 0.8976 with no drop at all, and `requests reaching the store/s` returns to 139.90. The scan still reads 400,000 records — those reads were unavoidable — but no longer evicts anyone's working set. The reason K01 put the end-of-day job in its own four-hour window becomes visible in this table. #### The Decision Rule Read together, the four measurements reduce the cache decision to four conditions, all four countable. **The repeat rate must be greater than one.** If the ratio of requests to unique keys in the window is 1.00, there is nothing to gain; the period scan stays in this condition. **Repetition must fit between two writes.** In the hot-write pattern, repetition is 5000, but the hit ratio stays at 0.3001 because the write share per read is high. What matters is not repetition but how many reads an entry gets over its lifetime. **The working set must fit the capacity.** This topic's first lesson measured a cache with capacity 2000 covering a 1000-shipment working set; below that capacity, layers leak overflow into each other. **No other pattern sharing the same cache may evict the working set.** This condition can break on its own even when every other one holds, turning the cache into a net loss: 85,252 extra store accesses. If the four do not hold, a cacheless design is not an anti-pattern; an unmeasured cache decision always is. #### Summary - The tracking query's repeat rate is 9.77, and a cacheless design loads 200,000 accesses onto the store; with a cache, 20,483. In K01's `requests reaching the store/s` line, this is the 3.67x gap between 513.89 and 139.90. - The period scan's repeat rate is 1.00, hit ratio 0.0000, and store access is 400,000 in both designs; with zero gained, 2.00 extra touches are paid per request. - The hot-write pattern's repeat rate is 5000, yet the hit ratio stays at 0.3001: a high repeat rate does not justify a cache — repetition fitting between two writes does. - When the scan passes through the same cache, the tracking hit ratio drops from 0.8976 to 0.4713 and total store access climbs from 420,483 to 505,735; the cache **adds** 85,252 accesses. - The decision comes down to four conditions: repeat rate greater than one, repetition fitting between two writes, the working set fitting the capacity, and no other pattern evicting it. #### Next Step This lesson separated out the patterns where a cache is needed, but one question stayed open. The value returned from the cache need not match the store's value at that moment; how long the two can drift apart was measured once in this topic's second lesson — 105,794 stale reads once invalidation was turned off — and tied to a parameter in the fourth lesson: the five-second upper-bound lifetime was chosen arbitrarily, and never justified. The next lesson builds that justification. How much staleness is acceptable is not an engineering preference but a number that comes out of the data itself: however often a shipment's state changes, a window shorter than that change does not produce a stale response. The next lesson computes the window from that relationship, counts how many stale responses per second are accepted, and measures how far the hit ratio can climb under the same staleness budget. ### Consistency and Stale Data Source: https://academia.sh/en/courses/scaling-the-data/consistency-and-stale-data Computing the staleness window from how fast the data changes: deriving an 8228.57-second peak-hour event interval from a 48-hour transit time and V4, the window saturating at 60 seconds to bring the hit ratio to 0.8968 and requests reaching the store to 140.23, a stale response always falling exactly one event behind, and replacing the earlier lessons' invalidation upper bound with the real 0.0010 share. The previous lesson separated out the patterns where a cache is needed, but one question stayed open. The value returned from the cache need not match the store's value at that moment. This topic's second lesson measured that drift once — 105,794 stale reads once invalidation was turned off — and its fourth lesson tied it to a parameter: the five-second upper-bound lifetime was chosen arbitrarily, and never justified. This lesson builds that justification. How much staleness is acceptable is not a preference but a number that comes out of the data itself: however often a shipment's state changes, a window far shorter than that change produces almost no stale response. #### Two Windows, the Same Response The Introduction to System Design course defined the **replication staleness window**: the time from a write's acknowledgment until every replica has seen that value. The window here is a different one, and its name is the **staleness window**: the longest a cached copy can lag behind the source; a copy served within that time is **stale data**. Their sources differ — one from replication's propagation, the other from an entry's lifetime — but what they produce is the same, called by the same name: a **stale response**. A system can carry both at once, and they add together. Consistency models were defined in that course and not retold here. Only one link is drawn: a cache's **eventual consistency** guarantee is the lifetime given to an entry. Without a lifetime, relying on invalidation alone leaves no convergence guarantee either — this topic's second lesson measured a lost delete leaving a stale entry with no counter showing it. Lifetime is the only mechanism that says an entry will eventually correct itself, and the staleness window is exactly the width of that guarantee. #### Where the Window Comes From A shipment produces seven state events per V4. How much time these events spread over is not in K01's table; the only duration there is the 730-day retention period, and that is a different matter. **B1 (this lesson's assumption): a shipment stays in transit for 48 hours from its first event to delivery.** Rationale: V4's seven states (accepted, in transit, out for delivery, delivered, and intermediate states) spread over a transit time, and a two-day transit time is enough for all of these states to occur. Since V8's peak multiplier speeds up events too, the peak-hour event interval per shipment is $48 \cdot 3600 / 7 / 3$ seconds. Its sensitivity is given in the last measurement. Not added to K01's table. ```js // cache/staleness.mjs — the in-process model of the staleness window. The clock is a counter; // event interval, window, and capacity are parameters. Time is not measured, events and reads are counted. export const K01 = { peakRead: (2_000_000 * 6 / 86_400) * 3, // 416.67 req/s peakWrite: (400_000 * 7 / 86_400) * 3, // 97.22 req/s eventsPerShipment: 7, // V4 peakMultiplier: 3, // V8 }; // B1: a shipment stays in transit for 48 hours from its first event to delivery. At peak hour // the event interval follows from V4 and V8 together. export const TRANSIT_HOURS = 48; export const eventInterval = (transitHours) => (transitHours * 3600) / K01.eventsPerShipment / K01.peakMultiplier; // The read stream and the event calendar are produced together: each shipment gets its own // event time when created. The clock is the read counter; one tick = 1/peakRead seconds. export function run({ reads, workingSet, burst, capacity, windowS, transitHours = TRANSIT_HOURS }) { let seed = 20260730; const rand = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648); const window = windowS * K01.peakRead; // ticks const intervalTicks = eventInterval(transitHours) * K01.peakRead; const box = new Map(); // key -> { version, expires } const truth = new Map(); // key -> { version, nextEvent } let clock = 0, next = 1; const s = { hit: 0, storeRead: 0, stale: 0, lagTotal: 0, maxLag: 0, event: 0 }; const newShipment = () => { const id = next++; truth.set(`track:${id}`, { version: 0, nextEvent: clock + rand() * intervalTicks }); return { id, remaining: burst }; }; const active = Array.from({ length: workingSet }, () => newShipment()); for (let i = 0; i < reads; i++) { clock += 1; const j = Math.floor(rand() * active.length); const key = `track:${active[j].id}`; if ((active[j].remaining -= 1) === 0) active[j] = newShipment(); const t = truth.get(key); // due events are applied while (clock >= t.nextEvent) { t.version += 1; s.event += 1; t.nextEvent += intervalTicks; } const c = box.get(key); if (c !== undefined && c.expires > clock) { s.hit += 1; const lag = t.version - c.version; if (lag > 0) { s.stale += 1; s.lagTotal += lag; s.maxLag = Math.max(s.maxLag, lag); } continue; } s.storeRead += 1; box.delete(key); box.set(key, { version: t.version, expires: clock + window }); if (box.size > capacity) box.delete(box.keys().next().value); } return s; } ``` **Lag** is how many events a stale response falls behind by. The stale-read count says how often the window gets violated; lag says what the user sees once it is. These are two separate questions, and the design decision looks at both at once. ```js // cache/window.mjs — as the window grows: hit ratio, stale responses, and store load import { K01, TRANSIT_HOURS, eventInterval, run } from "./staleness.mjs"; const READS = 200_000, CAPACITY = 2000, WORKING = 1000, BURST = 10; const WINDOW_S = 480; console.log(`B1: transit time ${TRANSIT_HOURS} hours, V4 = ${K01.eventsPerShipment} events, V8 = ${K01.peakMultiplier}`); const INTERVAL = eventInterval(TRANSIT_HOURS); console.log(`-> peak-hour event interval per shipment = ${INTERVAL.toFixed(2)} s`); console.log(`reads ${READS} (${WINDOW_S} s), capacity ${CAPACITY}, concurrent shipments ${WORKING}\n`); console.log("window (s) hit stale reads stale ratio stale responses/s avg lag max lag requests reaching the store/s"); console.log("---------- ------- ------------ ------------ ------------------ -------- -------- ------------------------------"); for (const windowS of [1, 5, 30, 60, 300, 1800]) { const r = run({ reads: READS, workingSet: WORKING, burst: BURST, capacity: CAPACITY, windowS }); const h = r.hit / READS; const staleRatio = r.stale / READS; const behind = K01.peakRead * (1 - h); console.log(`${String(windowS).padStart(10)} ${h.toFixed(4).padStart(7)} ${String(r.stale).padStart(12)} ` + `${staleRatio.toFixed(6).padStart(12)} ${(staleRatio * K01.peakRead).toFixed(3).padStart(18)} ` + `${(r.stale === 0 ? 0 : r.lagTotal / r.stale).toFixed(2).padStart(8)} ${String(r.maxLag).padStart(8)} ` + `${(behind + K01.peakWrite).toFixed(2).padStart(30)}`); } const r5 = run({ reads: READS, workingSet: WORKING, burst: BURST, capacity: CAPACITY, windowS: 5 }); console.log(`\nwindow 5 s: keys that received an event while cached = ${r5.event}`); console.log(`K01 peak write ${K01.peakWrite.toFixed(2)}/s -> total events in the window = ${Math.round(K01.peakWrite * WINDOW_S)}`); console.log(`share of events landing on a cached key = ${(r5.event / (K01.peakWrite * WINDOW_S)).toFixed(4)}`); console.log(`\nstaleness budget -> window (theory: window = event interval x budget)`); console.log("budget (stale response ratio) theoretical window (s) measured stale ratio"); for (const budget of [0.0001, 0.001, 0.01]) { const windowS = INTERVAL * budget; const r = run({ reads: READS, workingSet: WORKING, burst: BURST, capacity: CAPACITY, windowS }); console.log(`${budget.toFixed(4).padStart(30)} ${windowS.toFixed(2).padStart(24)} ${(r.stale / READS).toFixed(6).padStart(21)}`); } console.log(`\nB1's sensitivity (window 60 s):`); console.log("transit time (h) event interval (s) stale ratio stale responses/s hit"); for (const hours of [48, 12, 2]) { const r = run({ reads: READS, workingSet: WORKING, burst: BURST, capacity: CAPACITY, windowS: 60, transitHours: hours }); const o = r.stale / READS; console.log(`${String(hours).padStart(17)} ${eventInterval(hours).toFixed(2).padStart(20)} ${o.toFixed(6).padStart(12)} ` + `${(o * K01.peakRead).toFixed(3).padStart(19)} ${(r.hit / READS).toFixed(4).padStart(7)}`); } ``` ``` B1: transit time 48 hours, V4 = 7 events, V8 = 3 -> peak-hour event interval per shipment = 8228.57 s reads 200000 (480 s), capacity 2000, concurrent shipments 1000 window (s) hit stale reads stale ratio stale responses/s avg lag max lag requests reaching the store/s ---------- ------- ------------ ------------ ------------------ -------- -------- ------------------------------ 1 0.2700 3 0.000015 0.006 1.00 1 401.40 5 0.6313 32 0.000160 0.067 1.00 1 250.85 30 0.8866 221 0.001105 0.460 1.00 1 144.46 60 0.8968 239 0.001195 0.498 1.00 1 140.23 300 0.8968 239 0.001195 0.498 1.00 1 140.23 1800 0.8968 239 0.001195 0.498 1.00 1 140.23 window 5 s: keys that received an event while cached = 46 K01 peak write 97.22/s -> total events in the window = 46667 share of events landing on a cached key = 0.0010 staleness budget -> window (theory: window = event interval x budget) budget (stale response ratio) theoretical window (s) measured stale ratio 0.0001 0.82 0.000005 0.0010 8.23 0.000290 0.0100 82.29 0.001195 B1's sensitivity (window 60 s): transit time (h) event interval (s) stale ratio stale responses/s hit 48 8228.57 0.001195 0.498 0.8968 12 2057.14 0.004180 1.742 0.8968 2 342.86 0.032020 13.342 0.8968 ``` All the numbers belong to the **computed** class: they were counted over a deterministic stream. #### The Window Saturates The first table lands somewhere unexpected. Past 60 seconds, the window changes nothing: hit ratio 0.8968, stale ratio 0.001195, `requests reaching the store/s` 140.23 — three identical rows. The reason: the entry gets **evicted** before it can exhaust its lifetime. A shipment's ten queries spread over roughly twenty-four seconds; once that ends, the shipment leaves the working set and its entry drops under capacity pressure. A window longer than sixty seconds is being given to an entry that will never be read again. This makes the fourth lesson's five-second assumption a poor choice. At five seconds, the hit ratio is 0.6313 and `requests reaching the store/s` is 250.85; at sixty seconds, 0.8968 and 140.23. The five-second window loads **110.62 extra requests per second** onto the store, and in exchange lowers the stale ratio from 0.001195 to only 0.000160: 110.62 requests are paid to gain 0.43 stale responses per second. The second table gives the rule for choosing the window. A **staleness budget** — an accepted stale response ratio — is set, and the window becomes the event interval times the budget. A one-percent budget corresponds to an 82.29-second window; the measured ratio, though, is 0.001195, one-eighth of the budget. Theory gives the upper bound, and eviction pulls it lower still. #### The Magnitude of Staleness The stale-read count alone is not enough. The average-lag and max-lag columns read **1.00** and **1** in all six rows: a stale response never falls more than one event behind. To the user, this means seeing the shipment one state step behind — "at the transit hub" instead of "out for delivery." The state never falls two steps behind, and refreshing the page shows the correct one. This number is the real reason the window is acceptable. 0.498 stale responses per second comes to 239 responses over an eight-minute window, each of them exactly one step behind. The same budget would call for a different decision if the lag were two or three. The acceptance decision also has a layer dimension. This topic's first lesson measured 0.7002 of the hit ratio served at the client layer, and a copy placed there cannot be revoked until it expires. Most stale responses therefore come from the layer that cannot be revoked: there, the window is not an estimate but a commitment. #### A Computation in Place of an Assumption The last two output blocks carry this topic's most important correction. This topic's second lesson used an upper bound: it assumed all state events land on shipments being queried at that moment and noted that this is the direction where invalidation is strongest. B1 puts a computation in its place. The window carries 46,667 state events, and only **46** of them land on a key sitting in the cache: a share of **0.0010**. The rest go either to shipments not being queried or to keys not currently in the cache, and drop no entry at all. This turns the second and third lessons' numbers into upper bounds. The 0.7287 hit ratio cache-aside measured would hold if every event landed on a hot key; at the real share, the hit ratio stays at 0.8968, and the gap between cache-aside and write-through disappears on the read side. The third lesson's write-path analysis is unaffected: its length, coalescing ratio, and loss window are independent of where events land. The last table gives B1's sensitivity and draws the boundary of the decision. As transit time drops from 48 hours to 2 hours, the event interval falls from 8228.57 seconds to 342.86 seconds, and the same sixty-second window raises the stale ratio from 0.001195 to **0.032020**, and the stale response rate to 13.342 per second. The hit ratio does not change. The same cache design, with the same measures, produces twenty-seven times more stale responses for a shipment flow delivered the same day. The window depends on the data, not the system. #### Summary - The staleness window is how far a cached copy can lag behind the source; it differs from the replication staleness window in its source but produces the same stale response, and it is exactly the width of the cache's eventual consistency guarantee. - The window follows from how fast the data changes: with B1 and V4, the peak-hour event interval is 8228.57 seconds, and the window is that interval times the staleness budget — a one-percent budget gives 82.29 seconds. - The window saturates at 60 seconds: hit ratio 0.8968, `requests reaching the store/s` 140.23, stale ratio 0.001195. The fourth lesson's five-second window loaded 110.62 extra requests per second onto the store to gain 0.43 stale responses per second. - A stale response never fell more than one event behind in any run; average and max lag are both 1. The acceptance decision looks at these two numbers together, not at the stale response count. - Only 46 of the window's 46,667 events land on a key in the cache (a share of 0.0010); this turns the second lesson's invalidation assumption into an upper bound. At a 2-hour transit time, the same window raises the stale ratio to 0.032020. #### Next Step This topic met reads in front of the store and bounded their staleness with a window. `requests reaching the store/s` dropped from 513.89 to 140.23, and V9's 0.90 stopped being an assumption and became the outcome of a placement, a strategy, and a window decision. But this design never once questioned one thing: the **shape** of the value placed in the cache. The cache lightened the read path; what it reads is still a model designed for writing. The shipment record is organized so the carrier can write its state event, and because the tracking query has to read that same model, it first joins, then formats, then stores. The cache does not remove this borrowing — it only hides it from requests that do not miss; every missed request still goes to it. The next topic removes the borrowing and scales the read and write paths with **their own models**. Its first question is this: does keeping the same data in two separate structures let the read and write paths scale separately, and if so, past which ratio. ## Read–Write Separation Scaling the read and write paths separately: the decision to scale with distinct models, storing change as a record, deriving and rebuilding read models by folding, and eventual consistency showing up in the user interface as waiting and conflict. ### Command and Query Separation Source: https://academia.sh/en/courses/scaling-the-data/command-and-query-separation Scaling the read and write paths with separate models: the tracking query touching eight records in the single model and one in the projection, the command climbing from three touches to five, comparing the two layouts' node counts under read and write multipliers, and the read store's inability to escape K01's write/read ratio. The previous topic pulled the read path forward by a layer: the cache kept most requests from reaching the store — the `requests reaching the store/s` row fell to 140.23 — and bounded the staleness of the ones that did with a window. But the cache stores a copy of the store's response; every miss still goes to the **same model**, which arranges the shipment record, the state events, and the link between them to preserve the write side's invariants. The read path borrows a model built for writing and pays its cost on every miss. This topic stops borrowing. Its question: does keeping the same data in two separate structures let the read and write paths **scale separately**, and if so, past what ratio. #### Two Separate Separations Two similarly named concepts exist and are not interchangeable. **Command–query separation**, from the Programming Fundamentals course, concerns a method's signature: it either changes state or returns a value, never both. **Command and query responsibility segregation** separates not the method but the **model** — the same data kept in two separate structures. The second pattern was built in the Domain-Driven Design course and measured there: the objects the dashboard query walked through the write model grew with data volume, while the count through the read model stayed constant. That lesson also set a boundary — a query reading a single record by identity walks a single object in the write model too, and the separation gains it nothing. The tracking query looks exactly like that: a single tracking number. The measurement starts by testing whether that resemblance is misleading. #### Access Pattern The decision comes with a pattern: a single read by tracking number, writing the state event that arrives from the carrier, and a periodic batch scan by seller. The first two are this lesson's subject. The tracking response's body was defined in K01 (V5, 480 bytes): state, zone, update time, and the **last three route steps**. Those steps sit not in the shipment record but in the event records, so assembling the response through the write model requires reading the shipment record and all of that shipment's state events — 7 events under K01's V4 assumption. The apparatus is an in-process store model: no real database, disk, or network. Records sit in memory, and every access increments the record count it touches. The measure is **record touches**, not duration, since duration depends on the machine and touch count does not. ```js // separation/store.mjs — in-process store model. No real database: records sit in // memory and every access counts the records it touches. Measured: touches, not duration. const POINTS = ["34", "06", "35", "01", "16"]; export function createStore() { return { shipments: new Map(), events: new Map(), projections: new Map(), counters: { reads: 0, writes: 0 } }; } function buildResponse(g, events) { return { id: g.id, state: g.state, zone: g.zone, at: g.at, lastSteps: events.slice(-3).map((o) => o.point) }; } export function seed(d, count, eventsPerShipment) { let x = 7; const next = () => (x = (x * 48271 + 11) % 2147483647); for (let i = 1; i <= count; i += 1) { const id = `G${i}`; const g = { id, zone: POINTS[next() % POINTS.length], state: "in-transit", at: eventsPerShipment }; const events = Array.from({ length: eventsPerShipment }, (_, k) => ({ id, seq: k + 1, point: POINTS[next() % POINTS.length] })); d.shipments.set(id, g); d.events.set(id, events); d.projections.set(id, buildResponse(g, events)); } return d.shipments.size; } // Single model: the tracking response is assembled from the shipment record and its events. export function trackSingleModel(d, id) { const g = d.shipments.get(id); d.counters.reads += 1; const events = d.events.get(id); d.counters.reads += events.length; return buildResponse(g, events); } // Split model: the tracking response is read from a single projection row waiting ready. export function trackFromProjection(d, id) { d.counters.reads += 1; return d.projections.get(id); } // Command: an event is appended, the shipment record is updated with read-modify-write; // in the split layout the projection row is also refreshed with read-modify-write. export function command(d, id, point, writeProjection) { const events = d.events.get(id); events.push({ id, seq: events.length + 1, point }); d.counters.writes += 1; const g = d.shipments.get(id); d.counters.reads += 1; g.at = events.length; d.counters.writes += 1; if (writeProjection !== true) return; const old = d.projections.get(id); d.counters.reads += 1; d.projections.set(id, { ...old, at: g.at, lastSteps: events.slice(-3).map((o) => o.point) }); d.counters.writes += 1; } ``` **Projection**, as defined in the Domain-Driven Design course, is a read model derived from the changes the write side produces. Here the projection row is the tracking response itself. #### Measurement and Computed Values The measurement runs in two steps: the model runs and the touches per query and per command are counted — deterministic numbers — then those numbers are applied to K01's rates, and a node count comes out. That count needs a capacity. **OY1 — the record touches per second a store node sustains: 100.** This is this topic's own assumption, not added to K01's table; its rationale is that a division needs a unit, and the number's absolute value is not a measurement. Its sensitivity has a special form: taking 200 halves every node count, but the two layouts' **ratio** stays unchanged, since capacity is the same divisor on both sides. ```js // separation/measure.mjs — the layouts' touches per query and command are measured from the // model, then applied to K01's rates. All numbers are deterministic; a computed value in class. import { createStore, seed, trackSingleModel, trackFromProjection, command } from "./store.mjs"; const V4 = 7; // K01 assumption: state events per shipment const READ = 41.67; // K01 computed value: reads behind cache/s const WRITE = 97.22; // K01 computed value: peak write requests/s const OY1 = 100; // this topic's assumption: record touches/s a node sustains const N = 200; const d = createStore(); seed(d, N, V4); const ids = [...d.shipments.keys()]; d.counters.reads = 0; d.counters.writes = 0; const singleResponses = ids.map((id) => trackSingleModel(d, id)); const singleQuery = d.counters.reads / N; d.counters.reads = 0; d.counters.writes = 0; const projResponses = ids.map((id) => trackFromProjection(d, id)); const projQuery = d.counters.reads / N; const equal = JSON.stringify(singleResponses) === JSON.stringify(projResponses); d.counters.reads = 0; d.counters.writes = 0; for (const id of ids) command(d, id, "16", false); const singleCommand = { reads: d.counters.reads / N, writes: d.counters.writes / N }; d.counters.reads = 0; d.counters.writes = 0; for (const id of ids) command(d, id, "01", true); const splitCommand = { reads: d.counters.reads / N, writes: d.counters.writes / N }; const b = (x, n = 2) => x.toFixed(n); console.log(`model: ${N} shipments, ${V4} events per shipment (K01 V4)`); console.log(`tracking query / single model = ${singleQuery} record touches`); console.log(`tracking query / from projection = ${projQuery} record touches, two responses equal = ${equal}`); console.log(`command / single model = ${singleCommand.reads + singleCommand.writes} touches ` + `(${singleCommand.reads} reads, ${singleCommand.writes} writes)`); console.log(`command / split model = ${splitCommand.reads + splitCommand.writes} touches ` + `(${splitCommand.reads} reads, ${splitCommand.writes} writes)`); const single = (r, w) => READ * r * singleQuery + WRITE * w * (singleCommand.reads + singleCommand.writes); const splitWrite = (w) => WRITE * w * (singleCommand.reads + singleCommand.writes); const splitRead = (r, w) => READ * r * projQuery + WRITE * w * (splitCommand.reads + splitCommand.writes - singleCommand.reads - singleCommand.writes); const nodes = (load) => Math.ceil(load / OY1); console.log(`\nK01: reads behind cache ${READ}/s, peak writes ${WRITE}/s, ratio ${b(WRITE / READ)}`); console.log(`node capacity OY1 = ${OY1} touches/s (this topic's assumption)\n`); console.log(`${"read x".padStart(8)}${"write x".padStart(8)}${"single touches/s".padStart(18)}` + `${"single nodes".padStart(14)}${"split w+r touches/s".padStart(21)}${"split nodes".padStart(13)}`); for (const [r, w] of [[1, 1], [2, 1], [4, 1], [1, 2], [1, 4], [4, 4]]) { const t = single(r, w), sw = splitWrite(w), sr = splitRead(r, w); console.log(`${String(r).padStart(8)}${String(w).padStart(8)}${b(t).padStart(18)}` + `${String(nodes(t)).padStart(14)}${`${b(sw)} + ${b(sr)}`.padStart(21)}` + `${`${nodes(sw)} + ${nodes(sr)} = ${nodes(sw) + nodes(sr)}`.padStart(13)}`); } const gain = (r, w) => nodes(single(r, w)) - nodes(splitWrite(w)) - nodes(splitRead(r, w)); let firstLoss = null; for (let w = 1; w <= 20; w += 1) if (gain(1, w) < 0) { firstLoss = w; break; } console.log(`\nnode gain from the separation: baseline ${gain(1, 1)}, at read x4 ${gain(4, 1)}, ` + `at write x4 ${gain(1, 4)}`); console.log(`smallest write multiplier at which the separation raises the node count = ${firstLoss}`); const projUpdate = splitCommand.reads + splitCommand.writes - singleCommand.reads - singleCommand.writes; console.log(`\nrequests the read store sees: query ${b(READ)}/s, projection update ` + `${b(WRITE)}/s -> write/read ratio ${b(WRITE / READ)} (K01's ratio)`); console.log(`touches the read store sees: ${b(READ * projQuery)} + ${b(WRITE * projUpdate)} ` + `-> ratio ${b((WRITE * projUpdate) / (READ * projQuery))}`); ``` ``` model: 200 shipments, 7 events per shipment (K01 V4) tracking query / single model = 8 record touches tracking query / from projection = 1 record touches, two responses equal = true command / single model = 3 touches (1 reads, 2 writes) command / split model = 5 touches (2 reads, 3 writes) K01: reads behind cache 41.67/s, peak writes 97.22/s, ratio 2.33 node capacity OY1 = 100 touches/s (this topic's assumption) read x write x single touches/s single nodes split w+r touches/s split nodes 1 1 625.02 7 291.66 + 236.11 3 + 3 = 6 2 1 958.38 10 291.66 + 277.78 3 + 3 = 6 4 1 1625.10 17 291.66 + 361.12 3 + 4 = 7 1 2 916.68 10 583.32 + 430.55 6 + 5 = 11 1 4 1500.00 15 1166.64 + 819.43 12 + 9 = 21 4 4 2500.08 26 1166.64 + 944.44 12 + 10 = 22 node gain from the separation: baseline 1, at read x4 10, at write x4 -6 smallest write multiplier at which the separation raises the node count = 2 requests the read store sees: query 41.67/s, projection update 97.22/s -> write/read ratio 2.33 (K01's ratio) touches the read store sees: 41.67 + 194.44 -> ratio 4.67 ``` #### Reading the Numbers The first three lines show the resemblance is misleading. The tracking query arrives with a single identity, but through the write model it touches **8 records**: one shipment record and seven event records. Through the projection it touches 1, and the two paths' responses are equal. The separation brings no gain not when a query arrives with a single identity, but when **the response comes from a single record**; the tracking response does not. The command rows carry the cost. In the single model, a state event costs 3 touches: the event is appended, the shipment record read and written. In the split layout the projection row must also be read and written, and the command climbs to 5 touches — the write path grows to **1.67 times**. The node table compares the two paths at K01's rates. At baseline, the single model wants 7 nodes at 625.02 touches/s; the split layout wants 6, with 3 write and 3 read nodes — a gain of one node, not enough to justify the separation alone. The real difference is in the direction of growth. At read x4, the single model climbs to 17 nodes while the split layout stays at 7, a gain of 10, because read growth in the split layout only grows the projection queries, each 1 touch. At write x4 the table reverses: 15 nodes against 21, a loss of 6, and the threshold arrives early — the separation is already costly once the write multiplier reaches **2**. The separation is a bet that reads will grow faster than writes. #### The Ratio Does Not Disappear, It Moves The last two lines measure the part of the pattern most often misread. K01 had found the write/read ratio at the store to be 2.33; a system that looks read-heavy was write-heavy where the store saw it. The separation does not escape that ratio: the read store sees 41.67 queries a second and 97.22 projection updates, so in request terms the ratio is still **2.33**. In touches it sharpens further — 41.67 against 194.44, a ratio of **4.67**. The store named "read model" is twice as write-heavy as the original. The separation gives not a corrected ratio but a separated **cost per touch**: read-path touches drop from 8 to 1, write-path touches climb from 3 to 5. K01's 138.89 requests in the `requests reaching the store/s` row do not disappear — they split across two stores, 97.22 to the write store and 41.67 to the read store. Total touches drop from 625.02 to 527.77, a 1.18× factor. All of these numbers belong to the computed-value class; the only assumption is OY1, and it only does the dividing. #### The New Question the Separation Raises In the split layout, the projection row was updated together with the command, so it stayed fresh throughout — making the model look better than it should. Writing to two stores at once requires them to sit inside a single consistency boundary; to scale separately they cannot share that boundary, so the projection is updated **after** the command, opening a window in between. How long that window stays open, and why the read store needs to be rebuildable, depend on what the projection is derived from. #### Summary - Command–query separation is about a method's signature; command and query responsibility segregation is about the model. The second was built in the Domain-Driven Design course and is treated here as a scaling tool. - Though the tracking query arrives with a single identity, its response does not come from a single record: 8 record touches through the write model, 1 through the projection, and the two responses are equal. - The cost sits in the write path: the command climbs from 3 touches to 5, a 1.67× factor. - With OY1 = 100 touches/s capacity, nodes at baseline drop from 7 to 6; at read x4 it is 17 against 7, at write x4 it is 15 against 21. The separation turns to a loss once the write multiplier reaches 2. - K01's 2.33 does not disappear, it moves: the read store's request ratio is also 2.33, its touch ratio is 4.67. The separation changes not the ratio but the cost per touch. #### Next Step Updating the projection together with the command made this lesson's measurement easier, but it defeats the point of the separation: two stores in the same consistency boundary cannot scale separately. If the projection is updated after the command, at its own pace, what it is derived from must be settled. The shipment record holds only the current state, so it cannot be the source — a projection row has to know which change brought the record to that state. The next lesson takes up storing the change itself as a record: which read an append-only log removes from the write path, the fact that K01 never asked how many operations one write request turns into at the store, and what the event becoming the source-of-truth record adds to daily data growth and stored data rows. ### Event-Sourced Design Source: https://academia.sh/en/courses/scaling-the-data/event-sourced-design The scaling cost of storing change as a record: the append-only log removing the read-modify-write read from the write path, store operations per request dropping from 4.50 to 3.10, the source event's extra fields raising daily growth from 976 to 1,168 MB and stored data from 712.48 to 852.64 GB, and the periodic scan climbing from 10.80 to 35.04 GB. The previous lesson updated the projection together with the command and undercut the separation's point: two stores in the same consistency boundary cannot scale separately. If the projection is updated at its own pace, what it is derived from must be settled, and the shipment record is unsuited to that — it holds only the current state, not the change that produced it. Refreshing the projection needs the change itself. **Event sourcing** does exactly that: what is stored is not the state but the sequence of events that produced it. The pattern was built in the Domain-Driven Design course and defended there by answerability — questions not in the state table's schema come out of the event sequence. This lesson's question is different: what the pattern does to the data layer's **scaling numbers**. Does the log cheapen the write path, how much does it grow stored data, and which access pattern does it make expensive. #### The Read Hidden in the Write Path K01 had found the request rate reaching the store to be 138.89: 41.67 read requests and 97.22 write requests a second, ratio 2.33. That number counts **requests**. It never asked how many operations one write request turns into at the store. In a persistence that updates state in place, a state event costs three operations: the event is written for audit, the shipment record is read, the updated record is written back. The middle read is not a request, so it does not appear in K01's table — but it reaches the store all the same. An append-only log has no such read: a write only appends to the end of the sequence and does not need to know the prior state. Being the source-of-truth record has a cost. K01's event record (V6, 220 bytes) is an audit line: tracking number, state, route step, timestamp. When that same record becomes the sole source of truth it carries extra fields — event type, sequence number in the series, schema version, and an idempotency key. Without a sequence number the fold order is ambiguous; without a schema version an old-format event cannot be read; without an idempotency key a resent event applies twice. **OY2 — the source event record's extra load: 60 bytes.** This is this topic's own assumption and is not added to K01's table. Its rationale is the four fields above. Its sensitivity is given in the measurement's last lines: at zero overhead K01's numbers come out unchanged; at 120 bytes stored data grows once more. #### Apparatus Both persistence forms present the same interface and see the same events. It is an in-process model; there is no disk, network, or duration. Two things are counted: the **store operation** each access produces, and the **stored bytes** each record adds. The byte count comes not from JSON size but from K01's V6 and V7 assumptions; the goal is to carry K01's computed value forward, not the model's own shape. ```js // separation/persistence.mjs — in-process model of two persistence forms. Same interface, // same events; each access counts a store operation, each record adds K01's assumed bytes. export const V4 = 7, V6 = 220, V7 = 900; // K01 assumptions export const OY2 = 60; // this topic's assumption: source event's extra fields function createCounters() { return { reads: 0, writes: 0, bytes: 0 }; } // A: persistence that updates state in place; events are also written for audit. export function updateInPlace() { const records = new Map(), audit = new Map(); const s = createCounters(); return { name: "update in place", counters: s, create(id, zone) { records.set(id, { id, zone, state: "accepted", lastSteps: [] }); audit.set(id, []); s.writes += 1; s.bytes += V7; }, write(id, point) { audit.get(id).push({ id, point }); s.writes += 1; s.bytes += V6; const k = records.get(id); s.reads += 1; // read-modify-write: the existing row is read first k.state = "in-transit"; s.writes += 1; // the updated row is written back }, read(id) { // tracking response: row + events for the last three steps const k = records.get(id); s.reads += 1; const d = audit.get(id); s.reads += d.length; return { id: k.id, zone: k.zone, state: k.state, lastSteps: d.slice(-3).map((o) => o.point) }; }, scan(ids) { // periodic batch scan: only the shipment row is read let bytes = 0; for (const id of ids) { records.get(id); s.reads += 1; bytes += V7; } return bytes; }, }; } // B: append-only event log only; state is derived by folding, no in-place update exists. export function eventLog() { const log = new Map(); const s = createCounters(); const fold = (events) => { const first = events[0]; let state = "accepted"; const steps = []; for (const o of events.slice(1)) { state = "in-transit"; steps.push(o.point); } return { id: first.id, zone: first.zone, state, lastSteps: steps.slice(-3) }; }; return { name: "event log", counters: s, create(id, zone) { log.set(id, [{ type: "shipment-created", id, zone }]); s.writes += 1; s.bytes += V7 + OY2; }, write(id, point) { log.get(id).push({ type: "transferred", id, point }); s.writes += 1; s.bytes += V6 + OY2; // no read on the write path }, read(id) { const events = log.get(id); s.reads += events.length; return fold(events); }, scan(ids) { let bytes = 0; for (const id of ids) { const events = log.get(id); s.reads += events.length; bytes += V7 + OY2 + (events.length - 1) * (V6 + OY2); } return bytes; }, }; } ``` The fold function pulls the last three route steps from the sequence itself; the in-place persistence reads the same three steps from the audit events. Whether the two paths produce the same response is tested in the measurement. #### Measurement ```js // separation/measure.mjs — the same events are written to both persistence forms; store // operations per request and stored bytes per shipment are measured, then applied to K01's rates. import { updateInPlace, eventLog, V4, V6, V7, OY2 } from "./persistence.mjs"; const POINTS = ["34", "06", "35", "01", "16"]; const N = 200; function run(store) { let x = 7; const next = () => (x = (x * 48271 + 11) % 2147483647); const ids = Array.from({ length: N }, (_, i) => `G${i + 1}`); for (const id of ids) { store.create(id, POINTS[next() % POINTS.length]); for (let k = 0; k < V4; k += 1) store.write(id, POINTS[next() % POINTS.length]); } const setup = { ...store.counters }; store.counters.reads = 0; store.counters.writes = 0; const responses = ids.map((id) => store.read(id)); const query = store.counters.reads / N; store.counters.reads = 0; const scanBytes = store.scan(ids) / N; const scan = { ops: store.counters.reads / N, bytes: scanBytes }; store.counters.reads = 0; store.counters.writes = 0; for (const id of ids) store.write(id, "16"); const command = { reads: store.counters.reads / N, writes: store.counters.writes / N }; return { name: store.name, command, query, scan, shipmentBytes: setup.bytes / N, responses }; } const A = run(updateInPlace()), B = run(eventLog()); const b = (x, n = 2) => x.toFixed(n); const p = (x, n) => String(x).padStart(n); console.log(`model: ${N} shipments, ${V4} state events per shipment (K01 V4)`); console.log(`same tracking response from both persistence forms = ${JSON.stringify(A.responses) === JSON.stringify(B.responses)}\n`); console.log(`${"persistence".padEnd(16)}${"command ops".padStart(14)}${"query ops".padStart(14)}` + `${"scan ops".padStart(15)}${"shipment bytes".padStart(16)}`); for (const r of [A, B]) console.log(`${r.name.padEnd(16)}${`${r.command.reads}r + ${r.command.writes}w`.padStart(14)}` + `${p(r.query, 14)}${p(r.scan.ops, 15)}${p(r.shipmentBytes, 16)}`); // K01 rates and assumptions const READ = 41.67, WRITE = 97.22, DAILY_SHIPMENTS = 400_000, RETENTION = 730, BATCH_DAYS = 30, WINDOW = 4 * 3600; console.log(`\nK01: requests reaching the store/s = ${b(READ + WRITE)} (${READ} read + ${WRITE} write)` + `, ratio ${b(WRITE / READ)}`); console.log(`${"persistence".padEnd(16)}${"read ops/s".padStart(16)}${"write ops/s".padStart(16)}` + `${"total ops/s".padStart(16)}${"ops/request".padStart(13)}`); for (const r of [A, B]) { const o = READ * r.query + WRITE * r.command.reads, w = WRITE * r.command.writes; console.log(`${r.name.padEnd(16)}${b(o).padStart(16)}${b(w).padStart(16)}${b(o + w).padStart(16)}` + `${b((o + w) / (READ + WRITE)).padStart(13)}`); } console.log(`\n${"persistence".padEnd(16)}${"daily growth MB".padStart(17)}${"stored GB".padStart(13)}` + `${"batch read GB".padStart(17)}${"batch Mbit/s".padStart(14)}`); for (const r of [A, B]) { const daily = (DAILY_SHIPMENTS * r.shipmentBytes) / 1e6; const batch = (BATCH_DAYS * DAILY_SHIPMENTS * r.scan.bytes) / 1e9; console.log(`${r.name.padEnd(16)}${b(daily).padStart(17)}${b((daily * RETENTION) / 1000).padStart(13)}` + `${b(batch).padStart(17)}${b((batch * 1e9 * 8) / (WINDOW * 1e6)).padStart(14)}`); } const factor = (f) => f(B) / f(A); console.log(`\nlog's factors: daily growth x${b(factor((r) => r.shipmentBytes))}` + `, batch read x${b(factor((r) => r.scan.bytes))}, command ops x${b(factor((r) => r.command.reads + r.command.writes))}`); for (const extra of [0, OY2, 2 * OY2]) { const daily = (DAILY_SHIPMENTS * (V7 + extra + V4 * (V6 + extra))) / 1e6; console.log(`OY2 = ${p(extra, 3)} bytes -> daily growth ${b(daily)} MB` + `, stored ${b((daily * RETENTION) / 1000)} GB`); } ``` ``` model: 200 shipments, 7 state events per shipment (K01 V4) same tracking response from both persistence forms = true persistence command ops query ops scan ops shipment bytes update in place 1r + 2w 8 1 2440 event log 0r + 1w 8 8 2920 K01: requests reaching the store/s = 138.89 (41.67 read + 97.22 write), ratio 2.33 persistence read ops/s write ops/s total ops/s ops/request update in place 430.58 194.44 625.02 4.50 event log 333.36 97.22 430.58 3.10 persistence daily growth MB stored GB batch read GB batch Mbit/s update in place 976.00 712.48 10.80 6.00 event log 1168.00 852.64 35.04 19.47 log's factors: daily growth x1.20, batch read x3.24, command ops x0.33 OY2 = 0 bytes -> daily growth 976.00 MB, stored 712.48 GB OY2 = 60 bytes -> daily growth 1168.00 MB, stored 852.64 GB OY2 = 120 bytes -> daily growth 1360.00 MB, stored 992.80 GB ``` #### Reading the Numbers The first check is that the `update in place` row reproduces K01 exactly: 2,440 bytes per shipment, 976 MB a day, 712.48 GB over 730 days, 10.80 GB and 6.00 Mbit/s in the periodic scan; K01's computed value had assumed this persistence form, so a mismatch would invalidate the comparison. The second check is that both persistence forms give the same tracking response. The write path is where the log wins. A state event costs 3 operations in the `update in place` layout (1 read, 2 writes), 1 in the log — command cost falls to **0.33 times**. At K01's rates, 194.44 write operations a second drop to 97.22, and the 97.22 hidden reads the table never counted drop to zero. Store operations per request fall from 4.50 to 3.10: K01's 138.89 requests were producing 625.02 operations; in the log they produce 430.58. Storage is where the log loses. Bytes per shipment climb from 2,440 to 2,920; daily growth from 976 MB to **1,168 MB**, stored data from 712.48 GB to **852.64 GB**. The 1.20× growth comes entirely from OY2 — the sensitivity lines confirm it: at zero overhead the log's storage numbers match K01's, at 120 bytes stored data reaches 992.80 GB. The real finding is that **the event sequence was already in K01's table.** 616 MB of the daily 976 MB is event records; the log does not multiply stored data, it only adds the fields that come with being the source. The scan path is where the log breaks. The periodic batch scan walks a seller's thirty days and reads a single row per shipment in the `update in place` layout; in the log the same information needs eight events read and folded, so scanned bytes climb from 10.80 GB to **35.04 GB** and the four-hour window's bandwidth from 6.00 Mbit/s to **19.47 Mbit/s**. K01 had already flagged this row as its largest source line, 3.75 times peak read throughput; with the log it reaches 12.17 times. A decision made for one access pattern can make another more than three times as expensive. #### Where the Snapshot Falls Short The Domain-Driven Design course had put a remedy on the log's read cost: a snapshot starts the rebuild from a point rather than the start of the sequence, under a rule that says the number of events read does not drop unless the snapshot interval falls below the average history length. In this system the average history length is 8 (one create event and V4 state events), so the interval needs to be below 8; at an interval of 1, what is stored is a state row updated after every event — the `update in place` layout itself. In a domain with a short history, a snapshot produces no separate option; only a few steps sit between the two ends. This course's question is therefore not the snapshot but the projection: not a query reading the log, but a separate read model derived from it. The log's own limit belongs here too. Because a written event does not change, an append-only sequence is an obstacle when a record must be fully deleted; K01's 730-day retention assumption (V12) is therefore a contractual decision, not a technical one. #### Summary - The log removes the hidden read on the write path: a state event goes from 3 operations to 1, store operations per request from 4.50 to 3.10; K01's 138.89 requests' operation cost falls from 625.02 to 430.58. - Being the source costs OY2 = 60 bytes: daily growth 976 → 1,168 MB, stored data 712.48 → 852.64 GB, a 1.20× factor. At zero overhead K01's numbers come out unchanged. - The event sequence was already in K01's table — 616 MB of the daily 976 MB is event records; the log does not multiply storage, it only adds sequence, type, schema version, and idempotency fields. - The pattern that breaks is the periodic scan: 8 events are read per shipment instead of 1, scanned data 10.80 → 35.04 GB, window bandwidth 6.00 → 19.47 Mbit/s. - Because the average history length is 8, a snapshot produces no separate option; at an interval of 1 the snapshot is already the state row itself. #### Next Step The log cheapened the write path and made the read path expensive; the gap between the two numbers closes only if the read path is moved outside the log. The previous lesson's projection did that, but it was updated together with the command and so could not scale separately. The log now supplies the input for deriving it: a projection can read the log at its own pace, fall behind, and be rebuilt from scratch. The next lesson takes up that derivation — the extra data the projection stores, the number of events read to rebuild it, the condition under which new events flowing in during a rebuild are caught, and how long a rebuild takes when the schema changes. ### Deriving Read Models Source: https://academia.sh/en/courses/scaling-the-data/deriving-read-models Building the projection from the event log by folding, and rebuilding it from scratch: incremental maintenance producing the same rows as a rebuild, the projection adding 192 MB to daily growth and 140.16 GB to stored data, the open window reading 146 times fewer events than full retention, and a rebuild's 2.11-hour lag window against side-by-side rebuilding's 140.16 GB temporary space cost. The previous lesson made the log the source of the write path and showed its cost in two places: stored data grew to 1.20 times, the periodic scan to 3.24 times. The second number is the log's real limit — if a query's answer comes from folding eight events, the read path depends on the log's length. The first lesson had already seen the fix: keep the row the query wants ready. Now where that row is derived from is settled: the log. This lesson takes up the derivation itself. **Folding** is applying the log's events to a read model; its product is the **projection**. Three things are measured: how much space the projection takes, how many events a from-scratch rebuild reads, and whether new events flowing in during a rebuild are caught. #### One Log, Many Projections A projection is a fold function: it takes the event sequence and produces rows. Because folding is pure, the same log always gives the same rows, which makes a projection **disposable**. Being disposable is decisive for scaling — when a read model breaks, its schema changes, or it needs to be copied to a new node, it is built from the log, not restored from a backup. More than one projection can be derived from the same log, each shaped to its own access pattern. This topic has two: the **tracking projection**, serving a single read by tracking number, and the **point counter**, holding the count of shipments pending at each transfer point. The first's row count depends on the number of shipments, the second's on the number of transfer points; both feed from the same events. Storing precomputed results inside the store is taken up separately in this course's Data Distribution topic; the difference here is that the projection derives from the log, not the store. ```js // derive/projection.mjs — two read models derived from the event log. Each projection is a // fold function; counters keep events read and rows written separate. In-process model. export function trackingProjection(version) { const rows = new Map(); const s = { events: 0, writes: 0 }; return { name: `tracking v${version}`, rows, counters: s, apply(o) { s.events += 1; if (o.type === "shipment-created") { const r = { id: o.id, zone: o.zone, state: "accepted", lastSteps: [] }; if (version === 2) r.transfers = 0; rows.set(o.id, r); s.writes += 1; return; } const r = rows.get(o.id); if (o.type === "delivered") { r.state = "delivered"; s.writes += 1; return; } r.state = "in-transit"; r.lastSteps = [...r.lastSteps, o.point].slice(-3); if (version === 2) r.transfers += 1; s.writes += 1; }, view: () => JSON.stringify([...rows.entries()].sort()), }; } // Second read model: a counter of pending shipments by transfer point. Derived from the same // log, but its row count depends on the number of points, not the number of shipments. export function pointCounter() { const rows = new Map(), location = new Map(); const s = { events: 0, writes: 0 }; return { name: "point counter", rows, counters: s, apply(o) { s.events += 1; if (o.type === "shipment-created") return; const prior = location.get(o.id); if (prior !== undefined) { rows.set(prior, rows.get(prior) - 1); s.writes += 1; } if (o.type === "delivered") { location.delete(o.id); return; } rows.set(o.point, (rows.get(o.point) ?? 0) + 1); s.writes += 1; location.set(o.id, o.point); }, view: () => JSON.stringify([...rows.entries()].sort()), }; } // Log generator: every shipment gets one create and V4 transfer events; closed shipments // also carry a delivered event. The generator is deterministic: same input, same log. export function generateLog(count, eventsPerShipment) { const POINTS = ["34", "06", "35", "01", "16"]; let x = 7; const next = () => (x = (x * 48271 + 11) % 2147483647); const log = [], open = new Set(); for (let i = 1; i <= count; i += 1) { const id = `G${i}`; log.push({ type: "shipment-created", id, zone: POINTS[next() % POINTS.length] }); for (let k = 0; k < eventsPerShipment; k += 1) log.push({ type: "transferred", id, point: POINTS[next() % POINTS.length] }); if (i % 5 === 0) open.add(id); else log.push({ type: "delivered", id }); } return { log, open }; } ``` The `version` parameter models a schema change: the second version adds a transfer counter to the row. The only way to apply the new field to old rows is to rebuild the projection from the log. #### Measurement The measurement has two layers. The model layer gives the derivation's correctness and the writes per event; those are deterministic numbers. The computed-value layer applies those numbers to K01's volumes and needs two new assumptions. **OY3 — how long a shipment stays open: 5 days.** Its rationale is that K01's 730-day retention assumption (V12) is for contract disputes, not for a shipment's flow; the time from acceptance to delivery is shorter than that. Its sensitivity: at 10 days, the open window's event count and rebuild time double. **OY4 — the ratio of a rebuild's event-consumption speed to the live flow: 20.** Its rationale is that a rebuild does batch reads and carries no per-request network round trip or validation. A ratio is chosen, not an absolute speed, so the number is not a measurement. Its sensitivity is given in the table with three values, and it shows a threshold. ```js // derive/measure.mjs — the projection is built by incremental maintenance and by a from-scratch // rebuild; the two are compared, then applied to K01's volumes and this topic's two assumptions. import { trackingProjection, pointCounter, generateLog } from "./projection.mjs"; const V4 = 7, V5 = 480, V12 = 730, DAILY_SHIPMENTS = 400_000, V8 = 3, DAY = 86_400; const OY3 = 5, OY4 = 20; // this topic's assumptions: days open, rebuild speed multiplier const N = 200; const { log, open } = generateLog(N, V4); const incremental = trackingProjection(1); for (const o of log) incremental.apply(o); const full = trackingProjection(1); for (const o of log) full.apply(o); const activeLog = log.filter((o) => open.has(o.id)); const active = trackingProjection(1); for (const o of activeLog) active.apply(o); const v2 = trackingProjection(2); for (const o of log) v2.apply(o); const points = pointCounter(); for (const o of log) points.apply(o); const b = (x, n = 2) => x.toFixed(n); const p = (x, n) => String(x).padStart(n); console.log(`log: ${N} shipments, ${log.length} events, open shipments ${open.size}` + `, active log ${activeLog.length} events`); console.log(`incremental maintenance matches from-scratch rebuild = ${incremental.view() === full.view()}`); console.log(`active rebuild builds the same rows for open shipments = ` + `${[...open].every((id) => JSON.stringify(active.rows.get(id)) === JSON.stringify(full.rows.get(id)))}`); console.log(`v2 row has the new field = ${"transfers" in v2.rows.get("G5")}` + `, v1 row = ${"transfers" in full.rows.get("G5")}\n`); console.log(`${"projection".padEnd(21)}${"events read".padStart(13)}${"rows written".padStart(15)}` + `${"row count".padStart(14)}${"writes per event".padStart(19)}`); for (const [name, r] of [["tracking v1 full", full], ["tracking v1 active", active], ["tracking v2 full", v2], ["point counter", points]]) console.log(`${name.padEnd(21)}${p(r.counters.events, 13)}${p(r.counters.writes, 15)}` + `${p(r.rows.size, 14)}${b(r.counters.writes / r.counters.events).padStart(19)}`); // Back to K01: the data the projection stores const dailyProjection = (DAILY_SHIPMENTS * V5) / 1e6; console.log(`\nprojection row ${V5} bytes (K01 V5), ${DAILY_SHIPMENTS} rows a day`); console.log(`daily growth the projection adds = ${b(dailyProjection)} MB` + `, ${b((dailyProjection * V12) / 1000)} GB over ${V12} days`); for (const [name, day, stored] of [["K01", 976, 712.48], ["event log (previous lesson)", 1168, 852.64]]) console.log(`${name.padEnd(29)} ${b(day)} MB/day, ${b(stored)} GB -> with projection ` + `${b(day + dailyProjection)} MB/day, ${b(stored + (dailyProjection * V12) / 1000)} GB`); // Rebuild: events read and catching up const dailyEvents = DAILY_SHIPMENTS * (1 + V4); const peakFlow = (dailyEvents / DAY) * V8; const fullEvents = dailyEvents * V12, activeEvents = dailyEvents * OY3; console.log(`\npeak event flow = ${b(peakFlow)} events/s (K01: ${dailyEvents} events a day, peak multiplier ${V8})`); console.log(`${"window".padEnd(27)}${"events read".padStart(15)}${"factor".padStart(8)}`); console.log(`${`full retention (${V12} days)`.padEnd(27)}${p(fullEvents, 15)}${b(fullEvents / activeEvents).padStart(8)}`); console.log(`${`open window (OY3=${OY3})`.padEnd(27)}${p(activeEvents, 15)}${"1.00".padStart(8)}`); console.log(`\n${"OY4".padStart(5)}${"consume events/s".padStart(18)}${"net events/s".padStart(14)}` + `${"open window hours".padStart(19)}${"full window days".padStart(17)}`); for (const k of [OY4, OY4 / 2, 1.5, 1]) { const consume = k * peakFlow, net = consume - peakFlow; const hours = net <= 0 ? null : activeEvents / net / 3600; console.log(`${b(k, 1).padStart(5)}${b(consume).padStart(18)}${b(net).padStart(14)}` + `${(hours === null ? "cannot catch up" : b(hours)).padStart(19)}` + `${(hours === null ? "cannot catch up" : b(fullEvents / net / 86400)).padStart(17)}`); } console.log(`\nlag window rebuilding in place = ` + `${b(activeEvents / ((OY4 - 1) * peakFlow) / 3600)} hours; rebuilding side by side, 0`); console.log(`side-by-side rebuild's temporary space cost = ${b((dailyProjection * V12) / 1000)} GB` + `, peak storage ${b(852.64 + 2 * (dailyProjection * V12) / 1000)} GB`); ``` ``` log: 200 shipments, 1760 events, open shipments 40, active log 320 events incremental maintenance matches from-scratch rebuild = true active rebuild builds the same rows for open shipments = true v2 row has the new field = true, v1 row = false projection events read rows written row count writes per event tracking v1 full 1760 1760 200 1.00 tracking v1 active 320 320 40 1.00 tracking v2 full 1760 1760 200 1.00 point counter 1760 2760 5 1.57 projection row 480 bytes (K01 V5), 400000 rows a day daily growth the projection adds = 192.00 MB, 140.16 GB over 730 days K01 976.00 MB/day, 712.48 GB -> with projection 1168.00 MB/day, 852.64 GB event log (previous lesson) 1168.00 MB/day, 852.64 GB -> with projection 1360.00 MB/day, 992.80 GB peak event flow = 111.11 events/s (K01: 3200000 events a day, peak multiplier 3) window events read factor full retention (730 days) 2336000000 146.00 open window (OY3=5) 16000000 1.00 OY4 consume events/s net events/s open window hours full window days 20.0 2222.22 2111.11 2.11 12.81 10.0 1111.11 1000.00 4.44 27.04 1.5 166.67 55.56 80.00 486.67 1.0 111.11 0.00 cannot catch up cannot catch up lag window rebuilding in place = 2.11 hours; rebuilding side by side, 0 side-by-side rebuild's temporary space cost = 140.16 GB, peak storage 1132.96 GB ``` #### The Derivation's Correctness The first three lines test the three promises the derivation carries. The projection built by incremental maintenance and the one rebuilt from scratch give the same output: folding does not depend on the speed at which events are applied. The partial rebuild, done with only open shipments' events, built those shipments' rows identically to the full rebuild — closed shipments' events do not affect open ones. Third, the second-version row carries a transfer counter, the first does not: a schema change is the projection's own concern, not the log's. The writes-per-event column shows the difference between the two projections. The tracking projection writes one row per event (1.00), because the event's affected row is known. The point counter writes 1.57, because a shipment moving from one point to another changes two rows at once. The row-count column gives the second difference: the tracking projection holds 200 rows, the point counter 5. A projection's cost depends not on the log's length but on how many rows an event moves and how much space those rows take. #### Storage Cost The projection row is the tracking response's body, and its size was defined in K01 (V5, 480 bytes): 192 MB for 400,000 shipments a day, 140.16 GB over a 730-day retention window. That adds to the previous lesson's numbers: daily growth climbs from 1,168 MB to **1,360 MB**, stored data from 852.64 GB to **992.80 GB** — against K01's 976 MB and 712.48 GB, the two decisions' combined cost is 384 MB and 280.32 GB a day, a 1.39× factor. A coincidence in the table stands out: adding the projection to K01's persistence also comes to 1,168 MB/day and 852.64 GB — the same as the source event's extra-field cost. This is not a relationship but a coincidence of the chosen assumptions: 400,000 × 480 equals 3,200,000 × 60. K01 had flagged the same kind of coincidence in its own table; it is not memorized, it is recomputed. #### Rebuild and Catching Up The rebuild's measure is not duration but **events read**; duration depends on the environment, events read does not. The full retention window holds 2,336,000,000 events. The open window (OY3 = 5 days) holds 16,000,000: **146 times** fewer. That ratio settles the projection design's most important decision — if the tracking projection is only needed for open shipments, the rebuild can be bounded by the open window; if closed shipments' rows are also needed, it cannot. Converting to time needs OY4, and the table shows a threshold. At 20 times the live flow's speed, net progress is 2,111.11 events a second and the open window closes in 2.11 hours; the full window takes 12.81 days. At a ratio of 10, those times rise to 4.44 hours and 27.04 days. At 1.5, the open window takes 80 hours and the full window 486.67 days — longer than retention, so the rebuild never finishes. At a ratio of 1, consumption speed equals flow speed and the rebuild **cannot catch up**. The rule fits one sentence: duration is set not by consumption speed but by the **gap** between consumption speed and flow speed. #### In Place or Side by Side If the rebuild runs on top of the live projection, the read model is incomplete throughout it; the lag window is 2.11 hours, and tracking queries see missing rows during that time. The alternative is to build the new projection side by side with the old one and switch the read path to it once it finishes. The lag window is then 0, because the old projection keeps running throughout. The cost is space: both projections sit in storage at once during the rebuild, carrying a second 140.16 GB copy; stored data peaks at 1,132.96 GB. This is another form of the trade-off repeated in every lesson of this course — the lag window closes in exchange for space. Which one is chosen depends on how long the projection can stay incomplete, a domain decision, not a technical one. #### Summary - The projection is derived from the log by folding; incremental maintenance and a from-scratch rebuild gave the same rows, and the partial rebuild built open shipments' rows identically to the full rebuild. - A projection's cost depends not on the log's length but on how many rows an event moves: the tracking projection writes 1.00 row per event and holds 200 rows, the point counter 1.57 rows and 5 rows. - The projection adds 192 MB to daily growth and 140.16 GB to stored data: 1,168 → 1,360 MB/day, 852.64 → 992.80 GB. Against K01's baseline the two decisions combined come to 1.39 times. - The rebuild's measure is events read: the full window is 2,336,000,000, the open window (OY3 = 5 days) 16,000,000 — a 146× difference. - Duration comes from the gap between consumption speed and flow speed: at OY4 = 20 the open window is 2.11 hours, at 10 it is 4.44 hours, at 1.5 it is 80 hours, at 1 it never catches up. - Rebuilding in place leaves a 2.11-hour lag window; rebuilding side by side brings that to 0 in exchange for 140.16 GB of temporary space (1,132.96 GB at peak). #### Next Step Across three lessons the lag window was treated as a resource measure: how many events, how many hours, how many gigabytes. But what sits inside that window is not a number, it is a user. A recipient corrects an address and reloads the page to see the old one; a carrier sends a state event and cannot find it on the dashboard; two parties change the same shipment at the same time and one never learns the other's change was lost. The next lesson measures the window's counterpart in the interface: how many stale values the user sees, how many rounds they wait, how many conflict notifications occur, and the requests each solution adds back to K01's store load. ### Eventual Consistency and the User Experience Source: https://academia.sh/en/courses/scaling-the-data/eventual-consistency-and-the-user-experience The lag window's counterpart in the interface: the stale views, wait rounds, conflict notifications, and silent losses four client strategies produce over the same event sequence, the optimistic display producing silent loss once the lag exceeds the view count, and read-your-writes raising the write store from 97.22 to 137.22 ops/s. The previous three lessons treated the lag window as a resource measure: how many events, how many hours, how many gigabytes. What sits inside that window is not a number, it is a user. A recipient corrects an address and reloads the page to see the old one; two parties change the same shipment at once, and one never learns the other's change was lost. The term this course uses is **eventual consistency**, defined in the Introduction to System Design course: once writes stop, replicas converge to the same value, with silence about what reads see until then. Monotonic reads and read-your-writes were also defined there and are not repeated here. This lesson's question is how that silence gets filled in the interface, and what each fill adds to K01's computed value. #### The Source of the Measurement No browser output is produced for the interface measurement; what the user sees is drawn from a **state machine**. The user writes in one round and looks several times in later rounds; the projection applies writes at a fixed lag, and a round is an abstract step, not a second. In one in four sessions another party also writes to the same shipment; in the log, whichever comes last wins. The last one is the real measure. **Silent loss** is the user's write invalidated by another write, with the user never learning it. An interface decision's quality is measured by this number more than by the count of stale views. ```js // interface/session.mjs — in-process model of a user session. No browser, screen, or duration; // a round is an abstract step, and what the user sees comes out of the projection's lag window. // Four strategies see the same event sequence with the same lag; only the client's behavior differs. export function run({ strategy, lag, views, sessions }) { const writes = []; for (let i = 0; i < sessions; i += 1) { const t = i + 1, id = `G${i + 1}`; writes.push({ round: t, id, value: `k${i}`, writer: "user" }); if (i % 4 === 0) writes.push({ round: t, id, value: `r${i}`, writer: "rival" }); } const applied = (t) => { // projection: writes with round <= t - lag are applied const m = new Map(); for (const w of writes) if (w.round <= t - lag) m.set(w.id, w); return m; }; const fromLog = (id) => { // read from the write side: every write is visible let last = null; for (const w of writes) if (w.id === id) last = w; return last; }; const s = { stale: 0, wait: 0, requests: 0, writeSide: 0, notified: 0, silentLoss: 0, correctRound: 0 }; for (let i = 0; i < sessions; i += 1) { const t0 = i + 1, id = `G${i + 1}`; const winner = fromLog(id); const beaten = winner.writer !== "user"; let correctRound = null, notified = false; if (strategy === "raw") { for (let k = 1; k <= views; k += 1) { s.requests += 1; const m = applied(t0 + k); if (m.has(id) === false) { s.stale += 1; continue; } if (correctRound === null) correctRound = k; } } else if (strategy === "optimistic") { for (let k = 1; k <= views; k += 1) { s.requests += 1; // the client keeps its own write on screen: no stale value const m = applied(t0 + k); if (m.has(id) === false) continue; if (correctRound === null) correctRound = k; if (beaten && notified === false) { s.notified += 1; notified = true; } } } else if (strategy === "wait") { for (let k = 1; k <= views + lag; k += 1) { s.requests += 1; // the write returns a version token, the client polls const m = applied(t0 + k); if (m.has(id) === false) { s.wait += 1; continue; } correctRound = k; if (beaten) { s.notified += 1; notified = true; } break; } s.requests += views - 1; // normal views after the wait ends } else { for (let k = 1; k <= views; k += 1) { s.requests += 1; if (k <= lag) { // inside its own write window, read from the write side s.writeSide += 1; if (correctRound === null) correctRound = k; if (beaten && notified === false) { s.notified += 1; notified = true; } continue; } const m = applied(t0 + k); if (m.has(id) && correctRound === null) correctRound = k; } } if (beaten && notified === false) s.silentLoss += 1; s.correctRound += correctRound ?? views; } return { ...s, correctRound: s.correctRound / sessions }; } ``` Four strategies diverge. **Raw reads** show the projection as it is. **Optimistic display** is the client keeping its own write on screen — its source is the client, not the server. **Waiting for a token** polls until it reaches the version token the write returned. **Read-your-writes** sends reads to the write side instead of the projection throughout the lag window; the response there is produced by folding the log and costs 8 operations, as measured previously. ```js // interface/measure.mjs — four strategies are compared at the same lag, then lag is swept and // the results are applied to K01's read rate. All numbers are deterministic; a computed value. import { run } from "./session.mjs"; const SESSIONS = 200, VIEWS = 4, LAG = 3; const STRATEGIES = ["raw", "optimistic", "wait", "read-your-writes"]; const b = (x, n = 2) => x.toFixed(n); const p = (x, n) => String(x).padStart(n); console.log(`${SESSIONS} sessions, ${VIEWS} views per session, projection lag ${LAG} rounds` + `, a rival write hits the same shipment in one in four sessions`); console.log(`\n${"strategy".padEnd(16)}${"stale views".padStart(13)}${"wait rounds".padStart(14)}` + `${"read requests".padStart(14)}${"write side".padStart(14)}${"notification".padStart(14)}` + `${"silent loss".padStart(14)}${"correct round".padStart(15)}`); const baseline = {}; for (const st of STRATEGIES) { const r = run({ strategy: st, lag: LAG, views: VIEWS, sessions: SESSIONS }); baseline[st] = r; console.log(`${st.padEnd(16)}${p(r.stale, 13)}${p(r.wait, 14)}${p(r.requests, 14)}` + `${p(r.writeSide, 14)}${p(r.notified, 14)}${p(r.silentLoss, 14)}${b(r.correctRound).padStart(15)}`); } console.log(`\n${"lag".padStart(8)}${STRATEGIES.map((s) => `${s} stale`.padStart(24)).join("")}`); for (const g of [1, 3, 6, 12]) { const row = STRATEGIES.map((s) => { const r = run({ strategy: s, lag: g, views: VIEWS, sessions: SESSIONS }); return `${r.stale}/${r.silentLoss}`.padStart(24); }); console.log(`${p(g, 8)}${row.join("")}`); } console.log("(columns: stale views / silent loss)"); // Back to K01 const READ = 41.67, WRITE_OPS = 97.22, OY5 = 0.04, LOG_FOLD = 8; console.log(`\nK01: reads behind cache ${READ}/s; write store ${WRITE_OPS} ops/s (previous lesson)`); console.log(`OY5 = ${OY5} -> reads in the write window ${b(READ * OY5)}/s\n`); console.log(`${"strategy".padEnd(16)}${"reads/session".padStart(15)}${"extra reads/s".padStart(15)}` + `${"read store/s".padStart(15)}${"requests to store/s".padStart(21)}${"write store ops/s".padStart(19)}`); for (const st of STRATEGIES) { const r = baseline[st]; const perSession = r.requests / SESSIONS; const window = READ * OY5; const extra = window * (perSession - VIEWS); const shifted = window * (r.writeSide / SESSIONS); const readStore = READ + extra - shifted; console.log(`${st.padEnd(16)}${b(perSession).padStart(15)}${b(extra).padStart(15)}` + `${b(readStore).padStart(15)}${b(readStore + shifted + 97.22).padStart(21)}` + `${b(WRITE_OPS + shifted * LOG_FOLD).padStart(19)}`); } for (const oy of [OY5, 2 * OY5]) { const window = READ * oy; const wait = window * (baseline.wait.requests / SESSIONS - VIEWS); const ryw = window * (baseline["read-your-writes"].writeSide / SESSIONS) * LOG_FOLD; console.log(`OY5 = ${b(oy)} -> wait's extra reads ${b(wait)}/s (${b((100 * wait) / READ)}%)` + `, read-your-writes to write store ${b(ryw)} ops/s (${b((100 * ryw) / WRITE_OPS)}%)`); } ``` ``` 200 sessions, 4 views per session, projection lag 3 rounds, a rival write hits the same shipment in one in four sessions strategy stale views wait rounds read requests write side notification silent loss correct round raw 400 0 800 0 0 50 3.00 optimistic 0 0 800 0 50 0 3.00 wait 0 400 1200 0 50 0 3.00 read-your-writes 0 0 800 600 50 0 1.00 lag raw stale optimistic stale wait stale read-your-writes stale 1 0/50 0/0 0/0 0/0 3 400/50 0/0 0/0 0/0 6 800/50 0/50 0/0 0/0 12 800/50 0/50 0/0 0/0 (columns: stale views / silent loss) K01: reads behind cache 41.67/s; write store 97.22 ops/s (previous lesson) OY5 = 0.04 -> reads in the write window 1.67/s strategy reads/session extra reads/s read store/s requests to store/s write store ops/s raw 4.00 0.00 41.67 138.89 97.22 optimistic 4.00 0.00 41.67 138.89 97.22 wait 6.00 3.33 45.00 142.22 97.22 read-your-writes 4.00 0.00 36.67 138.89 137.22 OY5 = 0.04 -> wait's extra reads 3.33/s (8.00%), read-your-writes to write store 40.00 ops/s (41.15%) OY5 = 0.08 -> wait's extra reads 6.67/s (16.00%), read-your-writes to write store 80.01 ops/s (82.29%) ``` #### Reading the Numbers The first table compares the four strategies at the same window. Raw reads show the value from before the user's write in 400 of 800 views, and in 50 sessions the write silently vanishes — the user never sees their own value, so the system has nothing to tell them either. This is the cheapest and most damaging option. Optimistic display brings stale views to zero and reports all 50 conflicts, with no extra requests. But its correctness depends on a condition the second table shows: at a lag of 6 and 12 rounds, silent loss returns to 50. The reason is arithmetic — the user looks four times, the projection refreshes six rounds later, and the comparison never happens. **Optimistic display goes silent once the lag window exceeds the user's view count.** Waiting for a token keeps stale views and silent loss at zero but pays 400 wait rounds and 1,200 read requests: 6 per session instead of 4, 50 percent more. Read-your-writes also keeps both at zero and shows the correct value **on the first round** (3.00 for the other three), at the cost of 600 reads falling to the write side. The lag sweep ranks the strategies. At a lag of 1, all four are clean; as it grows, raw reads breaks first, then optimistic display. Wait and read-your-writes stay clean at every lag, because both close the window by **waiting** or by **skipping** it. #### Back to the Computed Value Tying the numbers to K01 needs a ratio. **OY5 — the share of tracking reads made in the lag window after the user's own write: 0.04.** This is this topic's own assumption and is not added to K01's table. Its rationale is that most state events come from carrier integration, and the only interface writers are the recipient correcting an address and the seller updating a shipment. Its sensitivity is given at 0.08. The last table shows what the four strategies do to K01's rows. Raw reads and optimistic display move no number: `requests reaching the store/s` stays at 138.89. Waiting for a token raises it to 142.22 with polling requests, an 8 percent increase landing on the read store. Read-your-writes does not change the request count but shifts 5.00 requests/s from the read store to the write store; at 8 operations each, the write store's load climbs from 97.22 to **137.22 ops/s**, a 41.15 percent increase. At double OY5 the increase is 82.29 percent. The comparison fits one sentence: waiting charges the **read** store, read-your-writes charges the **write** store. Three lessons tried to protect the write store, and the strongest session guarantee lands right there. The decision is therefore made per shipment: a user correcting an address reads their own write, a user only tracking a shipment reads the projection. #### Summary - Eventual consistency's measure in the interface is not stale views but **silent loss**: the user's write invalidated and never learned about. - Raw reads showed the old value in 400 of 800 views and produced 50 silent losses; optimistic display zeroed both and reported all 50 conflicts. - Optimistic display's condition is that lag stays below the view count: at a lag of 6 and 12 rounds, silent loss returned to 50. - Waiting for a token pays 400 wait rounds and 6 requests per session instead of 4; read-your-writes shows the correct value on the first round (3.00 for the others) and sends 600 reads to the write side. - Back to K01: waiting raises `requests reaching the store/s` from 138.89 to 142.22; read-your-writes leaves the request count unchanged but raises the write store from 97.22 to 137.22 ops/s (41.15 percent). At OY5 = 0.08 the increase is 82.29 percent. #### Course Wrap-Up The course removed the single-store assumption and asked the same three questions across three topics: what is the decision, which number in K01's computed value does it move, and which number grows in exchange. | Lesson | Decision | Number it moves | What grows in exchange | |---|---|---|---| | Data Store Selection | splitting stores by access pattern | records processed on the read side 291.69 → 41.67 | store operation rate 138.89 → 236.11; ratio 2.33 → 4.67 | | Replication | splitting reads across replicas | reads per replica 41.67 → 8.33 | `stored data GB` 718.24 → 4,309.44 (five replicas); ratio 23.33 | | Federation | splitting data by function | scan rate 833.33 records/s → 0 (online store) | 976 → 1,232 MB/day, 712.48 → 899.36 GB | | Sharding | choosing the key from the access pattern | 78.11 GB and 17.36 ops/s per node | touches in a non-filtering pattern 138.89 → 1,111.12/s | | Partitioning Strategies | the rule converting a key to a partition | hot spot ratio 8.00 → 1.00; scanned records 12,000,000 → 1,500,137 | `requests reaching the store/s` 138.89 → 277.78 | | Denormalization | shortening the read path | secondary records pulled 9,000 → 40 | ratio 2.33 → 8.71; 712.48 → 733.27 GB | | Materialized Views | precomputing the result | scan 833.33 → 8.33 records/s, 10,800 → 3.28 MB | ratio 2.33 → 7.13 (at a one-minute refresh) | | Store Types | choosing the type by access shape | a three-step graph traversal 3 → 1 request | key–value scan 1 → 3,001 requests; 138.89 → 236.11/s | | Cache Placement | which layer the hit is served from | reads reaching the app 416.67 → 60.51/s | 9 touches per miss, average 2.531 touches/request | | Cache-Aside | fetching responsibility sitting in the application | the 0.90 hit-ratio assumption comes out 0.7287 | `requests reaching the store/s` 138.89 → 210.26; ratio 2.33 → 0.86 | | Write-Through and Write-Behind | wiring the write path to the cache | hit ratio 0.7285 → 0.9170; requests 229.01 → 151.04/s | loss window 200 events, 44,000 bytes | | Refresh-Ahead | renewing before the ttl expires | requests held 57,815 → 34,270 | refreshes 165,730, wasted 27,825; requests 151.04 → 217.67/s | | Cacheless Design | where a cache is not needed | `requests reaching the store/s` 513.89 → 139.90 | 2.00 wasted touches per request in the scan pattern | | Consistency and Stale Data | accepting the staleness window | window 5 → 60 s; requests 250.85 → 140.23/s | stale ratio 0.001195, largest lag 1 event | | Command and Query Separation | separating the read and write paths | query touches 8 → 1; nodes 17 → 7 (read ×4) | command 3 → 5 touches; nodes 15 → 21 at write ×4 | | Event-Sourced Design | change becoming the source-of-truth record | operations per request 4.50 → 3.10 | 976 → 1,168 MB/day, 712.48 → 852.64 GB; scan 10.80 → 35.04 GB | | Deriving Read Models | deriving the projection from the log | rebuild 2,336,000,000 → 16,000,000 events | 1,168 → 1,360 MB/day, 852.64 → 992.80 GB | | Eventual Consistency and the User Experience | closing the window in the interface | silent loss 50 → 0; seeing the correct value 3.00 → 1.00 round | write store 97.22 → 137.22 ops/s | The table's rule deserves to be named. In this course, a data decision can only be defended together with an **access pattern**. The same decision is a gain under one pattern and a loss under another: a shard key is cheap under a filtering pattern, eight times more expensive under one that cannot filter; a cache gains on repeated reads, adds a touch on a scan; an event log gains on writes, loses 3.24 times on the periodic scan; read–write separation gains as reads grow, loses as writes grow. A decision given without its pattern cannot be measured, because which number moves is not settled. The course's own assumptions — the ones Data Distribution and Cache Architecture wrote under their own names, and this topic's OY1–OY5 — never mixed into K01's table; each was written with its rationale and its sensitivity. The question the course leaves behind sits in the shared assumption underneath all these parts. Data was replicated, split by function, sharded, cached, and the read model separated from the write model; every decision moved one number and grew another. But every measurement assumed the parts **worked**. What happens when a replica goes down, a shard slows, or the link between the log feeding a projection and the read side breaks was not designed in this course: partial failure, the spread of slowdown, network partition, circuit breaking, back pressure, and recovery targets were never counted. The next course, **Resilience and Reliability**, takes up that question.