--- title: 'Event-Sourced Design' source: 'https://academia.sh/en/courses/scaling-the-data/event-sourced-design' course: 'Scaling the Data Layer' language: en updated: '2026-08-23T07:01:34+00:00' license: 'CC BY-SA 4.0' --- # Event-Sourced Design The scaling cost of storing change as a record: the append-only log removing the read-modify-write read from the write path, store operations per request dropping from 4.50 to 3.10, the source event's extra fields raising daily growth from 976 to 1,168 MB and stored data from 712.48 to 852.64 GB, and the periodic scan climbing from 10.80 to 35.04 GB. The previous lesson updated the projection together with the command and undercut the separation's point: two stores in the same consistency boundary cannot scale separately. If the projection is updated at its own pace, what it is derived from must be settled, and the shipment record is unsuited to that — it holds only the current state, not the change that produced it. Refreshing the projection needs the change itself. **Event sourcing** does exactly that: what is stored is not the state but the sequence of events that produced it. The pattern was built in the Domain-Driven Design course and defended there by answerability — questions not in the state table's schema come out of the event sequence. This lesson's question is different: what the pattern does to the data layer's **scaling numbers**. Does the log cheapen the write path, how much does it grow stored data, and which access pattern does it make expensive. ## The Read Hidden in the Write Path K01 had found the request rate reaching the store to be 138.89: 41.67 read requests and 97.22 write requests a second, ratio 2.33. That number counts **requests**. It never asked how many operations one write request turns into at the store. In a persistence that updates state in place, a state event costs three operations: the event is written for audit, the shipment record is read, the updated record is written back. The middle read is not a request, so it does not appear in K01's table — but it reaches the store all the same. An append-only log has no such read: a write only appends to the end of the sequence and does not need to know the prior state. Being the source-of-truth record has a cost. K01's event record (V6, 220 bytes) is an audit line: tracking number, state, route step, timestamp. When that same record becomes the sole source of truth it carries extra fields — event type, sequence number in the series, schema version, and an idempotency key. Without a sequence number the fold order is ambiguous; without a schema version an old-format event cannot be read; without an idempotency key a resent event applies twice. **OY2 — the source event record's extra load: 60 bytes.** This is this topic's own assumption and is not added to K01's table. Its rationale is the four fields above. Its sensitivity is given in the measurement's last lines: at zero overhead K01's numbers come out unchanged; at 120 bytes stored data grows once more. ## Apparatus Both persistence forms present the same interface and see the same events. It is an in-process model; there is no disk, network, or duration. Two things are counted: the **store operation** each access produces, and the **stored bytes** each record adds. The byte count comes not from JSON size but from K01's V6 and V7 assumptions; the goal is to carry K01's computed value forward, not the model's own shape. ```js // separation/persistence.mjs — in-process model of two persistence forms. Same interface, // same events; each access counts a store operation, each record adds K01's assumed bytes. export const V4 = 7, V6 = 220, V7 = 900; // K01 assumptions export const OY2 = 60; // this topic's assumption: source event's extra fields function createCounters() { return { reads: 0, writes: 0, bytes: 0 }; } // A: persistence that updates state in place; events are also written for audit. export function updateInPlace() { const records = new Map(), audit = new Map(); const s = createCounters(); return { name: "update in place", counters: s, create(id, zone) { records.set(id, { id, zone, state: "accepted", lastSteps: [] }); audit.set(id, []); s.writes += 1; s.bytes += V7; }, write(id, point) { audit.get(id).push({ id, point }); s.writes += 1; s.bytes += V6; const k = records.get(id); s.reads += 1; // read-modify-write: the existing row is read first k.state = "in-transit"; s.writes += 1; // the updated row is written back }, read(id) { // tracking response: row + events for the last three steps const k = records.get(id); s.reads += 1; const d = audit.get(id); s.reads += d.length; return { id: k.id, zone: k.zone, state: k.state, lastSteps: d.slice(-3).map((o) => o.point) }; }, scan(ids) { // periodic batch scan: only the shipment row is read let bytes = 0; for (const id of ids) { records.get(id); s.reads += 1; bytes += V7; } return bytes; }, }; } // B: append-only event log only; state is derived by folding, no in-place update exists. export function eventLog() { const log = new Map(); const s = createCounters(); const fold = (events) => { const first = events[0]; let state = "accepted"; const steps = []; for (const o of events.slice(1)) { state = "in-transit"; steps.push(o.point); } return { id: first.id, zone: first.zone, state, lastSteps: steps.slice(-3) }; }; return { name: "event log", counters: s, create(id, zone) { log.set(id, [{ type: "shipment-created", id, zone }]); s.writes += 1; s.bytes += V7 + OY2; }, write(id, point) { log.get(id).push({ type: "transferred", id, point }); s.writes += 1; s.bytes += V6 + OY2; // no read on the write path }, read(id) { const events = log.get(id); s.reads += events.length; return fold(events); }, scan(ids) { let bytes = 0; for (const id of ids) { const events = log.get(id); s.reads += events.length; bytes += V7 + OY2 + (events.length - 1) * (V6 + OY2); } return bytes; }, }; } ``` The fold function pulls the last three route steps from the sequence itself; the in-place persistence reads the same three steps from the audit events. Whether the two paths produce the same response is tested in the measurement. ## Measurement ```js // separation/measure.mjs — the same events are written to both persistence forms; store // operations per request and stored bytes per shipment are measured, then applied to K01's rates. import { updateInPlace, eventLog, V4, V6, V7, OY2 } from "./persistence.mjs"; const POINTS = ["34", "06", "35", "01", "16"]; const N = 200; function run(store) { let x = 7; const next = () => (x = (x * 48271 + 11) % 2147483647); const ids = Array.from({ length: N }, (_, i) => `G${i + 1}`); for (const id of ids) { store.create(id, POINTS[next() % POINTS.length]); for (let k = 0; k < V4; k += 1) store.write(id, POINTS[next() % POINTS.length]); } const setup = { ...store.counters }; store.counters.reads = 0; store.counters.writes = 0; const responses = ids.map((id) => store.read(id)); const query = store.counters.reads / N; store.counters.reads = 0; const scanBytes = store.scan(ids) / N; const scan = { ops: store.counters.reads / N, bytes: scanBytes }; store.counters.reads = 0; store.counters.writes = 0; for (const id of ids) store.write(id, "16"); const command = { reads: store.counters.reads / N, writes: store.counters.writes / N }; return { name: store.name, command, query, scan, shipmentBytes: setup.bytes / N, responses }; } const A = run(updateInPlace()), B = run(eventLog()); const b = (x, n = 2) => x.toFixed(n); const p = (x, n) => String(x).padStart(n); console.log(`model: ${N} shipments, ${V4} state events per shipment (K01 V4)`); console.log(`same tracking response from both persistence forms = ${JSON.stringify(A.responses) === JSON.stringify(B.responses)}\n`); console.log(`${"persistence".padEnd(16)}${"command ops".padStart(14)}${"query ops".padStart(14)}` + `${"scan ops".padStart(15)}${"shipment bytes".padStart(16)}`); for (const r of [A, B]) console.log(`${r.name.padEnd(16)}${`${r.command.reads}r + ${r.command.writes}w`.padStart(14)}` + `${p(r.query, 14)}${p(r.scan.ops, 15)}${p(r.shipmentBytes, 16)}`); // K01 rates and assumptions const READ = 41.67, WRITE = 97.22, DAILY_SHIPMENTS = 400_000, RETENTION = 730, BATCH_DAYS = 30, WINDOW = 4 * 3600; console.log(`\nK01: requests reaching the store/s = ${b(READ + WRITE)} (${READ} read + ${WRITE} write)` + `, ratio ${b(WRITE / READ)}`); console.log(`${"persistence".padEnd(16)}${"read ops/s".padStart(16)}${"write ops/s".padStart(16)}` + `${"total ops/s".padStart(16)}${"ops/request".padStart(13)}`); for (const r of [A, B]) { const o = READ * r.query + WRITE * r.command.reads, w = WRITE * r.command.writes; console.log(`${r.name.padEnd(16)}${b(o).padStart(16)}${b(w).padStart(16)}${b(o + w).padStart(16)}` + `${b((o + w) / (READ + WRITE)).padStart(13)}`); } console.log(`\n${"persistence".padEnd(16)}${"daily growth MB".padStart(17)}${"stored GB".padStart(13)}` + `${"batch read GB".padStart(17)}${"batch Mbit/s".padStart(14)}`); for (const r of [A, B]) { const daily = (DAILY_SHIPMENTS * r.shipmentBytes) / 1e6; const batch = (BATCH_DAYS * DAILY_SHIPMENTS * r.scan.bytes) / 1e9; console.log(`${r.name.padEnd(16)}${b(daily).padStart(17)}${b((daily * RETENTION) / 1000).padStart(13)}` + `${b(batch).padStart(17)}${b((batch * 1e9 * 8) / (WINDOW * 1e6)).padStart(14)}`); } const factor = (f) => f(B) / f(A); console.log(`\nlog's factors: daily growth x${b(factor((r) => r.shipmentBytes))}` + `, batch read x${b(factor((r) => r.scan.bytes))}, command ops x${b(factor((r) => r.command.reads + r.command.writes))}`); for (const extra of [0, OY2, 2 * OY2]) { const daily = (DAILY_SHIPMENTS * (V7 + extra + V4 * (V6 + extra))) / 1e6; console.log(`OY2 = ${p(extra, 3)} bytes -> daily growth ${b(daily)} MB` + `, stored ${b((daily * RETENTION) / 1000)} GB`); } ``` ``` model: 200 shipments, 7 state events per shipment (K01 V4) same tracking response from both persistence forms = true persistence command ops query ops scan ops shipment bytes update in place 1r + 2w 8 1 2440 event log 0r + 1w 8 8 2920 K01: requests reaching the store/s = 138.89 (41.67 read + 97.22 write), ratio 2.33 persistence read ops/s write ops/s total ops/s ops/request update in place 430.58 194.44 625.02 4.50 event log 333.36 97.22 430.58 3.10 persistence daily growth MB stored GB batch read GB batch Mbit/s update in place 976.00 712.48 10.80 6.00 event log 1168.00 852.64 35.04 19.47 log's factors: daily growth x1.20, batch read x3.24, command ops x0.33 OY2 = 0 bytes -> daily growth 976.00 MB, stored 712.48 GB OY2 = 60 bytes -> daily growth 1168.00 MB, stored 852.64 GB OY2 = 120 bytes -> daily growth 1360.00 MB, stored 992.80 GB ``` ## Reading the Numbers The first check is that the `update in place` row reproduces K01 exactly: 2,440 bytes per shipment, 976 MB a day, 712.48 GB over 730 days, 10.80 GB and 6.00 Mbit/s in the periodic scan; K01's computed value had assumed this persistence form, so a mismatch would invalidate the comparison. The second check is that both persistence forms give the same tracking response. The write path is where the log wins. A state event costs 3 operations in the `update in place` layout (1 read, 2 writes), 1 in the log — command cost falls to **0.33 times**. At K01's rates, 194.44 write operations a second drop to 97.22, and the 97.22 hidden reads the table never counted drop to zero. Store operations per request fall from 4.50 to 3.10: K01's 138.89 requests were producing 625.02 operations; in the log they produce 430.58. Storage is where the log loses. Bytes per shipment climb from 2,440 to 2,920; daily growth from 976 MB to **1,168 MB**, stored data from 712.48 GB to **852.64 GB**. The 1.20× growth comes entirely from OY2 — the sensitivity lines confirm it: at zero overhead the log's storage numbers match K01's, at 120 bytes stored data reaches 992.80 GB. The real finding is that **the event sequence was already in K01's table.** 616 MB of the daily 976 MB is event records; the log does not multiply stored data by 5.7, it only adds the fields that come with being the source. The scan path is where the log breaks. The periodic batch scan walks a seller's thirty days and reads a single row per shipment in the `update in place` layout; in the log the same information needs eight events read and folded, so scanned bytes climb from 10.80 GB to **35.04 GB** and the four-hour window's bandwidth from 6.00 Mbit/s to **19.47 Mbit/s**. K01 had already flagged this row as its largest source line, 3.75 times peak read throughput; with the log it reaches 12.17 times. A decision made for one access pattern can make another more than three times as expensive. ## Where the Snapshot Falls Short The Domain-Driven Design course had put a remedy on the log's read cost: a snapshot starts the rebuild from a point rather than the start of the sequence, under a rule that says the number of events read does not drop unless the snapshot interval falls below the average history length. In this system the average history length is 8 (one create event and V4 state events), so the interval needs to be below 8; at an interval of 1, what is stored is a state row updated after every event — the `update in place` layout itself. In a domain with a short history, a snapshot produces no separate option; only a few steps sit between the two ends. This course's question is therefore not the snapshot but the projection: not a query reading the log, but a separate read model derived from it. The log's own limit belongs here too. Because a written event does not change, an append-only sequence is an obstacle when a record must be fully deleted; K01's 730-day retention assumption (V12) is therefore a contractual decision, not a technical one. ## Summary - The log removes the hidden read on the write path: a state event goes from 3 operations to 1, store operations per request from 4.50 to 3.10; K01's 138.89 requests' operation cost falls from 625.02 to 430.58. - Being the source costs OY2 = 60 bytes: daily growth 976 → 1,168 MB, stored data 712.48 → 852.64 GB, a 1.20× factor. At zero overhead K01's numbers come out unchanged. - The event sequence was already in K01's table — 616 MB of the daily 976 MB is event records; the log does not multiply storage, it only adds sequence, type, schema version, and idempotency fields. - The pattern that breaks is the periodic scan: 8 events are read per shipment instead of 1, scanned data 10.80 → 35.04 GB, window bandwidth 6.00 → 19.47 Mbit/s. - Because the average history length is 8, a snapshot produces no separate option; at an interval of 1 the snapshot is already the state row itself. ## Next Step The log cheapened the write path and made the read path expensive; the gap between the two numbers closes only if the read path is moved outside the log. The previous lesson's projection did that, but it was updated together with the command and so could not scale separately. The log now supplies the input for deriving it: a projection can read the log at its own pace, fall behind, and be rebuilt from scratch. The next lesson takes up that derivation — the extra data the projection stores, the number of events read to rebuild it, the condition under which new events flowing in during a rebuild are caught, and how long a rebuild takes when the schema changes.