Event-Sourced Design

The previous lesson updated the projection together with the command and undercut the separation’s point: two stores in the same consistency boundary cannot scale separately. If the projection is updated at its own pace, what it is derived from must be settled, and the shipment record is unsuited to that — it holds only the current state, not the change that produced it. Refreshing the projection needs the change itself.

Event sourcing does exactly that: what is stored is not the state but the sequence of events that produced it. The pattern was built in the Domain-Driven Design course and defended there by answerability — questions not in the state table’s schema come out of the event sequence. This lesson’s question is different: what the pattern does to the data layer’s scaling numbers. Does the log cheapen the write path, how much does it grow stored data, and which access pattern does it make expensive.

The Read Hidden in the Write Path

K01 had found the request rate reaching the store to be 138.89: 41.67 read requests and 97.22 write requests a second, ratio 2.33. That number counts requests. It never asked how many operations one write request turns into at the store.

In a persistence that updates state in place, a state event costs three operations: the event is written for audit, the shipment record is read, the updated record is written back. The middle read is not a request, so it does not appear in K01’s table — but it reaches the store all the same. An append-only log has no such read: a write only appends to the end of the sequence and does not need to know the prior state.

Being the source-of-truth record has a cost. K01’s event record (V6, 220 bytes) is an audit line: tracking number, state, route step, timestamp. When that same record becomes the sole source of truth it carries extra fields — event type, sequence number in the series, schema version, and an idempotency key. Without a sequence number the fold order is ambiguous; without a schema version an old-format event cannot be read; without an idempotency key a resent event applies twice.

OY2 — the source event record’s extra load: 60 bytes. This is this topic’s own assumption and is not added to K01’s table. Its rationale is the four fields above. Its sensitivity is given in the measurement’s last lines: at zero overhead K01’s numbers come out unchanged; at 120 bytes stored data grows once more.

Apparatus

Both persistence forms present the same interface and see the same events. It is an in-process model; there is no disk, network, or duration. Two things are counted: the store operation each access produces, and the stored bytes each record adds. The byte count comes not from JSON size but from K01’s V6 and V7 assumptions; the goal is to carry K01’s computed value forward, not the model’s own shape.

// separation/persistence.mjs — in-process model of two persistence forms. Same interface,
// same events; each access counts a store operation, each record adds K01's assumed bytes.
export const V4 = 7, V6 = 220, V7 = 900;   // K01 assumptions
export const OY2 = 60;                     // this topic's assumption: source event's extra fields

function createCounters() {
  return { reads: 0, writes: 0, bytes: 0 };
}

// A: persistence that updates state in place; events are also written for audit.
export function updateInPlace() {
  const records = new Map(), audit = new Map();
  const s = createCounters();
  return {
    name: "update in place", counters: s,
    create(id, zone) {
      records.set(id, { id, zone, state: "accepted", lastSteps: [] });
      audit.set(id, []);
      s.writes += 1; s.bytes += V7;
    },
    write(id, point) {
      audit.get(id).push({ id, point });
      s.writes += 1; s.bytes += V6;
      const k = records.get(id);
      s.reads += 1;                       // read-modify-write: the existing row is read first
      k.state = "in-transit";
      s.writes += 1;                       // the updated row is written back
    },
    read(id) {                             // tracking response: row + events for the last three steps
      const k = records.get(id);
      s.reads += 1;
      const d = audit.get(id);
      s.reads += d.length;
      return { id: k.id, zone: k.zone, state: k.state, lastSteps: d.slice(-3).map((o) => o.point) };
    },
    scan(ids) {                         // periodic batch scan: only the shipment row is read
      let bytes = 0;
      for (const id of ids) { records.get(id); s.reads += 1; bytes += V7; }
      return bytes;
    },
  };
}

// B: append-only event log only; state is derived by folding, no in-place update exists.
export function eventLog() {
  const log = new Map();
  const s = createCounters();
  const fold = (events) => {
    const first = events[0];
    let state = "accepted";
    const steps = [];
    for (const o of events.slice(1)) { state = "in-transit"; steps.push(o.point); }
    return { id: first.id, zone: first.zone, state, lastSteps: steps.slice(-3) };
  };
  return {
    name: "event log", counters: s,
    create(id, zone) {
      log.set(id, [{ type: "shipment-created", id, zone }]);
      s.writes += 1; s.bytes += V7 + OY2;
    },
    write(id, point) {
      log.get(id).push({ type: "transferred", id, point });
      s.writes += 1; s.bytes += V6 + OY2;   // no read on the write path
    },
    read(id) {
      const events = log.get(id);
      s.reads += events.length;
      return fold(events);
    },
    scan(ids) {
      let bytes = 0;
      for (const id of ids) {
        const events = log.get(id);
        s.reads += events.length;
        bytes += V7 + OY2 + (events.length - 1) * (V6 + OY2);
      }
      return bytes;
    },
  };
}

The fold function pulls the last three route steps from the sequence itself; the in-place persistence reads the same three steps from the audit events. Whether the two paths produce the same response is tested in the measurement.

Measurement

// separation/measure.mjs — the same events are written to both persistence forms; store
// operations per request and stored bytes per shipment are measured, then applied to K01's rates.
import { updateInPlace, eventLog, V4, V6, V7, OY2 } from "./persistence.mjs";

const POINTS = ["34", "06", "35", "01", "16"];
const N = 200;

function run(store) {
  let x = 7;
  const next = () => (x = (x * 48271 + 11) % 2147483647);
  const ids = Array.from({ length: N }, (_, i) => `G${i + 1}`);
  for (const id of ids) {
    store.create(id, POINTS[next() % POINTS.length]);
    for (let k = 0; k < V4; k += 1) store.write(id, POINTS[next() % POINTS.length]);
  }
  const setup = { ...store.counters };
  store.counters.reads = 0; store.counters.writes = 0;
  const responses = ids.map((id) => store.read(id));
  const query = store.counters.reads / N;
  store.counters.reads = 0;
  const scanBytes = store.scan(ids) / N;
  const scan = { ops: store.counters.reads / N, bytes: scanBytes };
  store.counters.reads = 0; store.counters.writes = 0;
  for (const id of ids) store.write(id, "16");
  const command = { reads: store.counters.reads / N, writes: store.counters.writes / N };
  return { name: store.name, command, query, scan, shipmentBytes: setup.bytes / N, responses };
}

const A = run(updateInPlace()), B = run(eventLog());
const b = (x, n = 2) => x.toFixed(n);
const p = (x, n) => String(x).padStart(n);

console.log(`model: ${N} shipments, ${V4} state events per shipment (K01 V4)`);
console.log(`same tracking response from both persistence forms = ${JSON.stringify(A.responses) === JSON.stringify(B.responses)}\n`);
console.log(`${"persistence".padEnd(16)}${"command ops".padStart(14)}${"query ops".padStart(14)}` +
  `${"scan ops".padStart(15)}${"shipment bytes".padStart(16)}`);
for (const r of [A, B])
  console.log(`${r.name.padEnd(16)}${`${r.command.reads}r + ${r.command.writes}w`.padStart(14)}` +
    `${p(r.query, 14)}${p(r.scan.ops, 15)}${p(r.shipmentBytes, 16)}`);

// K01 rates and assumptions
const READ = 41.67, WRITE = 97.22, DAILY_SHIPMENTS = 400_000, RETENTION = 730, BATCH_DAYS = 30, WINDOW = 4 * 3600;
console.log(`\nK01: requests reaching the store/s = ${b(READ + WRITE)} (${READ} read + ${WRITE} write)` +
  `, ratio ${b(WRITE / READ)}`);
console.log(`${"persistence".padEnd(16)}${"read ops/s".padStart(16)}${"write ops/s".padStart(16)}` +
  `${"total ops/s".padStart(16)}${"ops/request".padStart(13)}`);
for (const r of [A, B]) {
  const o = READ * r.query + WRITE * r.command.reads, w = WRITE * r.command.writes;
  console.log(`${r.name.padEnd(16)}${b(o).padStart(16)}${b(w).padStart(16)}${b(o + w).padStart(16)}` +
    `${b((o + w) / (READ + WRITE)).padStart(13)}`);
}

console.log(`\n${"persistence".padEnd(16)}${"daily growth MB".padStart(17)}${"stored GB".padStart(13)}` +
  `${"batch read GB".padStart(17)}${"batch Mbit/s".padStart(14)}`);
for (const r of [A, B]) {
  const daily = (DAILY_SHIPMENTS * r.shipmentBytes) / 1e6;
  const batch = (BATCH_DAYS * DAILY_SHIPMENTS * r.scan.bytes) / 1e9;
  console.log(`${r.name.padEnd(16)}${b(daily).padStart(17)}${b((daily * RETENTION) / 1000).padStart(13)}` +
    `${b(batch).padStart(17)}${b((batch * 1e9 * 8) / (WINDOW * 1e6)).padStart(14)}`);
}

const factor = (f) => f(B) / f(A);
console.log(`\nlog's factors: daily growth x${b(factor((r) => r.shipmentBytes))}` +
  `, batch read x${b(factor((r) => r.scan.bytes))}, command ops x${b(factor((r) => r.command.reads + r.command.writes))}`);
for (const extra of [0, OY2, 2 * OY2]) {
  const daily = (DAILY_SHIPMENTS * (V7 + extra + V4 * (V6 + extra))) / 1e6;
  console.log(`OY2 = ${p(extra, 3)} bytes -> daily growth ${b(daily)} MB` +
    `, stored ${b((daily * RETENTION) / 1000)} GB`);
}

model: 200 shipments, 7 state events per shipment (K01 V4)
same tracking response from both persistence forms = true

persistence        command ops     query ops       scan ops  shipment bytes
update in place        1r + 2w             8              1            2440
event log              0r + 1w             8              8            2920

K01: requests reaching the store/s = 138.89 (41.67 read + 97.22 write), ratio 2.33
persistence           read ops/s     write ops/s     total ops/s  ops/request
update in place           430.58          194.44          625.02         4.50
event log                 333.36           97.22          430.58         3.10

persistence       daily growth MB    stored GB    batch read GB  batch Mbit/s
update in place            976.00       712.48            10.80          6.00
event log                 1168.00       852.64            35.04         19.47

log's factors: daily growth x1.20, batch read x3.24, command ops x0.33
OY2 =   0 bytes -> daily growth 976.00 MB, stored 712.48 GB
OY2 =  60 bytes -> daily growth 1168.00 MB, stored 852.64 GB
OY2 = 120 bytes -> daily growth 1360.00 MB, stored 992.80 GB

Reading the Numbers

The first check is that the update in place row reproduces K01 exactly: 2,440 bytes per shipment, 976 MB a day, 712.48 GB over 730 days, 10.80 GB and 6.00 Mbit/s in the periodic scan; K01’s computed value had assumed this persistence form, so a mismatch would invalidate the comparison. The second check is that both persistence forms give the same tracking response.

The write path is where the log wins. A state event costs 3 operations in the update in place layout (1 read, 2 writes), 1 in the log — command cost falls to 0.33 times. At K01’s rates, 194.44 write operations a second drop to 97.22, and the 97.22 hidden reads the table never counted drop to zero. Store operations per request fall from 4.50 to 3.10: K01’s 138.89 requests were producing 625.02 operations; in the log they produce 430.58.

Storage is where the log loses. Bytes per shipment climb from 2,440 to 2,920; daily growth from 976 MB to 1,168 MB, stored data from 712.48 GB to 852.64 GB. The 1.20× growth comes entirely from OY2 — the sensitivity lines confirm it: at zero overhead the log’s storage numbers match K01’s, at 120 bytes stored data reaches 992.80 GB. The real finding is that the event sequence was already in K01’s table. 616 MB of the daily 976 MB is event records; the log does not multiply stored data by 5.7, it only adds the fields that come with being the source.

The scan path is where the log breaks. The periodic batch scan walks a seller’s thirty days and reads a single row per shipment in the update in place layout; in the log the same information needs eight events read and folded, so scanned bytes climb from 10.80 GB to 35.04 GB and the four-hour window’s bandwidth from 6.00 Mbit/s to 19.47 Mbit/s. K01 had already flagged this row as its largest source line, 3.75 times peak read throughput; with the log it reaches 12.17 times. A decision made for one access pattern can make another more than three times as expensive.

Where the Snapshot Falls Short

The Domain-Driven Design course had put a remedy on the log’s read cost: a snapshot starts the rebuild from a point rather than the start of the sequence, under a rule that says the number of events read does not drop unless the snapshot interval falls below the average history length.

In this system the average history length is 8 (one create event and V4 state events), so the interval needs to be below 8; at an interval of 1, what is stored is a state row updated after every event — the update in place layout itself. In a domain with a short history, a snapshot produces no separate option; only a few steps sit between the two ends. This course’s question is therefore not the snapshot but the projection: not a query reading the log, but a separate read model derived from it.

The log’s own limit belongs here too. Because a written event does not change, an append-only sequence is an obstacle when a record must be fully deleted; K01’s 730-day retention assumption (V12) is therefore a contractual decision, not a technical one.

Summary

• The log removes the hidden read on the write path: a state event goes from 3 operations to 1, store operations per request from 4.50 to 3.10; K01’s 138.89 requests’ operation cost falls from 625.02 to 430.58.
• Being the source costs OY2 = 60 bytes: daily growth 976 → 1,168 MB, stored data 712.48 → 852.64 GB, a 1.20× factor. At zero overhead K01’s numbers come out unchanged.
• The event sequence was already in K01’s table — 616 MB of the daily 976 MB is event records; the log does not multiply storage, it only adds sequence, type, schema version, and idempotency fields.
• The pattern that breaks is the periodic scan: 8 events are read per shipment instead of 1, scanned data 10.80 → 35.04 GB, window bandwidth 6.00 → 19.47 Mbit/s.
• Because the average history length is 8, a snapshot produces no separate option; at an interval of 1 the snapshot is already the state row itself.

Next Step

The log cheapened the write path and made the read path expensive; the gap between the two numbers closes only if the read path is moved outside the log. The previous lesson’s projection did that, but it was updated together with the command and so could not scale separately. The log now supplies the input for deriving it: a projection can read the log at its own pace, fall behind, and be rebuilt from scratch. The next lesson takes up that derivation — the extra data the projection stores, the number of events read to rebuild it, the condition under which new events flowing in during a rebuild are caught, and how long a rebuild takes when the schema changes.