Deriving Read Models

The previous lesson made the log the source of the write path and showed its cost in two places: stored data grew to 1.20 times, the periodic scan to 3.24 times. The second number is the log’s real limit — if a query’s answer comes from folding eight events, the read path depends on the log’s length. The first lesson had already seen the fix: keep the row the query wants ready. Now where that row is derived from is settled: the log.

This lesson takes up the derivation itself. Folding is applying the log’s events to a read model; its product is the projection. Three things are measured: how much space the projection takes, how many events a from-scratch rebuild reads, and whether new events flowing in during a rebuild are caught.

One Log, Many Projections

A projection is a fold function: it takes the event sequence and produces rows. Because folding is pure, the same log always gives the same rows, which makes a projection disposable. Being disposable is decisive for scaling — when a read model breaks, its schema changes, or it needs to be copied to a new node, it is built from the log, not restored from a backup.

More than one projection can be derived from the same log, each shaped to its own access pattern. This topic has two: the tracking projection, serving a single read by tracking number, and the point counter, holding the count of shipments pending at each transfer point. The first’s row count depends on the number of shipments, the second’s on the number of transfer points; both feed from the same events. Storing precomputed results inside the store is taken up separately in this course’s Data Distribution topic; the difference here is that the projection derives from the log, not the store.

// derive/projection.mjs — two read models derived from the event log. Each projection is a
// fold function; counters keep events read and rows written separate. In-process model.
export function trackingProjection(version) {
  const rows = new Map();
  const s = { events: 0, writes: 0 };
  return {
    name: `tracking v${version}`, rows, counters: s,
    apply(o) {
      s.events += 1;
      if (o.type === "shipment-created") {
        const r = { id: o.id, zone: o.zone, state: "accepted", lastSteps: [] };
        if (version === 2) r.transfers = 0;
        rows.set(o.id, r);
        s.writes += 1;
        return;
      }
      const r = rows.get(o.id);
      if (o.type === "delivered") { r.state = "delivered"; s.writes += 1; return; }
      r.state = "in-transit";
      r.lastSteps = [...r.lastSteps, o.point].slice(-3);
      if (version === 2) r.transfers += 1;
      s.writes += 1;
    },
    view: () => JSON.stringify([...rows.entries()].sort()),
  };
}

// Second read model: a counter of pending shipments by transfer point. Derived from the same
// log, but its row count depends on the number of points, not the number of shipments.
export function pointCounter() {
  const rows = new Map(), location = new Map();
  const s = { events: 0, writes: 0 };
  return {
    name: "point counter", rows, counters: s,
    apply(o) {
      s.events += 1;
      if (o.type === "shipment-created") return;
      const prior = location.get(o.id);
      if (prior !== undefined) { rows.set(prior, rows.get(prior) - 1); s.writes += 1; }
      if (o.type === "delivered") { location.delete(o.id); return; }
      rows.set(o.point, (rows.get(o.point) ?? 0) + 1);
      s.writes += 1;
      location.set(o.id, o.point);
    },
    view: () => JSON.stringify([...rows.entries()].sort()),
  };
}

// Log generator: every shipment gets one create and V4 transfer events; closed shipments
// also carry a delivered event. The generator is deterministic: same input, same log.
export function generateLog(count, eventsPerShipment) {
  const POINTS = ["34", "06", "35", "01", "16"];
  let x = 7;
  const next = () => (x = (x * 48271 + 11) % 2147483647);
  const log = [], open = new Set();
  for (let i = 1; i <= count; i += 1) {
    const id = `G${i}`;
    log.push({ type: "shipment-created", id, zone: POINTS[next() % POINTS.length] });
    for (let k = 0; k < eventsPerShipment; k += 1)
      log.push({ type: "transferred", id, point: POINTS[next() % POINTS.length] });
    if (i % 5 === 0) open.add(id);
    else log.push({ type: "delivered", id });
  }
  return { log, open };
}

The version parameter models a schema change: the second version adds a transfer counter to the row. The only way to apply the new field to old rows is to rebuild the projection from the log.

Measurement

The measurement has two layers. The model layer gives the derivation’s correctness and the writes per event; those are deterministic numbers. The computed-value layer applies those numbers to K01’s volumes and needs two new assumptions.

OY3 — how long a shipment stays open: 5 days. Its rationale is that K01’s 730-day retention assumption (V12) is for contract disputes, not for a shipment’s flow; the time from acceptance to delivery is shorter than that. Its sensitivity: at 10 days, the open window’s event count and rebuild time double.

OY4 — the ratio of a rebuild’s event-consumption speed to the live flow: 20. Its rationale is that a rebuild does batch reads and carries no per-request network round trip or validation. A ratio is chosen, not an absolute speed, so the number is not a measurement. Its sensitivity is given in the table with three values, and it shows a threshold.

// derive/measure.mjs — the projection is built by incremental maintenance and by a from-scratch
// rebuild; the two are compared, then applied to K01's volumes and this topic's two assumptions.
import { trackingProjection, pointCounter, generateLog } from "./projection.mjs";

const V4 = 7, V5 = 480, V12 = 730, DAILY_SHIPMENTS = 400_000, V8 = 3, DAY = 86_400;
const OY3 = 5, OY4 = 20;   // this topic's assumptions: days open, rebuild speed multiplier
const N = 200;

const { log, open } = generateLog(N, V4);
const incremental = trackingProjection(1);
for (const o of log) incremental.apply(o);

const full = trackingProjection(1);
for (const o of log) full.apply(o);

const activeLog = log.filter((o) => open.has(o.id));
const active = trackingProjection(1);
for (const o of activeLog) active.apply(o);

const v2 = trackingProjection(2);
for (const o of log) v2.apply(o);
const points = pointCounter();
for (const o of log) points.apply(o);

const b = (x, n = 2) => x.toFixed(n);
const p = (x, n) => String(x).padStart(n);
console.log(`log: ${N} shipments, ${log.length} events, open shipments ${open.size}` +
  `, active log ${activeLog.length} events`);
console.log(`incremental maintenance matches from-scratch rebuild = ${incremental.view() === full.view()}`);
console.log(`active rebuild builds the same rows for open shipments = ` +
  `${[...open].every((id) => JSON.stringify(active.rows.get(id)) === JSON.stringify(full.rows.get(id)))}`);
console.log(`v2 row has the new field = ${"transfers" in v2.rows.get("G5")}` +
  `, v1 row = ${"transfers" in full.rows.get("G5")}\n`);

console.log(`${"projection".padEnd(21)}${"events read".padStart(13)}${"rows written".padStart(15)}` +
  `${"row count".padStart(14)}${"writes per event".padStart(19)}`);
for (const [name, r] of [["tracking v1 full", full], ["tracking v1 active", active],
  ["tracking v2 full", v2], ["point counter", points]])
  console.log(`${name.padEnd(21)}${p(r.counters.events, 13)}${p(r.counters.writes, 15)}` +
    `${p(r.rows.size, 14)}${b(r.counters.writes / r.counters.events).padStart(19)}`);

// Back to K01: the data the projection stores
const dailyProjection = (DAILY_SHIPMENTS * V5) / 1e6;
console.log(`\nprojection row ${V5} bytes (K01 V5), ${DAILY_SHIPMENTS} rows a day`);
console.log(`daily growth the projection adds = ${b(dailyProjection)} MB` +
  `, ${b((dailyProjection * V12) / 1000)} GB over ${V12} days`);
for (const [name, day, stored] of [["K01", 976, 712.48], ["event log (previous lesson)", 1168, 852.64]])
  console.log(`${name.padEnd(29)} ${b(day)} MB/day, ${b(stored)} GB -> with projection ` +
    `${b(day + dailyProjection)} MB/day, ${b(stored + (dailyProjection * V12) / 1000)} GB`);

// Rebuild: events read and catching up
const dailyEvents = DAILY_SHIPMENTS * (1 + V4);
const peakFlow = (dailyEvents / DAY) * V8;
const fullEvents = dailyEvents * V12, activeEvents = dailyEvents * OY3;
console.log(`\npeak event flow = ${b(peakFlow)} events/s (K01: ${dailyEvents} events a day, peak multiplier ${V8})`);
console.log(`${"window".padEnd(27)}${"events read".padStart(15)}${"factor".padStart(8)}`);
console.log(`${`full retention (${V12} days)`.padEnd(27)}${p(fullEvents, 15)}${b(fullEvents / activeEvents).padStart(8)}`);
console.log(`${`open window (OY3=${OY3})`.padEnd(27)}${p(activeEvents, 15)}${"1.00".padStart(8)}`);

console.log(`\n${"OY4".padStart(5)}${"consume events/s".padStart(18)}${"net events/s".padStart(14)}` +
  `${"open window hours".padStart(19)}${"full window days".padStart(17)}`);
for (const k of [OY4, OY4 / 2, 1.5, 1]) {
  const consume = k * peakFlow, net = consume - peakFlow;
  const hours = net <= 0 ? null : activeEvents / net / 3600;
  console.log(`${b(k, 1).padStart(5)}${b(consume).padStart(18)}${b(net).padStart(14)}` +
    `${(hours === null ? "cannot catch up" : b(hours)).padStart(19)}` +
    `${(hours === null ? "cannot catch up" : b(fullEvents / net / 86400)).padStart(17)}`);
}
console.log(`\nlag window rebuilding in place = ` +
  `${b(activeEvents / ((OY4 - 1) * peakFlow) / 3600)} hours; rebuilding side by side, 0`);
console.log(`side-by-side rebuild's temporary space cost = ${b((dailyProjection * V12) / 1000)} GB` +
  `, peak storage ${b(852.64 + 2 * (dailyProjection * V12) / 1000)} GB`);

log: 200 shipments, 1760 events, open shipments 40, active log 320 events
incremental maintenance matches from-scratch rebuild = true
active rebuild builds the same rows for open shipments = true
v2 row has the new field = true, v1 row = false

projection             events read   rows written     row count   writes per event
tracking v1 full              1760           1760           200               1.00
tracking v1 active             320            320            40               1.00
tracking v2 full              1760           1760           200               1.00
point counter                 1760           2760             5               1.57

projection row 480 bytes (K01 V5), 400000 rows a day
daily growth the projection adds = 192.00 MB, 140.16 GB over 730 days
K01                           976.00 MB/day, 712.48 GB -> with projection 1168.00 MB/day, 852.64 GB
event log (previous lesson)   1168.00 MB/day, 852.64 GB -> with projection 1360.00 MB/day, 992.80 GB

peak event flow = 111.11 events/s (K01: 3200000 events a day, peak multiplier 3)
window                         events read  factor
full retention (730 days)       2336000000  146.00
open window (OY3=5)               16000000    1.00

  OY4  consume events/s  net events/s  open window hours full window days
 20.0           2222.22       2111.11               2.11            12.81
 10.0           1111.11       1000.00               4.44            27.04
  1.5            166.67         55.56              80.00           486.67
  1.0            111.11          0.00    cannot catch up  cannot catch up

lag window rebuilding in place = 2.11 hours; rebuilding side by side, 0
side-by-side rebuild's temporary space cost = 140.16 GB, peak storage 1132.96 GB

The Derivation’s Correctness

The first three lines test the three promises the derivation carries. The projection built by incremental maintenance and the one rebuilt from scratch give the same output: folding does not depend on the speed at which events are applied. The partial rebuild, done with only open shipments’ events, built those shipments’ rows identically to the full rebuild — closed shipments’ events do not affect open ones. Third, the second-version row carries a transfer counter, the first does not: a schema change is the projection’s own concern, not the log’s.

The writes-per-event column shows the difference between the two projections. The tracking projection writes one row per event (1.00), because the event’s affected row is known. The point counter writes 1.57, because a shipment moving from one point to another changes two rows at once. The row-count column gives the second difference: the tracking projection holds 200 rows, the point counter 5. A projection’s cost depends not on the log’s length but on how many rows an event moves and how much space those rows take.

Storage Cost

The projection row is the tracking response’s body, and its size was defined in K01 (V5, 480 bytes): 192 MB for 400,000 shipments a day, 140.16 GB over a 730-day retention window.

That adds to the previous lesson’s numbers: daily growth climbs from 1,168 MB to 1,360 MB, stored data from 852.64 GB to 992.80 GB — against K01’s 976 MB and 712.48 GB, the two decisions’ combined cost is 384 MB and 280.32 GB a day, a 1.39× factor.

A coincidence in the table stands out: adding the projection to K01’s persistence also comes to 1,168 MB/day and 852.64 GB — the same as the source event’s extra-field cost. This is not a relationship but a coincidence of the chosen assumptions: 400,000 × 480 equals 3,200,000 × 60. K01 had flagged the same kind of coincidence in its own table; it is not memorized, it is recomputed.

Rebuild and Catching Up

The rebuild’s measure is not duration but events read; duration depends on the environment, events read does not. The full retention window holds 2,336,000,000 events. The open window (OY3 = 5 days) holds 16,000,000: 146 times fewer. That ratio settles the projection design’s most important decision — if the tracking projection is only needed for open shipments, the rebuild can be bounded by the open window; if closed shipments’ rows are also needed, it cannot.

Converting to time needs OY4, and the table shows a threshold. At 20 times the live flow’s speed, net progress is 2,111.11 events a second and the open window closes in 2.11 hours; the full window takes 12.81 days. At a ratio of 10, those times rise to 4.44 hours and 27.04 days. At 1.5, the open window takes 80 hours and the full window 486.67 days — longer than retention, so the rebuild never finishes. At a ratio of 1, consumption speed equals flow speed and the rebuild cannot catch up. The rule fits one sentence: duration is set not by consumption speed but by the gap between consumption speed and flow speed.

In Place or Side by Side

If the rebuild runs on top of the live projection, the read model is incomplete throughout it; the lag window is 2.11 hours, and tracking queries see missing rows during that time. The alternative is to build the new projection side by side with the old one and switch the read path to it once it finishes. The lag window is then 0, because the old projection keeps running throughout.

The cost is space: both projections sit in storage at once during the rebuild, carrying a second 140.16 GB copy; stored data peaks at 1,132.96 GB. This is another form of the trade-off repeated in every lesson of this course — the lag window closes in exchange for space. Which one is chosen depends on how long the projection can stay incomplete, a domain decision, not a technical one.

Summary

• The projection is derived from the log by folding; incremental maintenance and a from-scratch rebuild gave the same rows, and the partial rebuild built open shipments’ rows identically to the full rebuild.
• A projection’s cost depends not on the log’s length but on how many rows an event moves: the tracking projection writes 1.00 row per event and holds 200 rows, the point counter 1.57 rows and 5 rows.
• The projection adds 192 MB to daily growth and 140.16 GB to stored data: 1,168 → 1,360 MB/day, 852.64 → 992.80 GB. Against K01’s baseline the two decisions combined come to 1.39 times.
• The rebuild’s measure is events read: the full window is 2,336,000,000, the open window (OY3 = 5 days) 16,000,000 — a 146× difference.
• Duration comes from the gap between consumption speed and flow speed: at OY4 = 20 the open window is 2.11 hours, at 10 it is 4.44 hours, at 1.5 it is 80 hours, at 1 it never catches up.
• Rebuilding in place leaves a 2.11-hour lag window; rebuilding side by side brings that to 0 in exchange for 140.16 GB of temporary space (1,132.96 GB at peak).

Next Step

Across three lessons the lag window was treated as a resource measure: how many events, how many hours, how many gigabytes. But what sits inside that window is not a number, it is a user. A recipient corrects an address and reloads the page to see the old one; a carrier sends a state event and cannot find it on the dashboard; two parties change the same shipment at the same time and one never learns the other’s change was lost. The next lesson measures the window’s counterpart in the interface: how many stale values the user sees, how many rounds they wait, how many conflict notifications occur, and the requests each solution adds back to K01’s store load.