Federation

Replication split the read but left the same 194.44 write on every node and multiplied stored data by the replica count. Both limits come from the same place: all the data sits everywhere. Yet the data the store carries is not a single whole — pricing’s questions and delivery operations’ questions touch different records, as lesson 01’s scan measurement already showed: the period scan dropped from 21,000 records to 3,000 with no need for event detail at all.

Federation is splitting data by function rather than by key: each context gets its own store, and that store serves only that context’s questions. The split has nothing to do with the key — the same tracking number can appear in both stores; what it has to do with is which question that number is asked with.

Two Contexts, Two Stores

The shipment tracking and pricing service has two contexts. Delivery operations carries P1 and P2: single lookup by tracking number and the state event write from the carrier. Pricing carries P3: periodic batch scan by seller, tariff, contract, and invoice line.

The split is not clean, because the two contexts both need some of the same records. Pricing must know when a shipment was accepted and when it was delivered, but it does not need to know the route steps in between.

VD4 — the share of events that concern pricing is 2/7. Rationale: the invoice depends on acceptance and delivery time; only two of V4’s seven events affect the fee. Its sensitivity is given below at 3/7.

VD5 — when the shipment record splits across the two contexts, the shares are 500 and 600 bytes. Rationale: in V7’s 900-byte record, weight, volume, contract, and tariff go to pricing, and zone and route fields go to operations; the tracking number, seller ID, and period tag have to sit in both stores, so the total climbs to 1,100 bytes, and the duplicated share is 200 bytes. Its sensitivity is given below with 400 bytes of duplication.

Neither assumption is added to K01’s table.

How Many Stores a Split Query Touches

The setup below builds three stores in memory — the unfederated single store and two federated stores — and asks the same questions of both. What is measured is the number of stores touched, the records processed, and, in the federated layout, the number of keys carried between stores.

// federation/cross.mjs — the stores touched, records processed, and keys carried between stores
// by the same queries in a single store and two federated stores. A real engine runs with node:sqlite.
import { DatabaseSync } from "node:sqlite";

const SELLER = 20, DAY = 30, DAILY = 100, EVENTS = 7, LAST = EVENTS - 1;   // model scale
const [single, pricing, ops] = [0, 1, 2].map(() => new DatabaseSync(":memory:"));
single.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, day INT, tariff INT, zone INT);
CREATE INDEX s_seller ON shipment(seller, day);
CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`);
pricing.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, day INT, tariff INT);
CREATE INDEX s_seller ON shipment(seller, day);
CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`);
ops.exec(`CREATE TABLE shipment(trackingNo TEXT PRIMARY KEY, seller INT, zone INT);
CREATE TABLE event(trackingNo TEXT, seq INT, state TEXT); CREATE INDEX e_trackingNo ON event(trackingNo);`);

const counter = { single: 0, pricing: 0, ops: 0 };
for (const [name, db] of [["single", single], ["pricing", pricing], ["ops", ops]])
  db.function("count", { deterministic: false }, (x) => { counter[name] += 1; return x; });
const total = () => counter.single + counter.pricing + counter.ops;

const sS = single.prepare("INSERT INTO shipment VALUES(?,?,?,?,?)"), eS = single.prepare("INSERT INTO event VALUES(?,?,?)");
const sP = pricing.prepare("INSERT INTO shipment VALUES(?,?,?,?)"), eP = pricing.prepare("INSERT INTO event VALUES(?,?,?)");
const sO = ops.prepare("INSERT INTO shipment VALUES(?,?,?)"), eO = ops.prepare("INSERT INTO event VALUES(?,?,?)");
for (const d of [single, pricing, ops]) d.exec("BEGIN");
for (let v = 0; v < SELLER; v += 1) for (let day = 0; day < DAY; day += 1) for (let k = 0; k < DAILY; k += 1) {
  const trackingNo = `TR-${v}-${day}-${k}`;
  sS.run(trackingNo, v, day, 100 + v, day % 5); sP.run(trackingNo, v, day, 100 + v); sO.run(trackingNo, v, day % 5);
  for (let i = 0; i < EVENTS; i += 1) {
    eS.run(trackingNo, i, `state${i}`); eO.run(trackingNo, i, `state${i}`);
    if (i === 0 || i === LAST) eP.run(trackingNo, i, `state${i}`);   // VD4: acceptance and delivery only
  }
}
for (const d of [single, pricing, ops]) d.exec("COMMIT");
const count = (db, t) => db.prepare(`SELECT count(*) c FROM ${t}`).get().c;
console.log(`model: ${count(single, "shipment")} shipments; event records -> single store ${count(single, "event")}, ` +
  `ops ${count(ops, "event")}, pricing ${count(pricing, "event")}`);

const cross = () => {                                            // carry keys to ask the second store
  const a = pricing.prepare("SELECT count(trackingNo) AS trackingNo FROM shipment WHERE seller = ?").all(10).map((r) => r.trackingNo);
  const q = ops.prepare(`SELECT count(trackingNo) FROM event WHERE trackingNo = ? AND seq = ${LAST}`);
  for (const k of a) q.all(k);
  return a.length;
};
const JOBS = [
  ["P1 single lookup / federated", 1, () =>
    (ops.prepare("SELECT count(trackingNo) FROM event WHERE trackingNo = ? ORDER BY seq DESC LIMIT 3").all("TR-10-15-50"), 0)],
  ["P3 period scan / federated", 1, () =>
    (pricing.prepare("SELECT count(trackingNo) FROM shipment WHERE seller = ? AND day BETWEEN 0 AND 29").all(10), 0)],
  ["Q4 last route step / single store", 1, () => (single.prepare(
    `SELECT count(s.trackingNo), e.state FROM shipment s JOIN event e ON e.trackingNo = s.trackingNo WHERE s.seller = ? AND e.seq = ${LAST}`).all(10), 0)],
  ["Q4 last route step / federated", 2, cross],
  ["Q5 fee by delivery date / federated", 1, () => (pricing.prepare(
    `SELECT count(s.trackingNo), s.tariff FROM shipment s JOIN event e ON e.trackingNo = s.trackingNo WHERE s.seller = ? AND e.seq = ${LAST}`).all(10), 0)],
  ["Q5 if the delivery event were not copied", 2, cross],
];
console.log(`\n${"query / layout".padEnd(40)}${"stores".padStart(7)}${"records processed".padStart(19)}${"keys carried".padStart(15)}`);
for (const [name, stores, run] of JOBS) {
  counter.single = 0; counter.pricing = 0; counter.ops = 0;
  const keys = run();
  console.log(`${name.padEnd(40)}${String(stores).padStart(7)}${String(total()).padStart(19)}${String(keys).padStart(15)}`);
}
const bytes = Buffer.byteLength("TR-10-15-50");
console.log(`\nkey is ${bytes} bytes: in the federated layout, Q4 carries ` +
  `${(3000 * bytes) / 1000} kB between stores; 0 in the single store`);

model: 60000 shipments; event records -> single store 420000, ops 420000, pricing 120000

query / layout                           stores  records processed   keys carried
P1 single lookup / federated                  1                  7              0
P3 period scan / federated                    1               3000              0
Q4 last route step / single store             1               3000              0
Q4 last route step / federated                2               6000           3000
Q5 fee by delivery date / federated           1               3000              0
Q5 if the delivery event were not copied      2               6000           3000

key is 11 bytes: in the federated layout, Q4 carries 33 kB between stores; 0 in the single store

These numbers belong to the measurement class; they come from an engine running on this machine and are deterministic.

The first two rows carry the good news: each context’s own questions stay in its own store and cost exactly what they cost in the single store. Federation neither helps nor hurts these questions.

The fourth and fifth rows carry the real information. Q4 — the last route step of a seller’s shipments over a period sits at the intersection of both contexts: seller and period are pricing’s field, route step is operations’ field. In the single store this is one query; the engine performs the join internally and processes 3,000 records. In the federated layout the query splits in two: 3,000 keys are read from pricing, the application carries them and asks the delivery store, and total records processed climbs to 6,000. What is lost is the join itself; what replaces it is 3,000 keys carried between stores — 33 kilobytes. This is the cost federation actually imposes on the application: work the engine did for free, application code now pays for.

The last two rows measure VD4’s rationale. Q5 — fee by delivery date stays in a single store and finishes at 3,000 records, because a copy of the delivery event sits in the pricing store. Without that copy, the same question would fall into the same shape as Q4: 2 stores, 6,000 records, 3,000 keys carried. In other words, the duplicated 2/7 of events brings a cross-context query down to a single store. That is the measure of whether a context boundary is drawn correctly: the boundary sitting in the right place means the number of cross-context questions stays small.

Back to the Arithmetic

Duplication’s cost collects in two of K01’s growth rows.

// federation/cost.mjs — what a store split into two contexts costs on K01's rows
const V3 = 400_000, V4 = 7, V6 = 220, V7 = 900, V8 = 3, V12 = 730;   // K01 assumptions
const READ = 41.67, EVENT_WRITE = 97.22, SCAN = 833.33;              // K01 arithmetic
const GROWTH = 976, STORED = 712.48;                                  // K01 arithmetic
const DOCUMENT = 194.44;       // lesson 01: state event writes to 2 targets on the delivery side
const VD4 = 2 / 7;             // assumption: share of events that concern pricing
const VD5 = [500, 600];        // assumption: shares of the split shipment record (pricing, ops) bytes
const SHIPMENT_WRITE = 13.89;  // K01 derived: (V3 / 86,400) x V8 = shipment creation, peak records/s
const b = (x, n = 2) => x.toFixed(n);

const ops = SHIPMENT_WRITE + DOCUMENT + READ;
const pricing = SHIPMENT_WRITE + EVENT_WRITE * VD4;
console.log(`${"store".padEnd(24)}${"requests+ops/s".padStart(15)}${"scan records/s".padStart(16)}` +
  `${"daily MB".padStart(10)}${"stored GB".padStart(12)}`);
const opsMB = (V3 * VD5[1] + V3 * V4 * V6) / 1e6, pricingMB = (V3 * VD5[0] + V3 * V4 * VD4 * V6) / 1e6;
console.log(`${"delivery operations".padEnd(24)}${b(ops).padStart(15)}${"0".padStart(16)}` +
  `${b(opsMB).padStart(10)}${b((opsMB * V12) / 1000).padStart(12)}`);
console.log(`${"pricing".padEnd(24)}${b(pricing).padStart(15)}${b(SCAN).padStart(16)}` +
  `${b(pricingMB).padStart(10)}${b((pricingMB * V12) / 1000).padStart(12)}`);
console.log(`${"single store (unfederated)".padEnd(24)}${b(SHIPMENT_WRITE + DOCUMENT + READ).padStart(15)}` +
  `${b(SCAN).padStart(16)}${b(GROWTH).padStart(10)}${b(STORED).padStart(12)}`);

const totalMB = opsMB + pricingMB;
console.log(`\ntotal ops ${b(ops + pricing)} /s = ${b((ops + pricing) / (SHIPMENT_WRITE + DOCUMENT + READ))}x the single store`);
console.log(`daily written ${b(totalMB)} MB (K01: ${GROWTH}) = ${b(totalMB / GROWTH)}x`);
console.log(`stored ${b((totalMB * V12) / 1000)} GB (K01: ${STORED}) = ${b((totalMB * V12) / 1000 / STORED)}x`);
const path = (SHIPMENT_WRITE * 2 + EVENT_WRITE * (1 + VD4)) / (SHIPMENT_WRITE + EVENT_WRITE);
console.log(`length of the write path = ${b(path, 3)} stores (shipment 2, state event ${b(1 + VD4, 3)})`);

const sensitivity = (vd4, dup) => ((V3 * (V7 + dup) + V3 * V4 * V6 * (1 + vd4)) / 1e6);
console.log(`\nsensitivity: VD4 = 3/7 -> daily ${b(sensitivity(3 / 7, 200))} MB, ` +
  `stored ${b((sensitivity(3 / 7, 200) * V12) / 1000)} GB`);
console.log(`             duplicated identifier 400 bytes -> daily ${b(sensitivity(VD4, 400))} MB, ` +
  `stored ${b((sensitivity(VD4, 400) * V12) / 1000)} GB`);
console.log(`all three together: lesson 01's document store adds 5.76 GB, with lesson 02's three replicas ` +
  `${b(3 * ((totalMB * V12) / 1000 + 5.76))} GB`);

store                    requests+ops/s  scan records/s  daily MB   stored GB
delivery operations              250.00               0    856.00      624.88
pricing                           41.67          833.33    376.00      274.48
single store (unfederated)         250.00          833.33    976.00      712.48

total ops 291.67 /s = 1.17x the single store
daily written 1232.00 MB (K01: 976) = 1.26x
stored 899.36 GB (K01: 712.48) = 1.26x
length of the write path = 1.375 stores (shipment 2, state event 1.286)

sensitivity: VD4 = 3/7 -> daily 1320.00 MB, stored 963.60 GB
             duplicated identifier 400 bytes -> daily 1312.00 MB, stored 957.76 GB
all three together: lesson 01's document store adds 5.76 GB, with lesson 02's three replicas 2715.36 GB

These rows belong to the calculation class.

The batch scan comes out of the online store. Delivery operations’ scan column is zero; the four-hour job reading 833.33 records per second no longer shares a disk, memory, or lock manager with the tracking query. This is federation’s scaling rationale, and it fits in a single number: the record rate the online path sees falls from 833.33 to zero.

Total ops climbs to a 1.17× factor. The single store’s 250.00 ops/s splits into 250.00 and 41.67; the extra 41.67 is pricing writing its own copies. The write path’s length climbs to 1.375 stores: shipment creation writes to two stores, and a state event writes to an average of 1.286. A longer write path raises a durability question — what happens if one of the two stores refuses the write — but that question is not this course’s subject.

daily data growth MB and stored data GB climb together to a 1.26× factor. 976 MB becomes 1,232 MB, 712.48 GB becomes 899.36 GB. The growth has two sources and each can be read separately: duplicated identifier fields push the shipment record from 900 to 1,100 bytes, and duplicated acceptance and delivery events add 176 MB a day. If VD4 rose to a third, stored data would be 963.60 GB; if the duplicated identifier rose to 400 bytes, it would be 957.76 GB — the two assumptions carry about the same magnitude of effect.

The last row gives the combination of three decisions. Applied together, lesson 01’s document store, this lesson’s federation, and lesson 02’s three replicas bring stored data to 2,715.36 GB — a 3.81× factor over K01’s 712.48 GB. This is the common trait of data-layer decisions: all of them grow the same number, and their effects multiply, not add.

Summary

• Federation splits data by function rather than by key; the boundary is drawn by the context’s questions, and the same tracking number can appear in both stores.
• A context’s own questions stay in its own store, but the Q4 question at the intersection of both contexts climbs from 1 store to 2, from 3,000 records to 6,000, and carries 3,000 keys (33 kB) between stores.
• Duplicating 2/7 of events brings a cross-context question (Q5) down to a single store; the measure of a correct boundary is the number of cross-context questions.
• The batch scan comes out of the online store: the scan rate delivery operations’ store sees falls from 833.33 records/s to zero.
• Total ops climbs from 250.00 to 291.67 ops/s (a 1.17× factor), and the write path stretches to 1.375 stores.
• Duplication brings two of K01’s growth rows to a 1.26× factor (976 to 1,232 MB, 712.48 to 899.36 GB); combined with lessons 01 and 02’s decisions, stored data reaches 2,715.36 GB.

Next Step

Federation split the store’s work, but each context’s own data still sits whole on a single node. The delivery operations store carries 624.88 GB and takes 250.00 operations per second; these numbers do not shrink by splitting the context further, because there is no context left to split. The one path left is to split the data within itself: distribute the same context’s records across independent nodes by a key. The next lesson takes up that decision — how the key is chosen, how the choice changes the number of nodes each of the three access patterns touches, how many times over the busiest node carries what the others do, and how many bytes change hands when the node count changes.