--- title: 'Performance Anti-Patterns and Monitoring' source: 'https://academia.sh/en/courses/performance-and-monitoring' language: en updated: '2026-08-23T07:01:27+00:00' license: 'CC BY-SA 4.0' lessons: 16 outcomes: - 'Recognizes common performance anti-patterns from their symptoms' - 'Builds the monitoring categories completely' - 'Turns load-test results into a capacity decision' --- # Performance Anti-Patterns and Monitoring Noticing the failure and naming the slowness: building ten common performance anti-patterns into a diagnostic path that runs from symptom to cause, recognizing each pattern by its measurement and counting what the fix enlarges in return, building the monitoring categories completely, turning the question of what is worth measuring into an instrumentation decision, and tying load-test results to capacity estimates and design decisions to operating cost. ## Anti-Patterns Ten diagnoses from symptom to cause: computation done in the wrong layer, heavy work piled onto the main thread, a large number of small calls, reading more than is needed, expensive clients being re-created, all the data piled into a single store, a repeated expensive computation left uncached, missing isolation on a shared resource, blocking calls' effect on capacity, and a retry loop that grows a failure. ### Busy Database Source: https://academia.sh/en/courses/performance-and-monitoring/busy-database Separating two opposite causes of the same symptom with a single measurement: the store layer staying busy while the app layer sits idle, non-reducing work left in the store and reducing work moved to the app producing the identical symptom, the boundary ratio choosing the direction of the fix, and the condition under which a single number recommends the wrong plan. The previous course defended twenty patterns with the arithmetic of two separate days, but all of them stood on the same assumption: the failure would be noticed. A circuit breaker would know a threshold had been crossed; a failover would know a replica had gone down. How the noticing itself would happen was never designed. This course starts there, and not from failure but from the ordinary day itself, the one without any failure at all: a system can degrade without ever breaking, and degradation cannot be fixed until it has a name. This lesson, and the nine that follow it, run a three-step path. The **symptom** is a number; "slow" is not a symptom. **Differentiation** is a measurement that separates at least two causes producing the same symptom; without a measurement, what is done is a guess, and this course calls it exactly that. The **fix** removes the pattern and counts what grows in return. Until all three are complete, an **antipattern** has not been diagnosed. And no antipattern is wrong at every scale; each lesson bounds, with a number, the condition under which the same shape is correct. #### Symptom: One Layer Busy, the Other Idle The symptom shows up in the end-of-day pricing flow. K01's rough sizing calculation wrote this job as scanning 12,000,000 records over thirty days within a four-hour window: 833.33 records per second. The job does not fit the window. Two **utilization** readings exist at the same time and they point in opposite directions: the store layer's processor utilization peaks, while the application nodes' utilization stays low. Utilization was defined in M19/K01 `fundamental-properties/01` and is not redefined here. The first reflex is a guess, and because it is a guess, it gets named as one: "the store is not enough, a bigger machine is needed." The danger in that sentence is not that it is wrong — it is that it is **directionless**. The two causes below produce the same symptom and their fixes are opposites; half of every intervention made without picking a direction makes the symptom worse. **Cause A — non-reducing work in the store.** The store is doing work that never changes the result's record count: per-row string generation, formatting, decoding for presentation. Input records equal output records; the processor work spent does not shave a single byte off the data crossing the boundary. **Cause B — reducing work in the app.** The application pulls raw records and does the aggregation itself. The store does no per-row computation at all, but it still has to read, serialize, and send twelve million records. The store is still busy — just with a different kind of work. **Busy database** is the shared name for these two causes: computation done in the wrong layer. This is where the name alone turns out to be useless — it covers both causes, while the fix requires knowing which one applies. #### Four Plans, One Rig For differentiation, the same end-of-day job runs under four plans. There are two requests: the seller's **itemized statement** (one row per shipment) and the **invoice line** (one row per seller-day, K01's V13 = 100-item ratio). Each request is written with two layer choices. The measurement runs on a real engine and the model scale is M19/K04's scale: 20 sellers, 30 days, 100 shipments a day. Three things are counted — records crossing the boundary, row work done in each of the two layers (the number of computations run per row, counted with a counter function registered on the engine), and bytes crossing the boundary. Duration is not measured: duration depends on this particular machine, and it is not one of the three quantities being counted. ```js // layer/plan.mjs — the same end-of-day job in four plans: records crossing the boundary, // boundary ratio, row work done in each layer, and bytes moved. node:sqlite runs a real engine. import { DatabaseSync } from "node:sqlite"; const SELLER = 20, DAY = 30, DAILY = 100, TARIFF = 40, ZONE = 12; // M19/K04 model scale const N = SELLER * DAY * DAILY, INVOICE = SELLER * DAY; // 60000 shipments, 600 invoice lines const db = new DatabaseSync(":memory:"); db.exec(`CREATE TABLE shipment(no TEXT PRIMARY KEY, seller INT, day INT, state INT, weight REAL, rate REAL, coefficient REAL, discount REAL);`); db.exec("BEGIN"); const insert = db.prepare("INSERT INTO shipment VALUES(?,?,?,?,?,?,?,?)"); for (let s = 0; s < SELLER; s += 1) for (let d = 0; d < DAY; d += 1) for (let k = 0; k < DAILY; k += 1) { const ti = (s * 7 + d + k) % TARIFF, bi = (s + k) % ZONE; insert.run(`TR-${s}-${d}-${k}`, s, d, k % 5, 1 + (k % 30), 12 + (ti % 7) * 0.5, 1 + (bi % 5) * 0.1, (s % 9) * 0.01); } db.exec("COMMIT"); const AMOUNT = "weight * rate * coefficient * (1 - discount)"; const STATE = "CASE state WHEN 0 THEN 'accepted' WHEN 1 THEN 'transferred' WHEN 2 THEN 'out-for-delivery' " + "WHEN 3 THEN 'delivered' ELSE 'pending' END"; const NAME = ["accepted", "transferred", "out-for-delivery", "delivered", "pending"]; const SQL = (i) => ({ // i: wrapper that counts row work D1: `SELECT no, ${i(`printf('%s / day %02d', no, day)`)} AS title, ${i(`printf('%.2f', ${i(AMOUNT)})`)} AS amount, ${i(`printf('%.1f kg', weight)`)} AS weight, ${i(STATE)} AS state FROM shipment`, D2: `SELECT no, day, ${i(AMOUNT)} AS amount, weight, state FROM shipment`, F1: `SELECT seller, day, count(*) AS items, sum(${i(AMOUNT)}) AS amount FROM shipment GROUP BY seller, day`, F2: `SELECT seller, day, weight, rate, coefficient, discount FROM shipment`, }); const APP = { // row work done in the app layer D1: (r) => 0, D2: (r) => { let c = 0; for (const x of r) { c += 4; void [`${x.no} / day ${String(x.day).padStart(2, "0")}`, x.amount.toFixed(2), `${x.weight.toFixed(1)} kg`, NAME[x.state]]; } return c; }, F1: (r) => { let c = 0; for (const x of r) { c += 2; void [x.amount.toFixed(2), `${x.items} items`]; } return c; }, F2: (r) => { const t = new Map(); let c = 0; for (const x of r) { c += 1; const k = `${x.seller}/${x.day}`; const v = t.get(k) ?? { items: 0, amount: 0 }; v.items += 1; v.amount += x.weight * x.rate * x.coefficient * (1 - x.discount); t.set(k, v); } for (const v of t.values()) { c += 2; void [v.amount.toFixed(2), `${v.items} items`]; } return c; }, }; const NEEDED = { D1: N, D2: N, F1: INVOICE, F2: INVOICE }; // record count in the request's result let counter = 0; db.function("i", (x) => { counter += 1; return x; }); const measure = {}; for (const [name, query] of Object.entries(SQL((e) => `i(${e})`))) { counter = 0; const rows = db.prepare(query).all(); const store = counter; const bytes = Buffer.byteLength(JSON.stringify(rows.map((r) => Object.values(r)))); measure[name] = { crossed: rows.length, needed: NEEDED[name], ratio: rows.length / NEEDED[name], store, app: APP[name](rows), bytes }; } const PLAN = ["D1", "D2", "F1", "F2"]; const LABEL = { D1: "D1 statement, formatting in store", D2: "D2 statement, formatting in app", F1: "F1 invoice, aggregation in store", F2: "F2 invoice, aggregation in app" }; const say = (x) => x.toLocaleString("en-US"); console.log(`model: ${say(N)} shipments, ${SELLER} sellers, ${DAY} days; the statement request returns ` + `${say(N)} rows, the invoice request returns ${INVOICE} rows; the records scanned are ${say(N)} in all four plans`); console.log(`\n${"plan".padEnd(35)}${"records crossed".padStart(17)}${"needed".padStart(10)}` + `${"boundary ratio".padStart(16)}${"store row work".padStart(16)}${"app".padStart(12)}` + `${"bytes crossed".padStart(15)}${"per record".padStart(13)}`); for (const p of PLAN) { const o = measure[p]; console.log(LABEL[p].padEnd(35) + say(o.crossed).padStart(17) + say(o.needed).padStart(10) + o.ratio.toFixed(2).padStart(16) + say(o.store).padStart(16) + say(o.app).padStart(12) + say(o.bytes).padStart(15) + (o.bytes / o.crossed).toFixed(1).padStart(13)); } const SCALE = 12_000_000 / N, WINDOW = 4 * 3600; // K01: 12,000,000 records over 30 days, 4 hours console.log(`\nat K01 scale (scale ${SCALE}, window ${WINDOW} s, scan rate ` + `${(12e6 / WINDOW).toFixed(2)} records/s); boundary bandwidth is scaled so F2 equals K01's 6 Mbit/s`); console.log(`${"plan".padEnd(35)}${"store row work/s".padStart(19)}${"app/s".padStart(14)}` + `${"boundary bandwidth Mbit/s".padStart(27)}`); for (const p of PLAN) { const o = measure[p]; console.log(LABEL[p].padEnd(35) + ((o.store * SCALE) / WINDOW).toFixed(2).padStart(19) + ((o.app * SCALE) / WINDOW).toFixed(2).padStart(14) + ((6 * o.bytes) / measure.F2.bytes).toFixed(3).padStart(27)); } console.log(`\nplan-pair selection under KK1 (a unit of store row work costs k times an app unit)`); console.log(`${"plan pair".padEnd(14)}${"k=2".padStart(12)}${"k=6".padStart(12)}${"k=12".padStart(12)}` + `${"records crossed".padStart(18)}${"boundary bandwidth Mbit/s".padStart(27)}`); for (const [a, b] of [["D1", "F1"], ["D2", "F1"], ["D1", "F2"], ["D2", "F2"]]) { const store = measure[a].store + measure[b].store, app = measure[a].app + measure[b].app; const bytes = measure[a].bytes + measure[b].bytes; console.log(`${a} + ${b}`.padEnd(14) + [2, 6, 12].map((k) => say(store * k + app).padStart(12)).join("") + say(measure[a].crossed + measure[b].crossed).padStart(18) + ((6 * bytes) / measure.F2.bytes).toFixed(3).padStart(27)); } ``` ``` model: 60,000 shipments, 20 sellers, 30 days; the statement request returns 60,000 rows, the invoice request returns 600 rows; the records scanned are 60,000 in all four plans plan records crossed needed boundary ratio store row work app bytes crossed per record D1 statement, formatting in store 60,000 60,000 1.00 300,000 0 4,123,401 68.7 D2 statement, formatting in app 60,000 60,000 1.00 60,000 240,000 1,988,049 33.1 F1 invoice, aggregation in store 600 600 1.00 60,000 1,200 13,407 22.3 F2 invoice, aggregation in app 60,000 600 100.00 0 61,200 1,302,519 21.7 at K01 scale (scale 200, window 14400 s, scan rate 833.33 records/s); boundary bandwidth is scaled so F2 equals K01's 6 Mbit/s plan store row work/s app/s boundary bandwidth Mbit/s D1 statement, formatting in store 4166.67 0.00 18.994 D2 statement, formatting in app 833.33 3333.33 9.158 F1 invoice, aggregation in store 833.33 16.67 0.062 F2 invoice, aggregation in app 0.00 850.00 6.000 plan-pair selection under KK1 (a unit of store row work costs k times an app unit) plan pair k=2 k=6 k=12 records crossed boundary bandwidth Mbit/s D1 + F1 721,200 2,161,200 4,321,200 60,600 19.056 D2 + F1 481,200 961,200 1,681,200 60,600 9.220 D1 + F2 661,200 1,861,200 3,661,200 120,000 24.994 D2 + F2 421,200 661,200 1,021,200 120,000 15.158 ``` The numbers in the table belong to the **measurement** class; the rows converted to K01 scale belong to the **calculation** class. The boundary bandwidth column carries no absolute claim about bytes: the model's raw row is narrower than K01's 900-byte shipment record, so the bandwidth was scaled by pinning plan F2 to the 6 Mbit/s that K01 wrote down. What travels across scales is the **ratio** between plans, not the absolute figure. #### The Boundary Ratio Picks the Direction The third column is the diagnosis itself. The **boundary ratio** is the number of records a request pushes across the boundary, divided by the number of records actually present in that request's result. Three of the four plans show 1.00; F2 shows 100.00. When the ratio is 1.00, the store has already reduced, and if it is still busy, the guilty party is non-reducing work. D1 shows this exactly: the boundary ratio is 1.00, but store row work is 300,000 — five per record. D2 produces the same result and the store's work drops to 60,000, one per record. The difference is formatting work, and that work does not shave off even one crossed record — it actually **increases** it: bytes crossed drop from 4,123,401 to 1,988,049, from 68.7 bytes per record to 33.1. The strings the store produces are wider than the numbers they're produced from. The direction of the fix is **up**. When the ratio is far above 1, the reduction is happening in the wrong layer. In F2, store row work is zero — the store does no computation at all — but it still reads, serializes, and sends 60,000 records, even though the request's result is 600 rows. F1 delivers the same result with 600 records and cuts boundary bandwidth from 6.000 to 0.062 Mbit/s: **96.8 times**. The direction of the fix is **down**. Both causes produce the "store busy, app idle" symptom, but one calls for moving work up and the other for moving it down. The boundary ratio separates these two cases with a single number, and what it separates is the **direction** of the fix. #### One Number Does Not Diagnose Finding the direction is not enough to pick a plan. The last table compares plan pairs under one assumption. **KK1 — a unit of row work done in the store costs $k$ times a unit of row work done in the app; the baseline value is $k = 6$.** Rationale: the application layer is stateless and scales horizontally by adding nodes (M19/K01 `fundamental-properties/03`), while in the store layer writes are single-pointed and adding cores means changing machines. A unit of work in the two layers is not bought at the same price. Sensitivity is given in the $k = 2$ and $k = 12$ columns. The weighted row-work measure recommends the same pair at all three $k$ values: **D2 + F2**, at 661,200 units (k = 6). This recommendation is wrong. That same row shows 120,000 records crossing the boundary and 15.158 Mbit/s of boundary bandwidth; the D2 + F1 pair shows 60,600 records and 9.220 Mbit/s. The row-work measure thinks F2 is cheap, because the work F2 loads onto the store is not row work at all — it is reading, serializing, and moving data, and none of that shows up in that column. This is the course's first rule: **a symptom is not explained by a single number.** The store is busy in two separate currencies — computation per row and bytes crossing the boundary — and if a plan is chosen by looking at only one, the other grows. The correct choice is D2 + F1: unnecessary work has been pulled out of the layer for both requests. #### The Condition Where Staying in the Store Is Correct Computation being done in the store is not forbidden; F1 does exactly that and is the cheapest of the four plans. The condition is measurable: **work done in the store is justified to the extent that it reduces the record count crossing the boundary.** In F1, 60,000 units of row work cut the crossed record count from 60,000 to 600 — a reduction of 100 units per record, which is exactly K01's V13 = 100 ratio. In D1, 240,000 extra units of row work do not reduce the crossed record count at all; they push bytes up to 2.07 times. The condition is bounded by a scale. If the number of items per invoice line dropped from 100 to 1 — that is, if every shipment produced its own invoice line — the 96.8-times bandwidth gap between F1 and F2 would vanish, and doing the aggregation in the store would have no payoff left. The same shape, in the same system, crosses to the wrong side at a different breakdown ratio. #### What Grows in Return for the Fix The move from D1 to D2 cut store row work from 300,000 to 60,000; app row work rose from 0 to 240,000. At K01 scale, that means dropping from 4166.67 units of row work per second in the store to 833.33, and rising from 0 to 3333.33 in the app. At KK1's $k = 6$, the trade pays off; it would not at $k = 1$ — meaning the fix is justified by the layers' **scaling asymmetry**, not by the work itself. The second payoff is in the code, and it is not counted, but it is visible: once the formatting rule leaves the store, every consumer of the query has to rewrite it on its own side. A rule that lived in one place in the store now lives in as many places as there are consumers in the app. This cost does not make D2 wrong; failing to name the cost does. #### Summary - Diagnosis is three steps: the symptom is a number, differentiation is a measurement that separates two causes of the same symptom, and the fix counts what grows in return. What is done without a measurement is a guess. - Busy database has two causes that produce the same symptom: non-reducing work in the store (D1) and reducing work in the app (F2). Their fixes are opposites. - The boundary ratio picks the direction: at 1.00, work moves up (D1 → D2, bytes crossed 4,123,401 → 1,988,049); far above 1, work moves down (F2 → F1, bandwidth 6.000 → 0.062 Mbit/s, 96.8 times). - A single number does not diagnose: the weighted row-work measure recommends D2 + F2 (661,200, k = 6), yet that pair crosses the boundary with 120,000 records and 15.158 Mbit/s; the correct choice is D2 + F1. - Computation staying in the store is justified to the extent that it reduces the crossed record count: in F1, 60,000 units of row work cut the record count by a factor of 100. At a breakdown ratio of 1, the same choice would flip to wrong. - The payoff of the fix grows in the app layer: row work goes from 0 to 240,000 (0 to 3333.33 per second at K01 scale), and the formatting rule spreads from one place to as many places as there are consumers. #### Next Step The boundary ratio settled which of two layers should do the work: store or app. But there is one more end of the chain, and it has never been measured. Plan D2 pulled formatting out of the store and put it in the app; the same work can go one step further and land on the **client** — the server sends the raw record, the browser formats it. This is the third layer of the same trade, and there the name of the measurement changes: work in the app layer is distributed by adding nodes; on the client, there is no node to add. The next lesson takes on that layer: how the load the server sends grows the work on the client, the measurement that tells apart a server-caused slowdown from a client-caused one, and what grows on the server in return for shrinking the load. ### Busy Front End Source: https://academia.sh/en/courses/performance-and-monitoring/busy-front-end The load the server sends growing the work on the client: the time visible to the user growing while server response time stays flat, load growth and client code growth producing the same symptom being separated by the work-per-record measurement, the 26.8-times gain from moving sorting to the server, and the price moving formatting pays against the cache key. The boundary ratio settled which of two layers should do the work: store or app. There is one more end of the chain, and there the trade changes its rule: work in the app layer is distributed by adding nodes; on the client, there is no node to add. This lesson takes on that third layer and runs the same three steps — the symptom is a number, a measurement separates two causes, and what grows in return for the fix is counted. The main thread, long tasks, and user-centric metrics were defined and measured in the Frontend Quality course; they are not redefined here. The question here is a system design question: **by what multiplier does the load the server sends grow the work on the client, and by what measurement is that multiplier seen.** #### Symptom: The Server Is Calm, the User Is Waiting The seller panel opens its list of open shipments. The time visible to the user has grown over the last two releases. The server-side response-time metric has not changed; the edge meets K01's peak load with the same numbers. The response's byte count was also checked, and the share taken by transfer has not changed. The symptom comes down to this: **the growth all happens after the work has already moved to the client.** At this point there are two causes, and neither shows up in the server's metrics. **Cause A — the server increased the record count it sends.** The client code has not changed, but the load has grown; the same code runs over more records. **Cause B — the client added work per record.** The load is the same, but the new release does two extra steps per record. An answer given without measuring is a guess, and the guess has two directions: the sentences "the server is sending too much data" and "the front end got heavier" are both drawn from the same symptom, and one of them is wrong. #### The Measurement Rig The rig below is a **model**. Browser time is not measured and not invented; what is counted is the steps the client has to perform on its single thread: parsed fields, transform steps, and sort comparisons. All three actually run, the counters count real calls, and the input to sorting comes from a seeded generator. **KK2 — rows visible on one screen of the panel: 25.** Rationale: the list view shows a page's worth of rows, and the rest arrives by scrolling. Its sensitivity turns into a call count in the next lesson. ```js // front-end/main-thread.mjs — how the load the server sends grows the work on the front end. // MODEL: the unit of work is an abstract step, duration is not measured; what is counted is // parsed fields, transform steps, and sort comparisons. All three actually run. const EVENT = 7, VISIBLE = 25, ZONE = 12, ROUTE = 40; // K01 V4 = 7 state events; KK2 = 25 rows function generate(N) { // the full data the server holds let s = 20260801 % 2147483647; // seed is visible; sort order is deterministic const rand = () => (s = (s * 48271) % 2147483647) % 100000; const shipments = []; for (let i = 0; i < N; i += 1) { const events = []; for (let j = 0; j < EVENT; j += 1) events.push({ code: (i * 3 + j) % 5, ts: rand(), zone: (i + j) % ZONE, route: `R-${(i + j) % ROUTE}`, carrier: (i + j) % 6 }); shipments.push({ no: `TR-${i}`, seller: i % 20, weight: 1 + (i % 30), updatedAt: rand(), events }); } return shipments; } const NAME = ["accepted", "transferred", "out-for-delivery", "delivered", "pending"]; function server(N, plan) { // the load leaving the edge const g = generate(N); if (plan === "raw") return g; const top = [...g].sort((a, b) => b.updatedAt - a.updatedAt).slice(0, VISIBLE); return top.map((x) => ({ no: x.no, updatedAt: x.updatedAt, events: [...x.events].sort((a, b) => b.ts - a.ts).slice(0, 3) .map((o) => (plan === "formatted" ? { line: `${NAME[o.code]} / ${o.route} / ${o.ts}` } : { code: o.code, ts: o.ts, route: o.route })) })); } const countFields = (x) => Array.isArray(x) ? x.reduce((a, y) => a + countFields(y), 0) : (x && typeof x === "object") ? Object.values(x).reduce((a, y) => a + countFields(y), 0) : 1; function client(text, { transform, sort }) { const s = { parsing: 0, transform: 0, comparison: 0, paint: 0 }; const data = JSON.parse(text); s.parsing = countFields(data); let list = data; if (sort) { // shipment list: N log N list = [...data].sort((a, b) => { s.comparison += 1; return b.updatedAt - a.updatedAt; }); for (const x of list) [...x.events].sort((a, b) => { s.comparison += 1; return b.ts - a.ts; }); } if (transform > 0) for (const x of data) for (const o of x.events) { // format first, then slice s.transform += transform; void [NAME[o.code] ?? "", String(o.ts).padStart(6, "0"), `${o.route}`, `${o.zone ?? 0}`, `${o.carrier ?? 0}`, `${o.code ?? 0}`].slice(0, transform); } for (const x of list.slice(0, VISIBLE)) { s.paint += 3; void x.no; } s.total = s.parsing + s.transform + s.comparison + s.paint; return s; } const SCENARIO = [ ["raw load, client v1", 200, "raw", { transform: 4, sort: true }], ["raw load, client v1", 400, "raw", { transform: 4, sort: true }], ["raw load, client v2", 200, "raw", { transform: 6, sort: true }], ["sorted load, client v1", 200, "sorted", { transform: 4, sort: false }], ["formatted load", 200, "formatted", { transform: 0, sort: false }], ]; const say = (x) => x.toLocaleString("en-US"); console.log(`model: a shipment carries ${EVENT} state events (K01 V4), ${VISIBLE} rows show ` + `on screen (KK2); the unit of work is an abstract step, duration is not measured`); console.log(`\n${"scenario".padEnd(26)}${"N".padStart(6)}${"records crossed".padStart(17)}` + `${"parsing".padStart(12)}${"transform".padStart(11)}${"comparison".padStart(13)}` + `${"total work".padStart(12)}${"per record".padStart(13)}${"per visible row".padStart(18)}`); const r = {}; for (const [name, N, plan, opts] of SCENARIO) { const load = server(N, plan), text = JSON.stringify(load); const s = client(text, opts); r[`${name}|${N}`] = { ...s, crossed: load.length, bytes: Buffer.byteLength(text) }; console.log(name.padEnd(26) + String(N).padStart(6) + say(load.length).padStart(17) + say(s.parsing).padStart(12) + say(s.transform).padStart(11) + say(s.comparison).padStart(13) + say(s.total).padStart(12) + (s.total / load.length).toFixed(2).padStart(13) + (s.total / VISIBLE).toFixed(1).padStart(18)); } console.log(`\nscaling (raw load, client v1): total work multiplier as N doubles`); console.log(`${"N".padStart(6)}${"total work".padStart(12)}${"x".padStart(8)}` + `${"per record".padStart(14)}${"comparison/N".padStart(15)}`); let previous = 0; for (const N of [50, 100, 200, 400, 800]) { const s = client(JSON.stringify(server(N, "raw")), { transform: 4, sort: true }); console.log(String(N).padStart(6) + say(s.total).padStart(12) + (previous ? (s.total / previous).toFixed(3) : "-").padStart(8) + (s.total / N).toFixed(2).padStart(14) + (s.comparison / N).toFixed(2).padStart(15)); previous = s.total; } const a = r["raw load, client v1|200"], b = r["raw load, client v1|400"]; const c = r["raw load, client v2|200"], d = r["sorted load, client v1|200"], e = r["formatted load|200"]; console.log(`\ndifferentiating measurement — work per record: A (server N 200 -> 400) ` + `${(a.total / a.crossed).toFixed(2)} -> ${(b.total / b.crossed).toFixed(2)}, ` + `B (client v1 -> v2, N fixed) ${(a.total / a.crossed).toFixed(2)} -> ${(c.total / c.crossed).toFixed(2)}`); console.log(`fix: total work in the sorted load ${say(a.total)} -> ${say(d.total)} ` + `(${(a.total / d.total).toFixed(1)}x), in the formatted load ${say(e.total)} ` + `(${(a.total / e.total).toFixed(1)}x); bytes crossed ${say(a.bytes)} -> ${say(d.bytes)} -> ${say(e.bytes)}`); const EDGE = 513.89, HIT = 0.9; // K01 peak edge requests/s, K02 cache hit rate console.log(`\ncondition for moving the work to the server: cacheable work runs ${(EDGE * (1 - HIT)).toFixed(2)} times/s, ` + `non-cacheable work runs ${EDGE.toFixed(2)} times/s (ratio ${(1 / (1 - HIT)).toFixed(2)}); ` + `on the client both cases run once per user`); ``` ``` model: a shipment carries 7 state events (K01 V4), 25 rows show on screen (KK2); the unit of work is an abstract step, duration is not measured scenario N records crossed parsing transform comparison total work per record per visible row raw load, client v1 200 200 7,800 5,600 3,920 17,395 86.97 695.8 raw load, client v1 400 400 15,600 11,200 8,243 35,118 87.80 1404.7 raw load, client v2 200 200 7,800 8,400 3,920 20,195 100.97 807.8 sorted load, client v1 200 25 275 300 0 650 26.00 26.0 formatted load 200 25 125 0 0 200 8.00 8.0 scaling (raw load, client v1): total work multiplier as N doubles N total work x per record comparison/N 50 4,321 - 86.42 17.92 100 8,649 2.002 86.49 18.74 200 17,395 2.011 86.97 19.60 400 35,118 2.019 87.80 20.61 800 70,963 2.021 88.70 21.61 differentiating measurement — work per record: A (server N 200 -> 400) 86.97 -> 87.80, B (client v1 -> v2, N fixed) 86.97 -> 100.97 fix: total work in the sorted load 17,395 -> 650 (26.8x), in the formatted load 200 (87.0x); bytes crossed 94,428 -> 3,871 -> 3,896 condition for moving the work to the server: cacheable work runs 51.39 times/s, non-cacheable work runs 513.89 times/s (ratio 10.00); on the client both cases run once per user ``` #### Work Per Record Separates the Cause The differentiating measurement is **work per record**: total work divided by the number of records the server sends. The two causes separate on this single number. In Cause A, the server goes from 200 records to 400. Total work goes from 17,395 to 35,118 — the user feels the symptom. But work per record goes from 86.97 to 87.80; a 0.95 percent change. The client code has not changed, and the measurement says exactly that. In Cause B, the records sent stay at 200, and total work goes from 17,395 to 20,195. Work per record goes from 86.97 to 100.97; a 16.1 percent change. The load has not changed, the code has. **Same symptom, a total growing in the same direction, a ratio pointing the opposite way.** Looking at the total does not diagnose; dividing does. The scaling table says a third thing. As N doubles, total work grows by a multiplier that climbs from 2.002 to 2.021, and work per record rises from 86.42 to 88.70. That does not mean the code changed; the comparison column gives the reason: comparisons per record rise from 17.92 to 21.61. The client sorts the list itself, and sorting does not grow linearly with the record count. So load growth also grows work per record a little — and if it is known that this share is **logarithmic** while the code-change share is a **step**, the two causes can still be told apart. #### Two Fixes, One Difference The real magnitude of the symptom is in the last column. In the raw load, main-thread work per visible row is 695.8 units; the screen shows 25 rows, and the client parses and formats all 200 records, then slices them down. This is the antipattern itself: **the work scales with what was sent, not with what is shown.** When the server takes over sorting and selection, total work drops from 17,395 to 650 — **26.8 times**. Work per visible row falls from 695.8 to 26.0. If the server also takes over formatting, work drops to 200 — 87.0 times against the raw load, 3.25 times against the sorted load. The two fixes are not the same thing, and the last line gives the difference. The move to the sorted load cuts bytes crossed from 94,428 to 3,871. The move to the formatted load raises bytes from 3,871 to 3,896 — the response does not shrink once formatting is done on the server. So the second step gains only work units; it carries a third cost, and that cost is the subject of the next section. #### The Condition Where Leaving the Work on the Client Is Correct Doing work on the client is not a mistake; moving it to the server has a price too, and that price can be written as a number. On the client, a piece of work runs **once per user**, and it runs on the user's own hardware. Once moved to the server, the same work runs **513.89 times a second** at K01's peak load — inside a single resource pool. What determines the condition is whether the work is **cacheable**. K02's edge cache covers nine-tenths of reads; work moved to the server runs 51.39 times a second if it is cacheable, 513.89 times if it is not. The ratio is 10.00. Sorting and selection are not user-specific; the same seller's same list is the same for everyone, it is cacheable, and it should be moved. Formatting depends on the user's locale; if it is moved, the response becomes user-specific, and the cache key splits into as many pieces as that dimension has values. To hold the same hit rate, the cache has to be enlarged by that same multiple. This is why the measurement separates the two fixes: the **26.8-times gain is taken; the 3.25-times gain is not**. What grows in return for the second is measurable and large; what the first costs in return is only a sort that runs 51.39 times a second on the server. #### What Grows in Return Moving sorting and selection to the server has three payoffs to count. The server now sorts 51.39 times a second — behind the cache, stacking on top of K01's load of 138.89 requests a second reaching the store. Second, the decision of which 25 records to show has moved to the server; as the user moves through the list, that decision spawns a new request each time. Third, the client no longer holds the full list: a local filter or a local re-sort now has to go to the server as well. The last two payoffs point in the same direction, and they are the subject of the next lesson. #### Summary - The symptom does not show up in server metrics: response time and bytes hold steady while the time visible to the user grows. Both load growth and client code growth produce the identical symptom. - The differentiating measurement is work per record: it goes 86.97 → 87.80 (0.95 percent) as the load rises from 200 to 400, and 86.97 → 100.97 (16.1 percent) as the code moves from v1 to v2. - Load growth also grows work per record a little, but logarithmically: comparisons per record rise from 17.92 to 21.61, because sorting is done on the client. - The antipattern's measure is work per visible row: 695.8 units in the raw load, though the screen shows only 25 rows. Moving sorting and selection to the server drops total work by 26.8 times. - Moving formatting as well gains another 3.25 times, but it raises bytes from 3,871 to 3,896 and makes the response user-specific; the cache key splits. - The condition is cacheability: work moved to the server runs 51.39 times a second if it is cacheable, 513.89 times if it is not (a ratio of 10.00); on the client, both cases run once per user. #### Next Step The server now sends 25 records instead of 200, and it decides for itself which 25 to show. As the user moves through the list, that decision has to be renewed: every scroll is a request, every filter is a request, and opening a row for detail is one request more. The load shrank, the **call count** grew, and that count has never been measured. The next lesson takes on that count: the trade-off between call count and bytes moved, the record count at which fixed per-call cost starts to dominate bytes moved, and when a single large call is worse than many small ones. ### Chatty I/O Source: https://academia.sh/en/courses/performance-and-monitoring/chatty-io The trade-off between call count and bytes moved: measuring the same panel open at three call granularities, splitting the useful share of bytes moved into an envelope share and an unused-payload share, the usage ratio below which a single large call is worse, and why a high cache hit rate does not mean fewer bytes. The previous lesson shrank the load: the server now sends 25 records instead of 200, and it decides for itself which 25 to show. That decision had a cost, and the cost was never counted. As the user moves through the list, the decision is renewed; opening a row asks for its detail; applying a filter re-fetches the list. **Call count** grew. This is the service-call-scale counterpart of the **N+1 query problem** established in the Data Access Layer and Business Logic course. The mechanics there — query count growing with record count, batch fetching, joining — are not repeated here. The question here is different: **by how much do the bytes moved grow in return for cutting call count, and by what measurement is the point where this trade-off reverses found.** #### Symptom: A Slowdown That Shows Up in No Metric The panel open is slow end to end. Every call's server-side duration is small and constant; none crosses an edge threshold. Total useful payload is around nine kilobytes, so transfer cannot be the explanation. On the store side, the boundary ratio is 1.00 — the first lesson's measurement is clean. The symptom slips through every metric. **Chatty I/O** is a job split into a large number of small calls. But naming it is not enough again, because the opposite arrangement produces the same symptom. **Cause A — call count.** Every call carries an **envelope** independent of its body: the request line, identity, trace context, content headers, response headers. The envelope is fixed; it multiplies with the call count. And if calls wait on each other, the **round** count grows too. **Cause B — over-packing.** Call count has been cut to one, but that single call also carries what is not needed. The envelope share is near zero, and most of what moves is never used. Both produce the symptom "calls are fast but the open is slow," and their fixes are opposites. #### The Measurement Rig The rig below is a **model**: a round is an abstract step and duration is not measured. What is measured is bytes, and the envelope's byte count is not invented — it is actually computed from the header text in the code. **KK3 — calls the client can hold open at once: 6.** Rationale: dependent calls do not complete in one round; they complete in as many rounds as this limit divides them into. Its sensitivity shows up in the round column. Three granularities are compared. **G1** makes one call each for the list, the seller, and the zone, plus one detail call for every row opened. **G2** fetches the list and all twenty-five details in one call, leaving the seller and the zone separate. **G3** fetches everything in one call. The variable $k$ is how many of the twenty-five rows actually have their detail opened. ```js // calls/granularity.mjs — the same panel open measured at three call granularities. Measure: // the useful share of bytes moved, and its two components (envelope share, unused-payload share). // MODEL: a round is an abstract step, duration is not measured; the envelope's bytes are measured // from the header text below. const VISIBLE = 25, EVENT = 7, TARIFF = 40, ZONE = 12; // KK2 = 25 rows, K01 V4 = 7 events const CONCURRENCY = 6; // KK3: calls the client keeps open at once const ENVELOPE = ["GET /panel/list HTTP/1.1", "host: edge", "accept: application/json", "accept-encoding: gzip", "authorization: Bearer ", "traceparent: ", "HTTP/1.1 200 OK", "content-type: application/json", "cache-control: max-age=30", "etag: ", "content-length: ", "traceparent: "].join("\r\n") + "\r\n\r\n"; const ENV = Buffer.byteLength(ENVELOPE); const bytes = (x) => Buffer.byteLength(JSON.stringify(x)); const list = () => Array.from({ length: VISIBLE }, (_, i) => ({ no: `TR-${i}`, updatedAt: (i * 97) % 100000, state: i % 5, weight: 1 + (i % 30) })); const detail = (i) => ({ no: `TR-${i}`, events: Array.from({ length: EVENT }, (_, j) => ({ code: (i * 3 + j) % 5, ts: (i * 97 + j * 13) % 100000, route: `R-${(i + j) % 40}` })) }); const seller = () => ({ name: "seller-10", contract: 0.07, tariff: Array.from({ length: TARIFF }, (_, i) => ({ id: i, rate: 12 + (i % 7) * 0.5 })) }); const zone = () => ({ zones: Array.from({ length: ZONE }, (_, i) => ({ id: i, coefficient: 1 + (i % 5) * 0.1 })) }); const L = bytes(list()), S = bytes(seller()), Z = bytes(zone()); const A = Array.from({ length: VISIBLE }, (_, i) => bytes(detail(i))); const firstK = (k) => A.slice(0, k).reduce((a, x) => a + x, 0); const ALL_DETAIL = firstK(VISIBLE); // k: how many of the 25 rows actually have their detail opened const PLAN = { "G1 chatty": (k) => ({ calls: 3 + k, rounds: 1 + Math.ceil(k / CONCURRENCY), payload: L + S + Z + firstK(k) }), "G2 batched": (k) => ({ calls: 3, rounds: 1, payload: L + S + Z + ALL_DETAIL }), "G3 single call": (k) => ({ calls: 1, rounds: 1, payload: L + S + Z + ALL_DETAIL }), }; const measure = (name, k) => { const p = PLAN[name](k), envelope = p.calls * ENV, moved = p.payload + envelope; const used = L + S + Z + firstK(k); return { ...p, envelope, moved, used, envelopeShare: envelope / moved, unusedShare: (p.payload - used) / moved, usefulShare: used / moved }; }; const NAMES = Object.keys(PLAN), say = (x) => x.toLocaleString("en-US"); console.log(`model: the panel shows ${VISIBLE} rows, envelope ${ENV} bytes (measured from the header ` + `text below), concurrent call limit ${CONCURRENCY} (KK3); list ${L}, seller ${S}, zone ${Z}, ` + `${VISIBLE} details ${say(ALL_DETAIL)} bytes`); for (const k of [25, 3]) { console.log(`\n${k} of the ${VISIBLE} rows have their detail opened`); console.log(`${"plan".padEnd(16)}${"calls".padStart(7)}${"rounds".padStart(8)}${"bytes moved".padStart(14)}` + `${"envelope share".padStart(16)}${"unused payload".padStart(16)}${"useful share".padStart(14)}`); for (const name of NAMES) { const o = measure(name, k); console.log(name.padEnd(16) + String(o.calls).padStart(7) + String(o.rounds).padStart(8) + say(o.moved).padStart(14) + o.envelopeShare.toFixed(4).padStart(16) + o.unusedShare.toFixed(4).padStart(16) + o.usefulShare.toFixed(4).padStart(14)); } } console.log(`\nwhich plan moves less, depending on how many details are opened (cold open)`); console.log(`${"opened".padStart(7)}${"G1 bytes".padStart(11)}${"G2 bytes".padStart(11)}` + `${"G1/G2".padStart(8)}${"G1 rounds".padStart(11)}${"moves least".padStart(17)}`); for (const k of [1, 3, 6, 12, 13, 18, 25]) { const g = NAMES.map((name) => measure(name, k)); const least = NAMES[g.indexOf(g.reduce((a, b) => (b.moved < a.moved ? b : a)))]; console.log(String(k).padStart(7) + say(g[0].moved).padStart(11) + say(g[1].moved).padStart(11) + (g[0].moved / g[1].moved).toFixed(3).padStart(8) + String(g[0].rounds).padStart(11) + least.padStart(17)); } const PIECE = { list: 5, detail: 20, seller: 100, zone: 1000 }; // opens between changes, per piece const CONTENT = { "G1 chatty": [["list"], ["seller"], ["zone"], ...Array.from({ length: VISIBLE }, () => ["detail"])], "G2 batched": [["list", "detail"], ["seller"], ["zone"]], "G3 single call": [["list", "detail", "seller", "zone"]] }; const BODY = { "G1 chatty": [L, S, Z, ...A], "G2 batched": [L + ALL_DETAIL, S, Z], "G3 single call": [L + ALL_DETAIL + S + Z] }; const T = 100; const result = {}; for (const name of NAMES) { let hits = 0, moved = 0, envelope = 0; for (let t = 1; t <= T; t += 1) CONTENT[name].forEach((pieces, i) => { const changed = t === 1 || pieces.some((p) => t % PIECE[p] === 0); moved += ENV + (changed ? BODY[name][i] : 0); // a hit still pays the envelope: a conditional request envelope += ENV; if (!changed) hits += 1; }); result[name] = { hit: hits / (T * CONTENT[name].length), moved, envelope }; } console.log(`\nover ${T} opens, all details are opened, every open validates with a conditional request; ` + `the piece change interval is list ${PIECE.list} / detail ${PIECE.detail} / ` + `seller ${PIECE.seller} / zone ${PIECE.zone} opens`); console.log(`${"plan".padEnd(16)}${"hit".padStart(9)}${"bytes moved".padStart(14)}` + `${"envelope bytes".padStart(16)}${"envelope share".padStart(15)}${"vs. G3".padStart(9)}`); for (const name of NAMES) { const o = result[name]; console.log(name.padEnd(16) + o.hit.toFixed(4).padStart(9) + say(o.moved).padStart(14) + say(o.envelope).padStart(16) + (o.envelope / o.moved).toFixed(4).padStart(15) + (o.moved / result["G3 single call"].moved).toFixed(3).padStart(9)); } ``` ``` model: the panel shows 25 rows, envelope 282 bytes (measured from the header text below), concurrent call limit 6 (KK3); list 1318, seller 871, zone 331, 25 details 6,784 bytes 25 of the 25 rows have their detail opened plan calls rounds bytes moved envelope share unused payload useful share G1 chatty 28 6 17,200 0.4591 0.0000 0.5409 G2 batched 3 1 10,150 0.0833 0.0000 0.9167 G3 single call 1 1 9,586 0.0294 0.0000 0.9706 3 of the 25 rows have their detail opened plan calls rounds bytes moved envelope share unused payload useful share G1 chatty 6 2 4,989 0.3391 0.0000 0.6609 G2 batched 3 1 10,150 0.0833 0.5918 0.3248 G3 single call 1 1 9,586 0.0294 0.6266 0.3439 which plan moves less, depending on how many details are opened (cold open) opened G1 bytes G2 bytes G1/G2 G1 rounds moves least 1 3,902 10,150 0.384 2 G1 chatty 3 4,989 10,150 0.492 2 G1 chatty 6 6,624 10,150 0.653 2 G1 chatty 12 9,933 10,150 0.979 3 G3 single call 13 10,492 10,150 1.034 4 G3 single call 18 13,287 10,150 1.309 4 G3 single call 25 17,200 10,150 1.695 6 G3 single call over 100 opens, all details are opened, every open validates with a conditional request; the piece change interval is list 5 / detail 20 / seller 100 / zone 1000 opens plan hit bytes moved envelope bytes envelope share vs. G3 G1 chatty 0.9379 860,055 789,600 0.9181 3.847 G2 batched 0.9200 256,815 84,600 0.3294 1.149 G3 single call 0.7900 223,584 28,200 0.1261 1.000 ``` #### Useful Share Names the Cause The differentiating measurement is the **useful share of bytes moved**: the bytes actually used, divided by the total bytes moved including the envelope. Where the loss goes shows up in two components, and whichever is bigger names the cause. When all twenty-five rows are opened, G1's useful share is 0.5409 — nearly half of what moves is envelope (0.4591). Unused payload is zero: not a single byte moves for nothing, and the plan is still the most expensive. The diagnosis is **chattiness**. When three rows are opened, the table changes direction. G3's useful share is 0.3439, and the source of the loss is not the envelope but unused payload: 0.6266. G1 is best in the same case, at 0.6609. The diagnosis is **over-packing**. **One measurement, two components, two opposite fixes.** If the envelope share is large, calls are merged; if the unused-payload share is large, calls are split. Looking at total bytes does not say this — at k = 3, G3's 9,586 bytes are the same as its 9,586 bytes at k = 25, yet one is healthy and the other is waste. #### Where the Trade-Off Turns The third table bounds the trade-off with a number. How many of the twenty-five rows have their detail opened decides which plan moves less, and the turning point sits between 12 and 13. At twelve rows, G1 moves 9,933 bytes, G3 moves 9,586; at thirteen rows, G1 climbs to 10,492 and passes G2 by a factor of 1.034. At one row opened, the ratio is 0.384 — the chatty plan moves less than a third of what G2 moves. This is the number for the condition under which the antipattern is correct: **if the usage ratio of the moved payload is below 12/25 = 0.48, many small calls move less.** Same panel, same system, same envelope — the only thing that changes is how many rows the user opens. The pattern is not a prohibition; it is a choice bound to a ratio. The round column is the second face of the trade-off, and it does not point the same way as bytes. G1 takes 2 rounds at one row, 6 rounds at twenty-five rows; G2 and G3 take 1 round in every case. KK3's six-call limit shows up here: even if calls go out in parallel, the limit splits them into rounds. The plan that moves fewer bytes can still make the caller wait through more rounds. #### A High Hit Rate Does Not Mean Fewer Bytes The last table breaks an intuition. Over a hundred opens, the chatty plan has the highest cache hit rate: 0.9379, against G3's 0.7900. The reason is **granularity** — small calls go stale independently of each other, while a combined response goes stale on whichever of its parts changes most often. In spite of that, G1 moves 3.85 times more bytes overall: 860,055 against 223,584. The table gives the reason — 789,600 of what moves is envelope, 91.81 percent of it. A call that hits does not move its body, but it still pays its envelope; a conditional request is still a request. The rule is this: **a hit rate is not a granularity measure.** A plan with twenty-eight calls can push its hit rate to 0.94 and still move nearly four times what a single-call plan moves. The two numbers have to be read together: hit rate and envelope share. #### What Grows in Return Moving from G1 to G2 cuts bytes moved from 3.85 times down to 1.15 times G3's baseline. Three things grow in return. Cache hit rate drops: 0.9379 to 0.9200, down to 0.7900 for the single call. Because the list changes once every five opens, every piece bundled with it now goes stale once every five opens too; the detail's own twenty-open lifespan is lost. On the server side, an assembly job is born: the single call has to gather twenty-five details, and that gathering runs on every request at K01's peak load. The first lesson's question comes back here — which layer this assembly belongs in is a separate decision. **Failure isolation** is lost: in the twenty-eight-call plan, one detail call dropping leaves one row missing; in the single-call plan, the same failure leaves the whole panel blank. **Graceful degradation** from the Resilience and Reliability course is bound to granularity for exactly this reason; it is not retold here, but it is part of the price paid when moving to a batched call. #### Summary - The symptom does not show up in any call metric: every call is fast, total useful payload is small, and the open is still slow. - The differentiating measurement is the useful share of bytes moved; the larger loss component names the cause. At k = 25, G1's envelope share is 0.4591 (chattiness); at k = 3, G3's unused-payload share is 0.6266 (over-packing). - The fixes are opposites: if the envelope share is large, calls are merged; if the unused-payload share is large, they are split. Total bytes alone does not separate the two. - The trade-off's turning point is measured: below a usage ratio of 12/25 = 0.48, many small calls move less (a ratio of 0.384 at one row); above it, a single call wins (1.695 at twenty-five rows). - Round count does not point the same way as bytes: G1 takes 6 rounds at twenty-five rows, because of KK3's six-call concurrency limit. - A high hit rate does not mean fewer bytes: G1's hit rate is 0.9379, but the bytes it moves are 3.85 times G3's, because 91.81 percent of what moves is envelope, and a conditional request pays the envelope too. #### Next Step Batching the call cut what moves down to a third, but nobody looked **inside** what it carries. Twenty-five details hold 6,784 bytes, and inside those bytes are fields the panel never reads: every event's carrier code, zone id, full timestamp. The unused-payload share cannot see this, because that measurement works at record granularity; waste happens at **field** granularity too. The next lesson takes on that layer: what over-fetched fields grow end to end — entries that fit in cache, hit rate, requests reaching the store, fields serialized, and network egress — and which decision hardens in return for trimming the unnecessary bytes. ### Extraneous Fetching Source: https://academia.sh/en/courses/performance-and-monitoring/extraneous-fetching Measuring over-fetching end to end: a wide record lowering the number of entries that fit in cache, key variety producing the identical hit-rate drop, the measurement that splits the two into the working set's two factors, and the threshold where moving extra fields is justified being bounded by a call's envelope. Batching the call cut what moves down to a third, but nobody looked inside what it carries. The previous lesson's measurement worked at record granularity: it counted whether a record arrived, not whether the fields inside it were read. Waste happens at field granularity too, and that is this lesson's subject. The catalog title is **Extraneous Fetching**; the term used in the body is **over-fetching**, the same shape as in the Backend curriculum's Data Access Layer and Business Logic course. The concept was built there and its bytes were measured: column selection, row limiting, the silent cost of selecting everything. That mechanic is not repeated here. The question here is a system design question: **what does the unnecessary byte grow in the layers outside the store, and what hardens when it is cut.** #### Symptom: Hit Rate Drops, Request Count Holds The symptom shows up in the edge cache. K02's edge covered nine-tenths of reads; K01's V9 assumption also projects a 0.90 hit rate, which puts reads reaching the store at 41.67 a second. The measured hit rate fell below 0.90, and reads reaching the store grew. The request count at the edge did not change. **Cause A — the record widened.** The same number of entries, each carrying more bytes. The number of entries that fit in cache fell, and eviction rose. **Cause B — the key diversified.** The record stays the same size, but the same shipment now produces multiple keys: responses that branch by language, format, or consumer. Entry count stayed the same; the **working set** grew. Both produce the same symptom, and their fixes live in different places: one concerns the response's **width**, the other its **variety**. #### The Measurement Rig The cache is a real **least recently used** (LRU) arrangement, bounded by a byte capacity. The request trace comes from a seeded Zipf generator; a handful of tracking numbers get queried a lot, most get queried rarely. **KK4 — memory the edge cache allots this record type: 1 MiB (at model scale).** Rationale: the full twenty-thousand-shipment universe must not fit, or eviction never happens and the price of width never shows up. Its sensitivity is given at 2x and 4x in the last table. ```js // fields/end-to-end.mjs — the end-to-end cost of unnecessary bytes: entries that fit in cache, // hit rate, requests reaching the store, and network egress. The cache is a real LRU, the trace // comes from a seeded Zipf generator. const UNIVERSE = 20_000, TRACE_LEN = 200_000, CAPACITY = 1 << 20; // KK4: cache 1 MiB (model scale) const EVENT = 7; // K01 V4 const record = (i) => ({ no: `TR-2026-${String(i).padStart(9, "0")}`, seller: i % 20, day: i % 30, tariffId: i % 40, zoneId: i % 12, weight: 1 + (i % 30), volume: 10 + (i % 90), contract: `CT-${i % 20}`, rate: 12 + (i % 7) * 0.5, coefficient: 1 + (i % 5) * 0.1, discount: (i % 9) * 0.01, createdAt: 1700000000 + i * 37, updatedAt: 1700000000 + i * 41, state: i % 5, carrier: `CAR-${i % 6}`, routeCode: `R-${i % 40}-${i % 12}`, deliveryWindow: `${8 + (i % 4)}:00-${12 + (i % 4)}:00`, events: Array.from({ length: EVENT }, (_, j) => ({ code: (i * 3 + j) % 5, ts: 1700000000 + i * 41 + j * 900, zone: (i + j) % 12, route: `R-${(i + j) % 40}`, carrier: `CAR-${(i + j) % 6}`, seq: j, note: `step ${j} record received at hub ${(i + j) % 12}` })), }); const NAME = ["accepted", "transferred", "out-for-delivery", "delivered", "pending"]; const FORMAT = { full: (r) => r, medium: (r) => ({ no: r.no, state: r.state, zoneId: r.zoneId, updatedAt: r.updatedAt, carrier: r.carrier, routeCode: r.routeCode, weight: r.weight, events: r.events.slice(-3).map((o) => ({ code: o.code, ts: o.ts, route: o.route, zone: o.zone, note: o.note })) }), needed: (r) => ({ no: r.no, state: NAME[r.state], zone: r.zoneId, updatedAt: r.updatedAt, events: r.events.slice(-3).map((o) => ({ code: o.code, ts: o.ts, route: o.route })) }), }; const countFields = (x) => Array.isArray(x) ? x.reduce((a, y) => a + countFields(y), 0) : (x && typeof x === "object") ? Object.values(x).reduce((a, y) => a + countFields(y), 0) : 1; function buildTrace() { // seeded Zipf: popular tracking numbers let s = 20260801 % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const w = Array.from({ length: UNIVERSE }, (_, i) => 1 / (i + 1)); const cum = w.reduce((a, x, i) => (a.push((a[i - 1] ?? 0) + x), a), []); const total = cum[UNIVERSE - 1]; const draw = () => { const h = rand() * total; let a = 0, b = UNIVERSE - 1; while (a < b) { const m = (a + b) >> 1; if (cum[m] < h) a = m + 1; else b = m; } return a; }; return Array.from({ length: TRACE_LEN }, draw); } const TRACE = buildTrace(); function run(format, variety, capacity = CAPACITY) { // variety: distinct keys the same record produces const cache = new Map(); // LRU: Map preserves insertion order let filled = 0, hits = 0, moved = 0, serializedFields = 0; for (let t = 0; t < TRACE.length; t += 1) { const i = TRACE[t], key = `${i}|${t % variety}`; const body = FORMAT[format](record(i)); const b = Buffer.byteLength(JSON.stringify(body)); moved += b; if (cache.has(key)) { hits += 1; cache.delete(key); cache.set(key, b); continue; } serializedFields += countFields(body); // serialized on a miss while (filled + b > capacity && cache.size > 0) { const [oldest, oldestBytes] = cache.entries().next().value; cache.delete(oldest); filled -= oldestBytes; } cache.set(key, b); filled += b; } const recordBytes = Buffer.byteLength(JSON.stringify(FORMAT[format](record(1)))); return { bytes: recordBytes, fields: countFields(FORMAT[format](record(1))), variety, entries: Math.floor(capacity / recordBytes), hit: hits / TRACE_LEN, serializedFields, moved }; } const READ = 416.67, K01_HIT = 0.9; // K01: peak read requests/s, V9 const SCENARIO = [["full response", "full", 1], ["medium response", "medium", 1], ["as-needed response", "needed", 1], ["as-needed, 4-variety key", "needed", 4]]; const say = (x) => x.toLocaleString("en-US"); console.log(`model: ${say(UNIVERSE)} shipments, ${say(TRACE_LEN)} requests (seeded Zipf, seed 20260801), ` + `cache ${CAPACITY / 1024} KiB; K01: V5 tracking response 480 bytes, V7 shipment record 900 bytes`); console.log(`\n${"scenario".padEnd(28)}${"record bytes".padStart(14)}${"fields".padStart(8)}` + `${"entries that fit".padStart(19)}${"key variety".padStart(13)}${"hit".padStart(9)}` + `${"requests to store/s".padStart(21)}${"network egress Mbit/s".padStart(23)}`); const results = {}; for (const [name, format, variety] of SCENARIO) { const o = run(format, variety); results[name] = o; console.log(name.padEnd(28) + say(o.bytes).padStart(14) + String(o.fields).padStart(8) + say(o.entries).padStart(19) + String(variety).padStart(13) + o.hit.toFixed(4).padStart(9) + (READ * (1 - o.hit)).toFixed(2).padStart(21) + ((READ * o.bytes * 8) / 1e6).toFixed(3).padStart(23)); } console.log(`K01 assumption: hit ${K01_HIT} -> requests to store ${(READ * (1 - K01_HIT)).toFixed(2)}/s`); const full = results["full response"], needed = results["as-needed response"], variety4 = results["as-needed, 4-variety key"]; console.log(`\ndifferentiating measurement — working set = key variety x record bytes`); console.log(`${"scenario".padEnd(28)}${"bytes per key".padStart(15)}${"variety".padStart(9)}` + `${"working-set bytes".padStart(20)}${"vs. needed".padStart(13)}`); for (const name of ["full response", "as-needed response", "as-needed, 4-variety key"]) console.log(name.padEnd(28) + say(results[name].bytes).padStart(15) + String(results[name].variety).padStart(9) + say(results[name].bytes * results[name].variety).padStart(20) + ((results[name].bytes * results[name].variety) / needed.bytes).toFixed(2).padStart(13)); console.log(`\nend-to-end difference (full -> needed): record ${say(full.bytes)} -> ${say(needed.bytes)} bytes ` + `(${(full.bytes / needed.bytes).toFixed(2)}x), entries that fit ${say(full.entries)} -> ${say(needed.entries)}, ` + `hit ${full.hit.toFixed(4)} -> ${needed.hit.toFixed(4)}, requests to store ` + `${(READ * (1 - full.hit)).toFixed(2)} -> ${(READ * (1 - needed.hit)).toFixed(2)}/s`); console.log(`fields serialized (on misses): ${say(full.serializedFields)} -> ${say(needed.serializedFields)} ` + `(${(full.serializedFields / needed.serializedFields).toFixed(2)}x); network egress ` + `${((READ * full.bytes * 8) / 1e6).toFixed(3)} -> ${((READ * needed.bytes * 8) / 1e6).toFixed(3)} Mbit/s`); console.log(`the cost of key variety (needed -> 4 varieties): hit ${needed.hit.toFixed(4)} -> ` + `${variety4.hit.toFixed(4)}, requests to store ${(READ * (1 - needed.hit)).toFixed(2)} -> ` + `${(READ * (1 - variety4.hit)).toFixed(2)}/s, network egress unchanged`); const ENVELOPE = 282; // lesson 03: a call's envelope console.log(`\nthe threshold where moving extra fields is justified: if it eliminates a call, the excess ` + `must be smaller than ${ENVELOPE} bytes (lesson 03's envelope); the measured excess is ` + `medium ${say(results["medium response"].bytes - needed.bytes)} bytes, full ${say(full.bytes - needed.bytes)} bytes`); console.log(`\nKK4 sensitivity (cache ${CAPACITY / 1024} KiB -> 2x and 4x) hit`); console.log(`${"scenario".padEnd(28)}${"1x".padStart(9)}${"2x".padStart(9)}${"4x".padStart(9)}`); for (const [name, format, variety] of SCENARIO) console.log(name.padEnd(28) + [1, 2, 4] .map((k) => run(format, variety, CAPACITY * k).hit.toFixed(4).padStart(9)).join("")); ``` ``` model: 20,000 shipments, 200,000 requests (seeded Zipf, seed 20260801), cache 1024 KiB; K01: V5 tracking response 480 bytes, V7 shipment record 900 bytes scenario record bytes fields entries that fit key variety hit requests to store/s network egress Mbit/s full response 1,110 66 944 1 0.6042 164.92 3.700 medium response 403 22 2,601 1 0.7246 114.77 1.343 as-needed response 214 13 4,899 1 0.8007 83.03 0.713 as-needed, 4-variety key 214 13 4,899 4 0.6315 153.55 0.713 K01 assumption: hit 0.9 -> requests to store 41.67/s differentiating measurement — working set = key variety x record bytes scenario bytes per key variety working-set bytes vs. needed full response 1,110 1 1,110 5.19 as-needed response 214 1 214 1.00 as-needed, 4-variety key 214 4 856 4.00 end-to-end difference (full -> needed): record 1,110 -> 214 bytes (5.19x), entries that fit 944 -> 4,899, hit 0.6042 -> 0.8007, requests to store 164.92 -> 83.03/s fields serialized (on misses): 5,224,494 -> 518,115 (10.08x); network egress 3.700 -> 0.713 Mbit/s the cost of key variety (needed -> 4 varieties): hit 0.8007 -> 0.6315, requests to store 83.03 -> 153.55/s, network egress unchanged the threshold where moving extra fields is justified: if it eliminates a call, the excess must be smaller than 282 bytes (lesson 03's envelope); the measured excess is medium 189 bytes, full 896 bytes KK4 sensitivity (cache 1024 KiB -> 2x and 4x) hit scenario 1x 2x 4x full response 0.6042 0.6859 0.7701 medium response 0.7246 0.8073 0.8847 as-needed response 0.8007 0.8793 0.9157 as-needed, 4-variety key 0.6315 0.7083 0.7763 ``` The numbers belong to the **measurement** class and depend on the seed; the columns converted to K01 scale, requests/s and Mbit/s, belong to the **calculation** class. The model's as-needed response comes out to 214 bytes, while K01's V5 assumption was 480 bytes — the assumption was more conservative, and what travels across scales is the **ratio** between scenarios, not the absolute byte count. #### Record Widening and Key Variety Produce the Same Symptom The first and fourth rows of the first table are this whole lesson's point. In the full response, the hit rate is 0.6042 and reads reaching the store are 164.92 a second. In the as-needed response that still produces four key varieties, the hit rate is 0.6315 and reads reaching the store are 153.55. **Both scenarios produce the same symptom** — a hit rate far below 0.90, reads reaching the store at roughly four times K01's projected 41.67 — yet one comes from a wide record, the other from key variety. The differentiating measurement is in the second table, and it is a single product: **working set = key variety × bytes per key.** Taking the as-needed response as the baseline, the full response's working set is 5.19 times larger, the four-variety response's is 4.00 times larger. Which of the two factors grew tells the cause: in the full response it is bytes (1,110 against 214), in the four-variety response it is variety (4 against 1). A third column confirms the diagnosis. Network egress is 3.700 Mbit/s for the full response, 0.713 Mbit/s for the four-variety response — unchanged from the as-needed baseline. **Same hit-rate drop, very different network egress.** A dashboard that looks only at hit rate cannot separate the two causes; the working set's two factors do. #### The Unnecessary Byte's Cost in Four Separate Places The move from the full response to the as-needed response shifts four quantities at once, and not by the same ratio. The record drops from 1,110 bytes to 214: **5.19 times.** Entries that fit in cache rise from 944 to 4,899; the ratio is again 5.19, because capacity is denominated in bytes. The hit rate rises from 0.6042 to 0.8007 — this ratio is not 5.19, because hit rate is not linearly tied to entry count, it is tied to the popularity distribution. Reads reaching the store drop from 164.92 to 83.03 a second: **1.99 times.** The count of fields serialized drops from 5,224,494 to 518,115: **10.08 times.** That is nearly double the record ratio (5.19), and the reason is compound: fields per record drop from 66 to 13, and misses themselves also thin out. The serialization job shrinks by two factors at once. Network egress drops from 3.700 Mbit/s to 0.713, **exactly 5.19 times.** K01's peak read egress was 1.6 Mbit/s; the full response pushes it to 2.31 times that. The KK4 sensitivity table carries a warning: quadrupling the cache brings the full response's hit rate from 0.6042 to 0.7701, and it still does not catch up to the as-needed response's 0.8007 at 1x. **Adding memory does not close the width gap.** #### The Threshold Where Moving Extra Fields Is Justified Moving extra fields is not always waste; if it eliminates a call, it has a payoff, and the threshold for that payoff comes from the previous lesson. A call's envelope was 282 bytes. So the rule is this: **the excess moved is justified if it is smaller than the envelope of the call it eliminates.** The measurement applies the threshold to both sides. The medium response's excess is 189 bytes; below 282, so moving these fields pays off if they make a second call unnecessary. The full response's excess is 896 bytes; larger than three envelopes, so it cannot be defended unless it eliminates three separate calls. The threshold depends on scale, and the direction flips when scale changes. If the envelope shrinks under a compressed protocol, the threshold drops, and the medium response also becomes indefensible; if a record's popularity is very high, it is already sitting in cache, and the excess's hit-rate cost approaches zero. The antipattern is not a prohibition; it is a comparison between these two numbers. #### What Grows in Return The payoff for narrowing the fields sits in the fourth row, and so does the trap. Trimming a response to one consumer's exact need does not serve a second consumer; the second consumer wants its own field set, and a second variety appears. Four consumers means four varieties, and the measurement gives the price: hit rate from 0.8007 to 0.6315, reads reaching the store from 83.03 to 153.55 a second. **Cause A's fix gives rise to Cause B.** Narrowing cannot be pushed indefinitely; past a point, a single medium response comes cheaper than four narrow ones. The measurement says which is cheaper: the medium response, with one variety, gives a 0.7246 hit rate and 114.77 requests a second, while the narrow response, with four varieties, gives 0.6315 and 153.55. At this model's scale, **the medium response wins.** The second payoff is in the contract, and it is not counted: a narrow response writes today's consumer need into the response's shape. Every time a new field is requested, either the response widens or a new variety is born; both have a price, and it is in the table above. #### Summary - The symptom is in the edge cache: while request count holds steady, the hit rate falls below K01's V9 assumption of 0.90, and reads reaching the store climb to multiples of 41.67 a second. - Both record widening (hit rate 0.6042) and key variety (0.6315) produce the same symptom; the differentiating measurement is the working set's two factors — bytes at 5.19 times in the full response, variety at 4.00 times in the four-variety response. Network egress confirms it: 3.700 against 0.713 Mbit/s. - The unnecessary byte grows four places at once, and not by the same ratio: record 5.19 times, entries that fit 5.19 times, reads reaching the store 1.99 times, fields serialized 10.08 times. - Adding memory does not close the width gap: quadrupling the cache brings the full response's hit rate to 0.7701, still short of the as-needed response's 0.8007 at 1x. - The excess is justified if it is smaller than the envelope of the call it eliminates (282 bytes): the medium response's excess is 189 bytes, the full response's is 896 bytes. - The payoff of narrowing is variety: giving every consumer its own narrow response drops the hit rate from 0.8007 to 0.6315, and at this model's scale, a single medium response (0.7246) comes cheaper. #### Next Step The four measurements so far all stood in the same place: the data a request carries. The layer doing the computation, the client's work, the call's granularity, the field's width — all of it concerned what was moved. But part of a request's cost never moves at all. A request cannot start before the objects that serve it are set up: a store connection, a serializer, a signature validator, a format parser. When these objects get set up has never been asked so far; all of them were assumed to already stand ready. The next lesson measures that assumption: an expensive client being re-created on every request, the per-object setup cost multiplied by how many times it repeats, and what a pool grows in return. ### Improper Instantiation Source: https://academia.sh/en/courses/performance-and-monitoring/improper-instantiation Expensive clients re-created on every request: the two opposite causes behind throughput not rising with concurrency, the creation ratio choosing the fix's direction, setup work eroding with repeat count, and the pool's held object-rounds weighed against waiting request-rounds. The four measurements so far counted the data a request carries. Part of a request's cost, though, is never carried: a request cannot start before the objects that serve it are set up. A store connection, a serializer, a signature validator, a format parser. When these objects get set up has not been asked so far; all of them were assumed to already stand ready. The connection pool itself was built in the Database Operations and Service Design courses; the pool size's dependence on the connection limit was measured there. That mechanism is not repeated here. The question here is a diagnosis: **which symptom points to object creation, what else gives the same symptom, and which number separates the two.** #### Symptom: Throughput Does Not Rise With Concurrency The symptom appears in K01's write path. The peak request rate reaching the store is 138.89 per second. Throughput does not rise as worker processes are added; each request's work is constant — the limit ratio is 1.00, the bytes carried are unchanged, the field count is the same. The previous four metrics are all clean. **Cause A — the object is re-created on every request.** Setup work gets multiplied by the request count and eats into the processor's budget for the round. This is **improper instantiation**. **Cause B — a single shared object exists.** It is never re-created, but only one request can use it at a time. Throughput is locked to the object's hold time. Both give the "flat throughput" symptom, and their fixes are opposite: one calls for **less** creation, the other for **more**. Moving in the wrong direction makes the symptom worse. #### Measurement Setup The run below is a **model**: a round is an abstract step, not a measured duration. What gets counted is the creation count, work units, waiting request-rounds, and held object-rounds. Arrivals come from a seeded generator, Poisson-distributed. **KK5 — the work that can be done in a round is 100 units; object setup is 60, use is 10 units; the object's hold time is 3 rounds; the concurrent objects the downstream resource accepts is 24.** Rationale: setup includes a handshake, authentication, and buffer allocation; use is preparing a single call; the hold is the downstream system waiting on a response; the last number says each object holds a session on the downstream side, and that this is not unlimited. Its sensitivity is given through the setup cost in the last table. The twenty-four-object limit does not bind in this run. ```js // object/instantiation.mjs — an expensive client being re-created on every request. MODEL: // a round is an abstract step, not a duration; what is counted is creation, work units, // waiting request-rounds, and held object-rounds. Arrivals come from a seeded Poisson generator. const T = 4000, ROUNDS_PER_SEC = 100; // 4000 rounds, 100 rounds = 1 s const WORK = 100, SETUP = 60, USE = 10, HOLD = 3; // KK5: per-round work budget and object costs const LIMIT = 24; // KK5: concurrent objects the downstream resource accepts const PEAK = 138.89, AVG = 46.30; // K01: peak and average requests/s reaching the store function run({ pool, lam }) { // pool: null means a new object every request let s = 20260801 % 2147483647; const rand = () => (s = (s * 48271) % 2147483647) / 2147483647; const poisson = (l) => { const L = Math.exp(-l); let k = 0, p = 1; do { k += 1; p *= rand(); } while (p > L); return k - 1; }; const queue = []; // number of waiting requests let idle = 0, open = 0; // objects idle in the pool, objects open at the downstream resource const busy = []; // { end } — objects currently in use const c = { arrived: 0, done: 0, created: 0, waiting: 0, held: 0, setupWork: 0, useWork: 0 }; for (let t = 0; t < T; t += 1) { for (let n = busy.length - 1; n >= 0; n -= 1) if (busy[n].end <= t) { busy.splice(n, 1); if (pool === null) open -= 1; else idle += 1; // a poolless setup discards the object } const arrived = poisson(lam); c.arrived += arrived; for (let n = 0; n < arrived; n += 1) queue.push(t); let budget = WORK; while (queue.length > 0) { let obj = false; if (pool === null) { // create on every request if (open >= LIMIT) break; if (budget < SETUP + USE) break; budget -= SETUP; c.setupWork += SETUP; c.created += 1; open += 1; obj = true; } else if (idle > 0) { // take from the pool idle -= 1; obj = true; } else if (open < Math.min(pool, LIMIT)) { // grow the pool if (budget < SETUP + USE) break; budget -= SETUP; c.setupWork += SETUP; c.created += 1; open += 1; obj = true; } if (!obj) break; if (budget < USE) break; budget -= USE; c.useWork += USE; queue.shift(); c.done += 1; busy.push({ end: t + HOLD }); } c.waiting += queue.length; c.held += open; } return c; } const fmt = (x) => x.toLocaleString("en-US"); const PLAN = [["create per request", null], ["single shared object", 1], ["pool 5", 5], ["pool 12", 12], ["pool 24", 24]]; console.log(`model: ${T} rounds (${T / ROUNDS_PER_SEC} s), per-round work budget ${WORK} units, object setup ` + `${SETUP} + use ${USE} units, hold ${HOLD} rounds, downstream resource limit ${LIMIT} objects (KK5); ` + `arrivals Poisson, seed 20260801`); for (const [label, lam] of [["average", AVG / ROUNDS_PER_SEC], ["peak", PEAK / ROUNDS_PER_SEC]]) { console.log(`\n${label} load: ${(lam * ROUNDS_PER_SEC).toFixed(2)} requests/s (${lam.toFixed(4)} arrivals/round)`); console.log(`${"plan".padEnd(22)}${"done".padStart(8)}${"throughput/s".padStart(13)}` + `${"created".padStart(9)}${"creation ratio".padStart(15)}${"repeat".padStart(9)}` + `${"setup work share".padStart(17)}${"waiting request-rounds".padStart(23)}${"held object-rounds".padStart(19)}`); for (const [name, pool] of PLAN) { const r = run({ pool, lam }); const repeat = r.done / Math.max(r.created, 1); console.log(name.padEnd(22) + fmt(r.done).padStart(8) + (r.done / (T / ROUNDS_PER_SEC)).toFixed(2).padStart(13) + fmt(r.created).padStart(9) + (r.created / Math.max(r.done, 1)).toFixed(4).padStart(15) + repeat.toFixed(2).padStart(9) + (r.setupWork / Math.max(r.setupWork + r.useWork, 1)).toFixed(4).padStart(17) + fmt(r.waiting).padStart(23) + fmt(r.held).padStart(19)); } } const r1 = run({ pool: null, lam: PEAK / ROUNDS_PER_SEC }), r5 = run({ pool: 5, lam: PEAK / ROUNDS_PER_SEC }); const r24 = run({ pool: 24, lam: PEAK / ROUNDS_PER_SEC }); console.log(`\nceilings (computed, full requests per round only): create per request ` + `${Math.floor(WORK / (SETUP + USE)) * ROUNDS_PER_SEC} requests/s; warmed pool p gives ` + `${(ROUNDS_PER_SEC / HOLD).toFixed(2)} x p, work-budget ceiling ${Math.floor(WORK / USE) * ROUNDS_PER_SEC} requests/s; ` + `downstream resource limit ${((LIMIT / HOLD) * ROUNDS_PER_SEC).toFixed(2)} requests/s (for p=5, ${((5 / HOLD) * ROUNDS_PER_SEC).toFixed(2)})`); console.log(`at peak load: created ${fmt(r1.created)} -> ${fmt(r5.created)} ` + `(${(r1.created / Math.max(r5.created, 1)).toFixed(1)}x fewer), setup work ` + `${fmt(r1.setupWork)} -> ${fmt(r5.setupWork)} units, waiting request-rounds ` + `${fmt(r1.waiting)} -> ${fmt(r5.waiting)}`); console.log(`the pool's cost: held object-rounds is ${fmt(r5.held)} at pool 5, ${fmt(r24.held)} ` + `at pool 24 (${(r24.held / r5.held).toFixed(2)}x); pool 24 throughput ` + `${(r24.done / (T / ROUNDS_PER_SEC)).toFixed(2)} requests/s, pool 5 throughput ${(r5.done / (T / ROUNDS_PER_SEC)).toFixed(2)}`); console.log(`\nhow the setup work share erodes with repeat (computed): share = ${SETUP} / (${SETUP} + repeat x ${USE})`); console.log(`${"repeat".padStart(8)}${"setup work share".padStart(17)}${"work per request".padStart(17)}`); for (const repeat of [1, 2, 5, 10, 50, 200]) console.log(String(repeat).padStart(8) + (SETUP / (SETUP + repeat * USE)).toFixed(4).padStart(17) + (SETUP / repeat + USE).toFixed(2).padStart(17)); console.log(`\nhow the pool's gain changes with setup cost (computed, repeat 200)`); console.log(`${"setup".padStart(8)}${"repeat 1".padStart(10)}${"repeat 200".padStart(12)}` + `${"gain factor".padStart(12)}${"units saved".padStart(18)}`); for (const setup of [5, 10, 30, 60]) { const one = setup + USE, two = setup / 200 + USE; console.log(String(setup).padStart(8) + one.toFixed(2).padStart(10) + two.toFixed(2).padStart(12) + (one / two).toFixed(2).padStart(12) + (one - two).toFixed(2).padStart(18)); } ``` ``` model: 4000 rounds (40 s), per-round work budget 100 units, object setup 60 + use 10 units, hold 3 rounds, downstream resource limit 24 objects (KK5); arrivals Poisson, seed 20260801 average load: 46.30 requests/s (0.4630 arrivals/round) plan done throughput/s created creation ratio repeat setup work share waiting request-rounds held object-rounds create per request 1,867 46.67 1,867 1.0000 1.00 0.8571 866 5,598 single shared object 1,333 33.33 1 0.0008 1333.00 0.0045 1,141,381 3,997 pool 5 1,870 46.75 5 0.0027 374.00 0.0158 29 19,966 pool 12 1,870 46.75 7 0.0037 267.14 0.0220 2 27,439 pool 24 1,870 46.75 7 0.0037 267.14 0.0220 2 27,439 peak load: 138.89 requests/s (1.3889 arrivals/round) plan done throughput/s created creation ratio repeat setup work share waiting request-rounds held object-rounds create per request 4,000 100.00 4,000 1.0000 1.00 0.8571 3,205,927 11,997 single shared object 1,334 33.35 1 0.0007 1334.00 0.0045 8,539,260 4,000 pool 5 5,601 140.03 5 0.0009 1120.20 0.0053 6,456 19,990 pool 12 5,608 140.20 12 0.0021 467.33 0.0127 28 45,257 pool 24 5,608 140.20 12 0.0021 467.33 0.0127 28 45,257 ceilings (computed, full requests per round only): create per request 100 requests/s; warmed pool p gives 33.33 x p, work-budget ceiling 1000 requests/s; downstream resource limit 800.00 requests/s (for p=5, 166.67) at peak load: created 4,000 -> 5 (800.0x fewer), setup work 240,000 -> 300 units, waiting request-rounds 3,205,927 -> 6,456 the pool's cost: held object-rounds is 19,990 at pool 5, 45,257 at pool 24 (2.26x); pool 24 throughput 140.20 requests/s, pool 5 throughput 140.03 how the setup work share erodes with repeat (computed): share = 60 / (60 + repeat x 10) repeat setup work share work per request 1 0.8571 70.00 2 0.7500 40.00 5 0.5455 22.00 10 0.3750 16.00 50 0.1071 11.20 200 0.0291 10.30 how the pool's gain changes with setup cost (computed, repeat 200) setup repeat 1 repeat 200 gain factor units saved 5 15.00 10.03 1.50 4.97 10 20.00 10.05 1.99 9.95 30 40.00 10.15 3.94 29.85 60 70.00 10.30 6.80 59.70 ``` The run's numbers are in the **measurement** class and depend on the seed; the ceilings and the last two tables are **computed**. #### The Creation Ratio Chooses the Direction The distinguishing measurement is the **creation ratio**: the number of objects created divided by the number of completed requests. At peak load it is 1.0000 for the create-per-request layout and 0.0007 for the single shared object. Both plans give the same symptom — throughput stays below the incoming 138.89 requests/s, at 100.00 and 33.35 respectively — but the diagnosis is opposite. If the ratio is close to 1, setup work is being paid again on every request; the fix is to **reduce** creation. If the ratio is close to zero and throughput is still flat, the object is not being created enough; the fix is to **increase** creation. One number, two opposite directions. The setup work share column gives the size of the first cause: in the create-per-request layout, 85.71% of the total work is setup. Of the 70 units of work per request, 60 are preparation that has nothing to do with the request's own work. In pool 5, the same share drops to 0.53%. #### The Pattern Does Not Show at Average Load The average-load table carries a warning. At 46.30 requests per second, the create-per-request layout gives 46.67 requests/s — nearly the same as pool 5's 46.75. Waiting request-rounds are 866 against 29 — visible, but small. **At average load, the anti-pattern looks absent.** At peak, the table breaks. The create-per-request layout's ceiling is, by computation, 100 requests/s: a round has 100 units of work, and a request with its setup takes 70 units, so one request fits per round. The incoming 138.89 requests/s is above that ceiling, and the difference is written to the queue: waiting request-rounds climbs from 866 to 3,205,927 — **3702x**. In pool 5, the same number climbs from 29 to 6,456, 223x. The single shared object lands at the same place in both loads: 33.33 and 33.35 requests/s. Its ceiling does not come from the load, it comes from the hold — with 100 rounds per second and a 3-round hold, one object can carry 33.33 requests per second. This plan falls short even at average load; waiting request-rounds is 1,141,381. #### The Condition Where Creating Per Request Is Correct Creating an object on every request is not a mistake by itself; the mistake is creating an **expensive** object on every request, and the measure of expense is the ratio of setup work to usage work. The setup-share table gives how this ratio erodes with the repeat count. At repeat 1, the share is 0.8571 and work per request is 70 units; at repeat 10, 0.3750 and 16 units; at repeat 200, 0.0291 and 10.30 units. The gain is bounded not by the repeat count itself but by the size of the setup: the last table shows that at the same repeat, a setup of 60 units gains 6.80x, while a setup of 5 units gains only 1.50x. The condition can be written with one number: **if setup work is less than half of usage work, the most the pool can gain is 1.50x** — at this model's scale, setup is 5 units and usage is 10 units. Since the pool also brings held object-rounds and maintenance load, not building a pool is defensible at this scale. When the same object's setup rises to 60 units, the same decision means leaving 6.80x on the table. #### What Grows in Exchange for the Pool The pool grows two things at once, and both are countable. **Held object-rounds.** Pool 5 holds 19,990 object-rounds at peak load; pool 24 holds 45,257 — 2.26x. These objects hold a session at the downstream resource and eat into KK5's twenty-four-object limit. This is the cost of growing the pool, and the cost does not show up in throughput: pool 24 completes 140.20 requests per second, pool 5 completes 140.03. The difference is 0.12%. **But it shows up in waiting.** Pool 5's waiting request-rounds is 6,456, pool 12's is 28. Growing the pool from 5 to 12 grows the resource 2.26x while shrinking waiting 231x. The two numbers move in opposite directions, and the choice depends on which one gives the binding ceiling. Pool 24 gives the exact same result as pool 12 — demand never exceeds twelve, so anything past twelve is only held resource. A third exchange is not counted but is visible: if an object taken from the pool is never returned, the pool shrinks permanently. In the create-per-request layout, such a leak slows one request and ends there; in the pooled layout, it permanently narrows the system. The pool turns a bug from transient into permanent. #### Summary - The symptom is throughput not rising with concurrency, and two opposite causes give the same symptom: creating per request (100.00 requests/s at peak) and a single shared object (33.35 requests/s). - The distinguishing measurement is the creation ratio: at 1.0000, creation is reduced; near zero, like 0.0007, creation is increased. One number, two opposite directions. - The pattern does not show at average load: at 46.30 requests/s, the create-per-request layout gives 46.67, the pool gives 46.75. At peak, waiting request-rounds climbs from 866 to 3,205,927 (3702x). - Setup work's share erodes with repeat: 0.8571 and 70 units per request at repeat 1, 0.0291 and 10.30 units at repeat 200. Pool 5 brings creation down from 4,000 to 5 (800x). - The condition is the ratio of setup to usage: at setup 5 and usage 10 units, the most the pool can gain is 1.50x; at setup 60 units, 6.80x. - The pool's cost is held object-rounds: pool 24 holds 2.26x the resource of pool 5 while raising throughput by 0.12%; in exchange, pool 12 brings waiting down from 6,456 to 28. #### Next Step All five patterns were measured in the same place: **inside a single process and in front of a single store.** The computation's layer, the client's work, the call's granularity, the field's breadth, and the object's lifetime — all of them counted a request's own waste, and every fix stayed inside that process or that response. The assumption shared by all five was never questioned: every one of these requests goes to **the same store**. The tracking query, the state event, and the end-of-day billing run through the same engine, the same resource pool, the same maintenance window. Why the store is single was never a design decision — it was an assumption. The next lesson tests that assumption starting from a symptom: which measurement reveals the single store's limit, and what grows in exchange for splitting it. ### Monolithic Persistence Source: https://academia.sh/en/courses/performance-and-monitoring/monolithic-persistence Recognizing all data classes piled into a single store from its symptom: separating two causes of the same slowdown, the read path showing nothing, the period scan crossing 801 times its own class's record count, a write consulting two indexes that do not serve its own query, and the write path stretching to two stores in exchange for splitting. The five patterns so far measured waste inside a single store and a single process: computation done in the wrong layer, work piled onto the main thread, too many small calls, reading more than what is needed, and an expensive client being re-created. All of them took the store's singleness as a given. This lesson questions that assumption. The symptom is this: the end-of-day billing job no longer fills its four-hour window, but in that same window, the work a write does per request as it reaches the store has measurably grown. In K01's calculation, peak write is 97.22 requests/s, and that number has not changed; what changed is how much work one write generates in the store. **Monolithic persistence** is all data classes piled into a single store. The "monolithic" in this name is the same word as the monolith in the Architectural Styles course, but a separate thing: there, the single thing is the deployment unit; here, the single thing is the data store. #### Same Symptom, Two Causes Growth in per-write work can have two separate causes, and the two cannot be told apart without measuring. **First cause: the record grew.** New fields have been added to the state event; every write carries more bytes and updates more structures. This is the write-side counterpart of the waste the fourth lesson measured. **Second cause: the store is shared.** The record has not changed at all. An index has been added to the same table for another data class's query, and from that day on, every state event write has to also maintain a structure its own query never uses. Guessing does not help here; both weigh down the write path and both give the same symptom. The distinguishing measurement is this: break down the records an access path touches and the indexes it consults, by class. In the first cause, a write's work grows, but all of that work serves its own class; in the second cause, the work splits off a share that does not serve its own class. That share is measurable. #### Two Layouts The setup below is not a model but a real measurement: the same logical data is entered into both layouts, and the counts are read from the store itself. The Scaling the Data Layer course's federation decision is not retold here; the question here is which symptom that decision grows out of. **KK6 — scale.** One-thousandth of K01's daily volume, over a thirty-day period. Rationale: the ratios between classes come from K01's assumption table (V3: 400,000 shipments per day, V4: 7 events per shipment, V13: 100 items per invoice line); the divisor only shortens the run time, it does not change the ratios. Its sensitivity is given below through the class mix. This assumption is not added to K01's table. ```js // persistence/store.mjs — the same logical data set up in two layouts: all classes in one // table, and each class in its own store. Built for real with node:sqlite; counts are read // from SQL. The scale is one-thousandth of K01's daily volume (KK6), the period is 30 days. import { DatabaseSync } from "node:sqlite"; export const KK6 = 1000; // scale divisor export const DAYS = 30; // K01 V11: billing period export const K01 = { shipment: 400_000, eventFactor: 7, invoiceLine: 100, peakWrite: 97.22 }; export const COUNT = { shipment: (K01.shipment / KK6) * DAYS, event: ((K01.shipment * K01.eventFactor) / KK6) * DAYS, period: (K01.shipment / K01.invoiceLine / KK6) * DAYS, }; // The data is generated once; the same rows go into both layouts. export function rows() { const c = []; const daily = { shipment: COUNT.shipment / DAYS, event: COUNT.event / DAYS, period: COUNT.period / DAYS }; for (let day = 1; day <= DAYS; day += 1) { for (let i = 1; i <= daily.shipment; i += 1) { const trackingNo = `TK${day}-${i}`; c.push({ type: "shipment", key: trackingNo, seller: null, day }); for (let k = 1; k <= K01.eventFactor; k += 1) c.push({ type: "event", key: trackingNo, seller: null, day }); } for (let s = 1; s <= daily.period; s += 1) c.push({ type: "period", key: null, seller: `S${s}`, day }); } return c; } // Monolithic layout: one table, with indexes serving every class's queries. export function monolithic(data) { const d = new DatabaseSync(":memory:"); d.exec(`CREATE TABLE record(id INTEGER PRIMARY KEY, type TEXT, key TEXT, seller TEXT, day INTEGER, body BLOB); CREATE INDEX ix_key ON record(key); CREATE INDEX ix_day ON record(day); CREATE INDEX ix_seller ON record(seller)`); const y = d.prepare("INSERT INTO record(type,key,seller,day,body) VALUES(?,?,?,?,?)"); d.exec("BEGIN"); for (const r of data) y.run(r.type, r.key, r.seller, r.day, null); d.exec("COMMIT"); return d; } // Separated layout: each class in its own store, with only its own query's index. export function separated(data) { const shipment = new DatabaseSync(":memory:"); const event = new DatabaseSync(":memory:"); const period = new DatabaseSync(":memory:"); shipment.exec("CREATE TABLE shipment(key TEXT PRIMARY KEY, day INTEGER, body BLOB)"); event.exec(`CREATE TABLE event(id INTEGER PRIMARY KEY, key TEXT, day INTEGER, body BLOB); CREATE INDEX ix_event_key ON event(key)`); period.exec("CREATE TABLE period(seller TEXT, day INTEGER, body BLOB, PRIMARY KEY(seller,day)) WITHOUT ROWID"); const ys = shipment.prepare("INSERT INTO shipment VALUES(?,?,?)"); const ye = event.prepare("INSERT INTO event(key,day,body) VALUES(?,?,?)"); const yp = period.prepare("INSERT INTO period VALUES(?,?,?)"); for (const d of [shipment, event, period]) d.exec("BEGIN"); for (const r of data) { if (r.type === "shipment") ys.run(r.key, r.day, null); else if (r.type === "event") ye.run(r.key, r.day, null); else yp.run(r.seller, r.day, null); } for (const d of [shipment, event, period]) d.exec("COMMIT"); return { shipment, event, period }; } // The number of indexes consulted while writing to a table is read from SQL (PRAGMA index_list). export function indexCount(d, table) { return d.prepare(`SELECT count(*) n FROM pragma_index_list(?)`).get(table).n; } ``` ```js // persistence/measurement.mjs — the record and index count three access paths touch in two layouts import { KK6, DAYS, K01, COUNT, rows, monolithic, separated, indexCount } from "./store.mjs"; const data = rows(); const mono = monolithic(data); const sep = separated(data); const q = (d, s, ...p) => d.prepare(s).get(...p).n; console.log(`scale 1/${KK6}, period ${DAYS} days -> shipment ${COUNT.shipment}, event ${COUNT.event}, period ${COUNT.period}`); console.log(`monolithic store record count = ${q(mono, "SELECT count(*) n FROM record")}`); console.log(`separated stores' record count = ${q(sep.shipment, "SELECT count(*) n FROM shipment")}` + ` / ${q(sep.event, "SELECT count(*) n FROM event")} / ${q(sep.period, "SELECT count(*) n FROM period")}\n`); // S1 tracking read: one tracking number's shipment record and its events. const TRACKING_NO = "TK15-100"; const s1Mono = q(mono, "SELECT count(*) n FROM record WHERE key=?", TRACKING_NO); const s1Sep = q(sep.shipment, "SELECT count(*) n FROM shipment WHERE key=?", TRACKING_NO) + q(sep.event, "SELECT count(*) n FROM event WHERE key=?", TRACKING_NO); // S2 period scan: 30 days of seller-day records; the shared day index walks every class. const s2Mono = q(mono, "SELECT count(*) n FROM record WHERE day BETWEEN 1 AND ?", DAYS); const s2Sep = q(sep.period, "SELECT count(*) n FROM period WHERE day BETWEEN 1 AND ?", DAYS); console.log("path monolithic separated ratio needed"); console.log("--------------------- ----------- --------- ----- --------"); const row = (name, a, b, needed) => console.log(`${name.padEnd(21)} ${String(a).padStart(11)} ${String(b).padStart(9)} ` + `${(a / b).toFixed(2).padStart(5)} ${String(needed).padStart(8)}`); row("S1 tracking read", s1Mono, s1Sep, 4); row("S2 period scan", s2Mono, s2Sep, COUNT.period); // S3 state event write: indexes consulted, and the index serving its own query. const dMono = indexCount(mono, "record"); const dSep = indexCount(sep.event, "event"); console.log(`\nS3 state event write: monolithic consults ${dMono} indexes, separated store consults ${dSep}`); console.log(`the share serving their own class's query: 1 / ${dMono} = ${(1 / dMono).toFixed(2)}` + ` and 1 / ${dSep} = ${(1 / dSep).toFixed(2)}`); console.log(`K01 peak write ${K01.peakWrite} requests/s -> index maintenance ${(K01.peakWrite * dMono).toFixed(2)}/s and` + ` ${(K01.peakWrite * dSep).toFixed(2)}/s, difference ${(K01.peakWrite * (dMono - dSep)).toFixed(2)}/s`); // What splitting costs: the write path now spans two stores (the event log + the shipment's latest state). const splitWrites = 2, monoWrites = 1; console.log(`\nwhat splitting costs: one state event is written to ${monoWrites} store instead of ${splitWrites}` + ` -> ${(K01.peakWrite * splitWrites).toFixed(2)} store writes/s (${(K01.peakWrite * monoWrites).toFixed(2)} in the monolithic store)`); console.log(`stores operated 1 -> 3; the write path is no longer atomic at ${K01.peakWrite.toFixed(2)} requests/s`); // Retention rule: one rule applies to every class (K01 V12 = 730 days, the period only needs 30 days). const [RETENTION, PERIOD_NEEDED] = [730, DAYS]; const periodDaily = COUNT.period / DAYS; console.log(`\nretention: one rule, 730 days -> period records are held ${RETENTION} days, needed ${PERIOD_NEEDED} days`); console.log(`at scale, period records held ${periodDaily * RETENTION} instead of ${periodDaily * PERIOD_NEEDED}` + ` -> ${(RETENTION / PERIOD_NEEDED).toFixed(2)}x too many`); // KK6's sensitivity: if the class mix changes, what does S2's crossing factor become. console.log(`\nclass mix (K01 V13 = items per invoice line) -> S2 crossing factor`); for (const item of [100, 20, 5, 1]) { const d = (K01.shipment / item / KK6) * DAYS; const total = COUNT.shipment + COUNT.event + d; console.log(` items ${String(item).padStart(3)} -> period records ${String(d).padStart(6)},` + ` period share ${(d / total).toFixed(4)}, factor ${(total / d).toFixed(2)}`); } ``` ``` scale 1/1000, period 30 days -> shipment 12000, event 84000, period 120 monolithic store record count = 96120 separated stores' record count = 12000 / 84000 / 120 path monolithic separated ratio needed --------------------- ----------- --------- ----- -------- S1 tracking read 8 8 1.00 4 S2 period scan 96120 120 801.00 120 S3 state event write: monolithic consults 3 indexes, separated store consults 1 the share serving their own class's query: 1 / 3 = 0.33 and 1 / 1 = 1.00 K01 peak write 97.22 requests/s -> index maintenance 291.66/s and 97.22/s, difference 194.44/s what splitting costs: one state event is written to 1 store instead of 2 -> 194.44 store writes/s (97.22 in the monolithic store) stores operated 1 -> 3; the write path is no longer atomic at 97.22 requests/s retention: one rule, 730 days -> period records are held 730 days, needed 30 days at scale, period records held 2920 instead of 120 -> 24.33x too many class mix (K01 V13 = items per invoice line) -> S2 crossing factor items 100 -> period records 120, period share 0.0012, factor 801.00 items 20 -> period records 600, period share 0.0062, factor 161.00 items 5 -> period records 2400, period share 0.0244, factor 41.00 items 1 -> period records 12000, period share 0.1111, factor 9.00 ``` Record and index counts are in the **measurement** class: they were read from the store. Rows converted to K01 scale are **computed**; scale and class mix are in the **assumption** class. #### The Read Path Shows Nothing The table's first row is the most important part of the diagnosis. S1's tracking read touches 8 records in both layouts; the ratio is 1.00. Monolithic persistence is invisible on this path. Someone who tries to build a diagnosis from the tracking query's measurement alone concludes that the store being single has no cost, and is wrong. Both layouts read more than needed (8 records, 4 needed) — but that is the fourth lesson's subject, not this one's. The second row gives the separation. The period scan touches 120 records in the separated store — exactly as many as needed. In the monolithic store, the shared day index sorts every class's rows into the same order, so the scan crosses 96,120 records: 801 times its own class. This number does not come from the store being slow; it comes from the scan having to pass over records that do not belong to its own class. The third measurement meets the symptom directly. In the monolithic store, a state event write consults three indexes; in the separated store, one. Of these three indexes, only one (`ix_key`) serves the event's own query — the other two are there for the period class: the service ratio is 0.33. At K01's peak write of 97.22 requests/s, this means 291.66 index maintenance operations per second; in the separated layout, 97.22. The difference — 194.44 — is work no query uses. The number points to the second cause, not the first: the record has not grown at all; only the share that does not serve its own class has grown. This is the measurement that makes the separation. #### The Condition Where a Single Store Is Correct A single store is not wrong at every scale. The last table gives the boundary with a number: the crossing factor depends on the class mix. When period records are 0.0012 of the total, the factor is 801; at 0.0244, it is 41; at 0.1111, it drops to 9. So a single store is a defensible choice as long as no class's share disappears next to the others', and as the factor approaches one, splitting has nothing left to gain. The second condition is on the index side. If all three indexes served the same class, the service ratio would be 1.00 and there would be no cost paid on the write side. The cost comes not from the index count, but from the count of indexes placed for another class. The third condition is retention. K01's V12 assumption keeps every record for 730 days, while the period record's job ends once the thirty-day period closes. Because the single store carries one retention rule, period records are held 24.33 times longer than needed. If the classes' retention periods were equal, this condition would drop out too. When all three hold at once — shares close to each other, indexes shared, retention single — a single store is not just correct, it is also cheap. #### What Grows in Exchange for Splitting Splitting is not free, and its cost is on the write path. In a single store, a state event is one write: it is appended to the event log and the shipment's latest state is updated in the same transaction. Once the classes are split, the same event is written to two stores; at K01's peak write, this means 194.44 store writes per second instead of 97.22. What is heavier is not the count — it is that the two writes are no longer a single transaction. The consistency in between is handed off to the arrangement built in the Resilience and Reliability course's Compensating Transactions and Idempotent Operations lessons — a write path of 97.22 requests per second loses its atomicity. The second cost is operational surface: instead of one store, three stores get backed up, versioned, monitored, and put on call. This is the direct input to the load the next topic will count. #### Summary - The symptom is per-write work growing; it has two causes — the record has grown, or the store is shared with another class. The distinguishing measurement is the share of work that serves its own class. - The tracking read touches 8 records in both layouts (ratio 1.00): monolithic persistence is invisible on the read path, and a diagnosis cannot be made from that path alone. - The period scan passes over 120 records in the separated store and 96,120 in the monolithic store — 801 times its own class; the factor's source is the class mix. - A state event write consults 3 indexes in the monolithic store and only 1 of them serves its own query (0.33); at K01's 97.22 requests/s write, 194.44 maintenance operations per second serve no query at all. - A single store is correct when three conditions hold: class shares are close to each other (a factor of 9, not 801), indexes are shared (service ratio 1.00), and retention is single (period records are not held 24.33 times too long). - What splitting costs is a longer write path: 194.44 store writes per second instead of 97.22, and a write path that loses its atomicity; the number of stores operated goes from 1 to 3. #### Next Step This lesson questioned the store's structure and counted the share of a write that does not serve its own class. The next question is on the read side, and at first looks already answered: Scaling the Data Layer's caching topic measured where caching is not needed — it showed that putting a cache in front of a scan with a repeat ratio of 1.00 costs 2.00 extra touches per request. The next lesson is that measurement's mirror image: where caching is needed but was not put in place. But what gets measured is not the same thing. There, a repeated read was counted; here, a repeated computation is counted, and the two can diverge within the same stream: a stream with a low read repeat can have a very high computation repeat. The next lesson measures this distinction, counts the computation's repeat ratio through the computation key, and shows what choosing an incomplete key costs. ### No Caching Source: https://academia.sh/en/courses/performance-and-monitoring/no-caching Measuring an expensive, repeated computation left without a cache: the period scan's read repeat of 1.00 turning into a computation repeat of 125.00, the key-building cost setting memoization's ceiling (2.53x), the break-even repeat computing to 1.63, memoization dropping to a 0.71x ratio and holding 119,365 entries under continuous weight, and the key not including the version making half the results wrong. The previous lesson questioned the store's structure and counted the share of a write that does not serve its own class. This lesson's question is on the read side, and at first looks already answered. The Scaling the Data Layer course's caching topic measured where caching is not needed: for a period scan with a repeat ratio of 1.00, a cache brings 2.00 extra touches per request and gains nothing. This lesson is that measurement's mirror image — where caching is needed but was not put in place. But the quantity measured is not the same. There, a repeated read was counted; here, a repeated computation is counted. The symptom is this: the application layer's processor utilization is approaching saturation in the end-of-day window, even though the request rate reaching the store has not changed. In K01's calculation, `requests reaching the store` is 138.89/s and cache hit rate is 0.90 under the V9 assumption; both numbers still hold. What is slowing down is not the store — it is the application itself. #### Same Symptom, Two Causes **First cause: work per request genuinely grew.** New fields have entered the response; serialization has gotten longer. This is the processor-side counterpart of the over-fetching the fourth lesson measured. **Second cause: the same computation repeats with the same input.** Work per request has not changed at all; what changed is how many times the same work is repeated. This is called **no caching**, and the form measured throughout this lesson fixes it with **memoization**: the computation's result is stored against a **computation key** derived from its inputs. The measurement that tells the two apart is de-duplicating calls by the computation key, not the read key. If the computation repeat is close to one, the cause is the first, and memoization gains nothing; if the repeat is well above one, the cause is the second. These two repeat ratios can diverge within the same stream, and that is this lesson's central finding. #### The Computation and Three Streams The pricing computation is a real function: the weight tier is scanned, the ordered tariff rule list is walked, and the contract multiplier is found. The measured quantity is not duration, it is the number of steps executed during the run; duration would depend on this machine, the step count does not. **KK8 — the computation's shape.** Eight zones, five weight tiers, three contract classes, and seventeen ordered tariff rules. Rationale: the `zone`, `tariff`, and `contract` fields in M18's shipment library give this shape; the rule list is three layers — same zone, origin zone, and a catch-all. Its sensitivity is given through the third stream: if the weight enters the key unbucketed, the repeat collapses. This assumption is not added to K01's table. ```js // computation/tariff.mjs — the pricing computation and three streams. Counters increment during // the real run; what gets measured is not duration, it is the number of steps executed. export const ZONE = 8; export const TIER = [1, 5, 20, 50, Infinity]; // weight-tier upper bounds export const CONTRACT = ["standard", "contracted", "bulk"]; export const RULE = []; for (let k = 1; k <= ZONE; k += 1) RULE.push({ o: `Z${k}`, d: `Z${k}`, factor: 1.0 }); for (let k = 1; k <= ZONE; k += 1) RULE.push({ o: `Z${k}`, d: "*", factor: 1.0 + k / 10 }); RULE.push({ o: "*", d: "*", factor: 2.5 }); // catch-all rule export const counter = { calc: 0, key: 0, lookup: 0 }; const reset = () => { counter.calc = 0; counter.key = 0; counter.lookup = 0; }; // The computation itself: the tier is scanned, the rule list is walked in order, the contract multiplier is found. export function price(g) { let d = 0; while (TIER[d] < g.weight) { counter.calc += 1; d += 1; } let factor = 0; for (const rule of RULE) { counter.calc += 1; if ((rule.o === "*" || rule.o === g.origin) && (rule.d === "*" || rule.d === g.dest)) { factor = rule.factor; break; } } let s = 0; while (CONTRACT[s] !== g.contract) { counter.calc += 1; s += 1; } counter.calc += 1; return factor * (d + 1) * (1 + s / 4); } // The computation key is built only for memoization; four fields are read, the tier is scanned again. export function key(g) { let d = 0; while (TIER[d] < g.weight) { counter.key += 1; d += 1; } counter.key += 3; return g.continuous ? `${g.origin}>${g.dest}|${g.weight}|${g.contract}` : `${g.origin}>${g.dest}|${d}|${g.contract}`; } // A seeded generator using 32-bit integer arithmetic; Math.imul avoids precision loss. export function seededRand(seed) { let s = seed >>> 0; return () => ((s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296); } // Period scan: each seller-day record is read once (read repeat 1.00), but the computation's // inputs come from a small set. When `continuous` is on, the weight enters the key unbucketed. export function periodStream(count, { continuous = false, seed = 20260730 } = {}) { const rand = seededRand(seed); return Array.from({ length: count }, (_, i) => ({ read: `period:${i + 1}`, continuous, origin: `Z${Math.floor(rand() * ZONE) + 1}`, dest: `Z${Math.floor(rand() * ZONE) + 1}`, weight: continuous ? Math.round(rand() * 60000) / 1000 : [0.5, 3, 12, 35, 80][Math.floor(rand() * 5)], contract: CONTRACT[Math.floor(rand() * CONTRACT.length)], })); } // Tracking query: the same shipment is queried batch by batch; the computation's inputs come from the shipment itself. export function trackingStream(requests, workingSet = 1000, batch = 10, seed = 4242) { const rand = seededRand(seed); const shipment = (no) => { const r = seededRand(no * 7919 + 13); return { read: `tracking:${no}`, continuous: false, origin: `Z${Math.floor(r() * ZONE) + 1}`, dest: `Z${Math.floor(r() * ZONE) + 1}`, weight: [0.5, 3, 12, 35, 80][Math.floor(r() * 5)], contract: CONTRACT[Math.floor(r() * CONTRACT.length)] }; }; let next = 1; const active = Array.from({ length: workingSet }, () => ({ no: next++, remaining: batch })); return Array.from({ length: requests }, () => { const j = Math.floor(rand() * active.length); const g = shipment(active[j].no); if ((active[j].remaining -= 1) === 0) active[j] = { no: next++, remaining: batch }; return g; }); } export function uncached(stream) { reset(); for (const g of stream) price(g); return { ...counter, entries: 0 }; } export function memoized(stream) { reset(); const m = new Map(); for (const g of stream) { const a = key(g); counter.lookup += 1; if (m.has(a)) continue; m.set(a, price(g)); } return { ...counter, entries: m.size }; } ``` ```js // computation/measurement.mjs — separating read repeat from computation repeat, and what memoization costs in return import { price, key, periodStream, trackingStream, uncached, memoized } from "./tariff.mjs"; const RECORDS = 120_000; // K01: period seller-day const REQUESTS = 200_000; // K01: 480 s x 416.67 requests/s peak read // Column width comes from one place; the header and rows always stay aligned. const W = [25, 14, 14, 15, 21, 18]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(W[i]) : String(x).padEnd(W[i]))).join(" ")); const line = () => console.log(W.map((n) => "-".repeat(n)).join(" ")); const STREAM = [ ["period scan", periodStream(RECORDS)], ["tracking query", trackingStream(REQUESTS)], ["period, continuous weight", periodStream(RECORDS, { continuous: true })], ]; write(["stream", "calls", "distinct reads", "read repeat", "distinct computations", "computation repeat"]); line(); const measured = []; for (const [name, stream] of STREAM) { const o = uncached(stream); const m = memoized(stream); const distinctReads = new Set(stream.map((g) => g.read)).size; measured.push([name, stream.length, o, m]); write([name, stream.length, distinctReads, (stream.length / distinctReads).toFixed(2), m.entries, (stream.length / m.entries).toFixed(2)]); } console.log(); write(["stream", "uncached steps", "memoized steps", "ratio", "steps/call", "cache entries"]); line(); for (const [name, n, o, m] of measured) { const mt = m.calc + m.key + m.lookup; write([name, o.calc, mt, (o.calc / mt).toFixed(2), (o.calc / n).toFixed(2), m.entries]); } // The break-even point comes out of the measured averages: C steps computation, K steps key, L steps lookup. console.log(); write(["stream", "C", "K", "L", "break-even repeat", "measured repeat"]); line(); for (const [name, n, o, m] of measured) { const [C, K, L] = [o.calc / n, m.key / n, m.lookup / n]; write([name, C.toFixed(2), K.toFixed(2), L.toFixed(2), (C / (C - K - L)).toFixed(2), (n / m.entries).toFixed(2)]); } // KK9: a tariff revision takes effect mid-period. If the key does not include the version, // the value returned from the cache is wrong for records after the revision; the wrongs are counted. const REVISION = 1.05; const versioned = periodStream(RECORDS).map((g, i) => ({ ...g, version: i < RECORDS / 2 ? 1 : 2 })); const correctValue = (g) => price(g) * (g.version === 2 ? REVISION : 1); console.log(); for (const missing of [true, false]) { const m = new Map(); let wrong = 0; for (const g of versioned) { const a = missing ? key(g) : `${key(g)}|v${g.version}`; if (!m.has(a)) m.set(a, correctValue(g)); if (Math.abs(m.get(a) - correctValue(g)) > 1e-9) wrong += 1; } console.log(`key includes version ${missing ? "no " : "yes"}: cache entries ${String(m.size).padStart(6)},` + ` wrong results ${String(wrong).padStart(6)}, wrong ratio ${(wrong / RECORDS).toFixed(4)}`); } ``` ``` stream calls distinct reads read repeat distinct computations computation repeat ------------------------- -------------- -------------- --------------- --------------------- ------------------ period scan 120000 120000 1.00 960 125.00 tracking query 200000 20444 9.78 960 208.33 period, continuous weight 120000 120000 1.00 119365 1.01 stream uncached steps memoized steps ratio steps/call cache entries ------------------------- -------------- -------------- --------------- --------------------- ------------------ period scan 1860493 734534 2.53 15.50 960 tracking query 3102366 1214858 2.55 15.51 960 period, continuous weight 1948593 2745984 0.71 16.24 119365 stream C K L break-even repeat measured repeat ------------------------- -------------- -------------- --------------- --------------------- ------------------ period scan 15.50 5.00 1.00 1.63 125.00 tracking query 15.51 5.00 1.00 1.63 208.33 period, continuous weight 16.24 5.73 1.00 1.71 1.01 key includes version no : cache entries 960, wrong results 60000, wrong ratio 0.5000 key includes version yes: cache entries 1920, wrong results 0, wrong ratio 0.0000 ``` Step counts and distinct-key counts are in the **measurement** class: they were counted during the run. Call counts coming from K01 are **computed**; the computation's shape is in the **assumption** class. #### Read Repeat Does Not Say Computation Repeat The first table's first row is this lesson's rationale. In the period scan, read repeat is 1.00: every seller-day record is read once and never read again. The Scaling the Data Layer course had made the correct call for this row — a read cache does not belong on this stream. The same stream's computation repeat, though, is 125.00: 120,000 records fall onto only 960 distinct computation keys, because the price depends not on the shipment itself but on the zone pair, weight tier, and contract class. A stream can both not deserve a read cache and carry a computation that repeats on every record; the two measures diverge within the same stream. In the tracking query, both are large: read repeat is 9.78, computation repeat is 208.33. In this stream, a read cache already prevents part of the computation; no caching stands here as a second layer. The third row shows the boundary. When the weight enters the key unbucketed, 120,000 records produce 119,365 distinct keys, and the computation repeat drops to 1.01. The computation itself has not changed; what changed is the granularity of what the key distinguishes. #### Memoization's Ceiling The second table gives the gain in steps. In the period scan, the uncached run executes 1,860,493 steps, the memoized run 734,534: a ratio of 2.53. The gain is not proportional to the 125x repeat, and the reason is in the third table: computation averages 15.50 steps per call, but building the computation key costs 5.00 steps on every call, and the lookup itself costs 1.00 step. Even if memoization eliminated repeats entirely, 6 steps per call would keep being paid. The gain's ceiling is therefore 15.50 / 6.00, that is, 2.58; the measured 2.53 sits just under that ceiling. This flips memoization's measure around. What determines the gain is not the repeat ratio, but the computation's ratio to its key. The repeat ratio only says how close the run gets to the ceiling. The third table gives the break-even point. The threshold derived from the three measured costs takes the form $C / (C - K - L)$, and for the period scan it is 1.63: if the computation repeat is above this value, memoization pays off. Since the measured repeat is 125.00, it pays off many times over. In the continuous-weight stream, the threshold is 1.71 and the measured repeat is 1.01; memoization is at a **loss** there — the ratio is 0.71, meaning the memoized run executes 1.41 times more steps than the uncached one. #### What Grows in Return Memoization grows two things, and both are countable. **Memory.** 960 entries are held in the period scan; 119,365 in the continuous-weight stream. The second number shows how the cost explodes where there is no gain: nearly one entry per record. This is the counterpart, here, of the 2.00 extra touches the Scaling the Data Layer course measured. **Wrongness surface.** The last two rows measure this. The tariff is revised mid-period (**KK9**; rationale: billing is settled on a monthly period, and a revision can take effect within the period). When the computation key does not include the version, memoization holds 960 entries and 0.5000 of the results come out wrong — every record after the revision gets the value computed before the revision. When the version is added to the key, entries rise to 1920 and the wrong ratio becomes 0.0000. The cost is memory doubling; what is gained is correctness. An uncached computation cannot be wrong. Memoization ties correctness to the key's completeness, and that tie breaks silently if it is never measured. #### The Condition Where No Caching Is Correct No caching is correct design, not an anti-pattern, under two conditions, and both are given with a number. **When the computation repeat is below the break-even threshold.** In the continuous-weight stream, the repeat is 1.01 and the threshold is 1.71; memoization increases the step count by 1.41x and holds 119,365 entries. In this stream, doing the computation every time is correct. **When the computation is cheaper than the key.** The threshold $C / (C - K - L)$ is undefined when $C \le K + L$: if building the key costs more than doing the computation, no repeat ratio saves memoization. In this measurement, $C$ is 15.50 and $K + L$ is 6.00, so the condition holds, but it would not hold for a cheap computation. #### Summary - The symptom is the application layer approaching saturation; it has two causes — work per request has grown, or the same computation is being repeated. The distinguishing measurement is de-duplicating calls by the computation key. - In the period scan, read repeat is 1.00 while computation repeat is 125.00: a stream can go without deserving a read cache while still carrying a computation that repeats on every record. This is the mirror image of the pattern the Scaling the Data Layer course measured. - Memoization's gain is 2.53x and its ceiling is 15.50 / 6.00 = 2.58; what determines the ceiling is not the repeat ratio, but the computation's ratio to its key-building cost. - The break-even repeat is derived from the measured costs and is 1.63; in the continuous-weight stream, the repeat is 1.01, the threshold is 1.71, and memoization drops to a 0.71x ratio — there, no caching is correct design. - What memoization costs is memory and a wrongness surface: 960 entries against 119,365 entries, and when the computation key does not include the tariff version, 0.5000 of the results come out wrong. #### Next Step All seven patterns so far measured a single owner's work: the system's excess work was shared out among its own streams. One assumption was never questioned — that everyone using a shared resource is on the same side. The shipment tracking and pricing service does not serve a single customer; sellers pass through the same edge, the same store, and the same worker pool. The next lesson picks up the symptom from there: latency for one group of sellers grows even though the total request rate never changes. This too has two causes, and the measure that tells them apart is not the request count — how many units of work one request generates varies sixty-fold from stream to stream, and a measurement that counts share by request can never see the noise. ### Noisy Neighbor Source: https://academia.sh/en/courses/performance-and-monitoring/noisy-neighbor Recognizing insufficient isolation on a shared resource from its symptom: separating the same latency growth's two causes via request share versus work share, the noise ratio climbing from 1.01 to 1.43, the work share jumping from 0.1702 to 0.2415 while the request share never moves, and fair share allocation's cost in the largest tenant's own delay rising from 0.47 to 2.96 rounds. The seven patterns so far each measured a single owner's work. One assumption was never questioned: that everyone using a shared resource is on the same side. The shipment tracking and pricing service does not serve a single customer; sellers all pass through the same edge, the same store, and the same worker pool. The symptom is this: the average wait small sellers see has quadrupled, but the total request rate at the edge never changed. K01's peak read rate stays at 416.67 requests/s; capacity is the same too. This is called a **noisy neighbor**: on a shared resource, one **tenant's** behavior changes the performance the others see. #### Same Symptom, Two Causes **First cause: the small sellers' own rate grew.** No one is bothering anyone; demand genuinely grew and the queue lengthened accordingly. This is not noise, it is a capacity question. **Second cause: a neighbor grew its share.** The small sellers' rate never changed; the largest seller tightened its periodic scan and began consuming a large portion of the shared pool's capacity. Both produce the same symptom. The measurement that separates them is counting **request share** and **work share** separately, per tenant. The work a request generates varies by hundreds of times from one stream to another — the sixth lesson measured this: a periodic scan does work proportional to the seller's own record count. A measurement that counts share by request can never see the noise. #### The Shared Pool The run below is a **model**: a round is an abstract step (one round is one second; the cycle coefficient is an assumption), and no real cluster or tenant is set up. Two regimes are compared — a shared queue that processes in arrival order, and work-conserving round-robin allocation. **KK10 — seller volume.** Two hundred sellers' shipment volume is skewed by rank, and its total is K01's assumption of 400,000 shipments per day. Rationale: a single large seller's volume can carry the whole queue by itself. **KK11 — unit and capacity.** A tracking query is 1 unit, a periodic scan is units equal to one percent of the seller's shipment count; the pool processes 540 units per round. **KK12 — two scenarios.** A: sellers past the fifth rank have a request rate 1.20 times higher. B: the largest seller tightens its scan fivefold. Both grow the usual demand by a similar amount. These assumptions are not added to K01's table. ```js // tenant/sharing.mjs — MODEL of a shared work pool. A round is an abstract step // (one round = one second; the cycle coefficient is an assumption). No real // cluster or tenant is set up. export const CAPACITY = 540; // units/round export const TENANTS = 200; export const INTERVAL = 60; // periodic-scan interval (rounds) export const K01 = { peakReads: 416.67, dailyShipments: 400_000 }; // KK10: seller volume is skewed (share ~ 1/rank), its total is K01's daily shipments. export function sellers(n = TENANTS) { const h = Array.from({ length: n }, (_, i) => 1 / (i + 1)); const t = h.reduce((a, b) => a + b, 0); return h.map((x, i) => ({ name: `S${i + 1}`, share: x / t, shipments: Math.round((x / t) * K01.dailyShipments) })); } // A periodic scan's unit cost is proportional to the seller's own record count (the sixth lesson's measure). export const scan = (s) => Math.max(1, Math.round(s.shipments / 100)); export function run({ round = 600, fairShare = false, bigFrequent = false, smallGrew = false, chunk = 0 }) { const S = sellers(); const backlog = S.map(() => 0); const queue = S.map(() => []), head = S.map(() => 0); // per-tenant queue and head pointer const fifo = []; let fifoHead = 0; // shared-regime overall arrival order const s = S.map(() => ({ requests: 0, units: 0, done: 0, delay: 0, maxDelay: 0 })); const enqueue = (i, units, t) => { // chunk > 0: a large request is split into chunks const n = chunk > 0 ? Math.ceil(units / chunk) : 1; for (let c = 0; c < n; c += 1) { const o = { tenant: i, remaining: Math.ceil(units / n), arrival: t }; if (fairShare) queue[i].push(o); else fifo.push(o); s[i].requests += 1; s[i].units += o.remaining; } }; const finish = (o, t) => { const d = t - o.arrival; s[o.tenant].done += 1; s[o.tenant].delay += d; s[o.tenant].maxDelay = Math.max(s[o.tenant].maxDelay, d); }; for (let t = 1; t <= round; t += 1) { S.forEach((x, i) => { const factor = i >= 5 && smallGrew ? 1.20 : 1; // A: the small sellers' own rate is growing backlog[i] += K01.peakReads * x.share * factor; while (backlog[i] >= 1) { backlog[i] -= 1; enqueue(i, 1, t); } const freq = i === 0 && bigFrequent ? 5 : 1; // B: the largest neighbor is tightening its scan if ((t + i) % Math.round(INTERVAL / freq) === 0) enqueue(i, scan(x), t); }); let budget = CAPACITY; if (fairShare) { // fair share allocation: work-conserving round-robin for (let pass = 0; pass < 12 && budget > 0; pass += 1) { const active = []; for (let i = 0; i < TENANTS; i += 1) if (head[i] < queue[i].length) active.push(i); if (active.length === 0) break; const allotment = Math.max(1, Math.floor(budget / active.length)); const start = t % active.length; // starting point rotates, order favors no one for (let n = 0; n < active.length && budget > 0; n += 1) { const i = active[(start + n) % active.length]; let left = allotment; while (left > 0 && budget > 0 && head[i] < queue[i].length) { const o = queue[i][head[i]]; const taken = Math.min(o.remaining, left, budget); o.remaining -= taken; left -= taken; budget -= taken; if (o.remaining === 0) { finish(o, t); head[i] += 1; } } } } } else { // shared queue: in arrival order while (budget > 0 && fifoHead < fifo.length) { const o = fifo[fifoHead]; const taken = Math.min(o.remaining, budget); o.remaining -= taken; budget -= taken; if (o.remaining === 0) { finish(o, t); fifoHead += 1; } } } } return { S, s }; } ``` ```js // tenant/measure.mjs — splitting request share from work share, and fair share allocation's cost import { CAPACITY, TENANTS, INTERVAL, K01, sellers, scan, run } from "./sharing.mjs"; const WIDTHS = [10, 9, 11, 11, 10, 12, 12]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" ")); const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" ")); const PICK = [0, 9, 49, 199]; // S1, S10, S50, S200 const avg = (x) => (x.delay / x.done).toFixed(2); const smallAvg = (r) => { // S6-S200 together const k = r.s.slice(5); return (k.reduce((a, x) => a + x.delay, 0) / k.reduce((a, x) => a + x.done, 0)).toFixed(2); }; const S = sellers(); const baseline = S.reduce((a, x) => a + K01.peakReads * x.share + scan(x) / INTERVAL, 0); console.log(`capacity ${CAPACITY} units/round, ${TENANTS} sellers, baseline demand ${baseline.toFixed(2)} units/round` + ` (utilization ${(baseline / CAPACITY).toFixed(4)}); the largest seller's daily shipments ${S[0].shipments},` + ` the smallest's ${S[TENANTS - 1].shipments}; periodic scan ${scan(S[0])} and ${scan(S[TENANTS - 1])} units\n`); const DAYS = [["fault-free", {}], ["A: small sellers sped up", { smallGrew: true }], ["B: the largest neighbor tightened its scan", { bigFrequent: true }]]; const SHARED = [], ALLOCATED = []; for (const [name, extra] of DAYS) { const r = run(extra); SHARED.push(r); ALLOCATED.push(run({ ...extra, fairShare: true })); const tReq = r.s.reduce((a, x) => a + x.requests, 0), tUnit = r.s.reduce((a, x) => a + x.units, 0); console.log(`-- ${name} --`); write(["seller", "requests", "req share", "work share", "noise", "avg delay", "max delay"]); line(); for (const i of PICK) { const x = r.s[i]; write([r.S[i].name, x.requests, (x.requests / tReq).toFixed(4), (x.units / tUnit).toFixed(4), (x.units / tUnit / (x.requests / tReq)).toFixed(2), avg(x), x.maxDelay]); } const k = r.s.slice(5); console.log(`S6-S200: avg delay ${smallAvg(r)} rounds, work share ${(k.reduce((a, x) => a + x.units, 0) / tUnit).toFixed(4)},` + ` req share ${(k.reduce((a, x) => a + x.requests, 0) / tReq).toFixed(4)}\n`); } console.log("-- with fair share allocation on: work-conserving round-robin allocation --"); write(["day", "S1 delay", "S10 delay", "S50 delay", "S200 delay", "S6-S200 avg", "S1 max delay"]); line(); DAYS.forEach(([name], i) => { const r = ALLOCATED[i]; write([name.split(":")[0], ...PICK.map((i) => avg(r.s[i])), smallAvg(r), r.s[0].maxDelay]); }); console.log(`\nthe cost: on day B, S1's own average delay goes from ${avg(SHARED[2].s[0])} -> ${avg(ALLOCATED[2].s[0])} rounds,` + ` max from ${SHARED[2].s[0].maxDelay} -> ${ALLOCATED[2].s[0].maxDelay} rounds`); console.log(`even on the fault-free day S1 pays: ${avg(SHARED[0].s[0])} -> ${avg(ALLOCATED[0].s[0])} rounds`); console.log(`accounting: ${TENANTS} counters, 1 touch per request; S1's indivisible request is ${scan(S[0])} units,` + ` its per-round share ${Math.floor(CAPACITY / TENANTS)} units`); console.log("\nsame day B, with the large request chunked (no fair share allocation):"); for (const chunk of [681, 50]) { const r = run({ bigFrequent: true, chunk }); console.log(` largest chunk ${String(chunk).padStart(3)} units (${(chunk / CAPACITY).toFixed(2)} of the round's capacity)` + ` -> S6-S200 avg delay ${smallAvg(r)} rounds`); } ``` ``` capacity 540 units/round, 200 sellers, baseline demand 483.34 units/round (utilization 0.8951); the largest seller's daily shipments 68050, the smallest's 340; periodic scan 681 and 3 units -- fault-free -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42541 0.1689 0.1702 1.01 0.10 2 S10 4263 0.0169 0.0170 1.01 0.20 2 S50 860 0.0034 0.0034 1.00 0.25 2 S200 222 0.0009 0.0008 0.95 0.33 2 S6-S200: avg delay 0.24 rounds, work share 0.6114, req share 0.6143 -- A: small sellers sped up -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42541 0.1506 0.1540 1.02 0.48 2 S10 5113 0.0181 0.0180 1.00 0.72 3 S50 1030 0.0036 0.0036 0.99 0.90 3 S200 265 0.0009 0.0009 0.95 1.12 3 S6-S200: avg delay 0.85 rounds, work share 0.6485, req share 0.6560 -- B: the largest neighbor tightened its scan -- seller requests req share work share noise avg delay max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ S1 42581 0.1690 0.2415 1.43 0.47 3 S10 4263 0.0169 0.0156 0.92 0.85 3 S50 860 0.0034 0.0031 0.91 1.02 3 S200 222 0.0009 0.0008 0.87 1.23 3 S6-S200: avg delay 0.98 rounds, work share 0.5589, req share 0.6142 -- with fair share allocation on: work-conserving round-robin allocation -- day S1 delay S10 delay S50 delay S200 delay S6-S200 avg S1 max delay ---------- --------- ----------- ----------- ---------- ------------ ------------ fault-free 0.48 0.00 0.00 0.00 0.00 7 A 3.27 0.00 0.00 0.00 0.00 13 B 2.96 0.00 0.00 0.00 0.00 9 the cost: on day B, S1's own average delay goes from 0.47 -> 2.96 rounds, max from 3 -> 9 rounds even on the fault-free day S1 pays: 0.10 -> 0.48 rounds accounting: 200 counters, 1 touch per request; S1's indivisible request is 681 units, its per-round share 2 units same day B, with the large request chunked (no fair share allocation): largest chunk 681 units (1.26 of the round's capacity) -> S6-S200 avg delay 0.98 rounds largest chunk 50 units (0.09 of the round's capacity) -> S6-S200 avg delay 1.00 rounds ``` Delays and shares are in the **measurement** class: they are counted over a deterministic arrival sequence. Capacity, unit costs, and the seller distribution are **assumption**; the rates coming from K01 are **computed**. #### Request Share Does Not Reveal the Noise The two scenarios produce the same symptom. Small sellers' average wait is 0.24 rounds on the fault-free day; 0.85 on day A, 0.98 on day B. Someone looking only at this number cannot distinguish the two days. Looking at the shares sharpens the distinction. **On day A** the small sellers' request share climbs from 0.6143 to 0.6560, work share from 0.6114 to 0.6485; the two move together. The largest seller's noise ratio — work share over request share — stays at 1.01. What grows the load is the waiters themselves. **On day B** the small sellers' request share does not change at all: 0.6143 versus 0.6142. Their work share, however, drops from 0.6114 to **0.5589**. The share they lose passes to the largest seller: its work share climbs from 0.1702 to **0.2415**, its noise ratio from 1.01 to **1.43**. Its request share, meanwhile, sits between 0.1689 and 0.1690 — that is, fixed. This is the measurement that makes the distinction. An accounting kept by request count shows no stir in any tenant on day B: the largest seller sends only a handful of extra requests, but each carries 681 units. A small seller's same request is 3 units — a **227-fold** difference. A system that measures share by request cannot see the noisy neighbor, because the quantity it measures is not the quantity the resource is exhausted by. #### Fair Share Allocation and Its Cost When round-robin allocation is turned on, small sellers' wait drops to **0.00 rounds** on all three days. Order is now given by tenant rather than arrival moment, and the remaining budget is not wasted — the regime is work-conserving. The bulkhead pattern from the Resilience and Reliability course is a harder solution to the same problem: the pool is partitioned up front, neighborliness disappears, and its cost was measured there as idle capacity. Fair share allocation here does not leave the resource idle; its cost is elsewhere. The cost falls on the largest tenant. On day B, S1's own average delay rises from 0.47 to **2.96 rounds**, its max delay from 3 to 9 rounds. The reason is in the last line: its one indivisible request is 681 units, but its per-round share is 2 units. It pays even on the fault-free day — 0.10 to 0.48 rounds — while nothing has broken. The second cost is accounting: two hundred counters are kept, and every request adds a record touch. This overhead grows with the tenant count. A third measurement rules out one wrong fix. Chunking the large request by itself changes nothing: when the largest chunk shrinks from 681 to 50 units, the small sellers' wait rises from 0.98 to 1.00 rounds — the chunks still enter the same queue back to back. What fixes it is not the chunk's smallness but giving out turns **round-robin** across tenants. One warning: fair share allocation removes the symptom without distinguishing the cause. On day A too, small sellers' wait drops to zero and S1 pays the cost again (3.27 rounds), even though S1 did not grow that day's load. A share rule imposed without measurement punishes the wrong tenant. #### The Condition Under Which Sharing Is Correct Sharing is not wrong at every scale, and its limit is given by two numbers. **As long as the noise ratio stays close to one.** On the fault-free day, four sellers' ratios fall between 0.95 and 1.01; the pool runs at 0.8951 utilization and average wait is 0.24 rounds. Under this condition, sharing is not just harmless but profitable: no tenant has idle capacity set aside for it. **As long as no single request exceeds the per-round capacity.** S1's scan is 681 units, the round's capacity is 540 — a ratio of 1.26. If an indivisible request takes longer than one round, every tenant behind it waits at least one round, and no rate rule can prevent that. #### Summary - The symptom is small tenants' wait increasing; it has two causes — their own rate grew, or a neighbor grew its share. Both push the wait from 0.24 to 0.85 and 0.98 rounds. - The measurement that separates them is splitting request share from work share. On day A the two grow together; on day B request share goes from 0.6143 to 0.6142 — essentially unchanged — while work share drops to 0.5589. - The noise ratio for the largest seller climbs from 1.01 to 1.43 and its work share jumps from 0.1702 to 0.2415; its request share stays fixed. An accounting kept by request cannot see the noise. - Fair share allocation drops small tenants' wait to 0.00 rounds on all three days; in exchange, S1's delay rises from 0.47 to 2.96 rounds, and from 0.10 to 0.48 rounds even on the fault-free day, and 200 counters are kept. - Chunking the large request alone is not enough (0.98 → 1.00 rounds); what fixes it is giving out turns round-robin across tenants. - Sharing is correct when the noise ratio is close to one and no single request exceeds the per-round capacity; S1's 681-unit scan exceeds the 540-unit round by 1.26 times. #### Next Step Every measurement in this lesson held one number fixed: that the pool processes 540 units per round. Where that number comes from was never asked. The work a process can carry is bounded not by its processor's speed but by how many requests it can advance at the same time, and that bound usually fills up long before the processor does. The next lesson picks up the symptom from there: throughput stops growing even as workers are added, and processor utilization is low. This too has two causes — the downstream dependency is saturated, or the workers are waiting on blocking calls. The measure that separates the two lies not in the workers but in the dependency's own utilization; the workers' wait share comes out nearly identical under both causes and so cannot tell them apart. ### Synchronous I/O Source: https://academia.sh/en/courses/performance-and-monitoring/synchronous-io Measuring blocking calls' effect on capacity: separating the two causes of the same throughput ceiling by dependency utilization (0.9999 and 0.1819), the blocking design capping throughput at the concurrency limit divided by wait plus work time, the worker count needed to carry the same throughput growing 21-fold, and the fix's cost of in-flight requests rising from 8 to 46. The previous lesson's every measurement held one number fixed: that the pool processes 540 units per round. Where that number comes from was never asked. The work a process can carry is bounded not by its processor's speed but by how many requests it can advance at the same time. The symptom is this: throughput capped at 0.36 requests per round, and adding workers does not grow that number. Worker utilization is 0.95 — the pool looks saturated. This lesson shows that appearance can be misleading. #### Same Symptom, Two Causes **First cause: the downstream dependency is saturated.** Throughput is whatever number of calls the dependency can satisfy per round; adding workers changes nothing, because the bottleneck is not in the caller. **Second cause: the workers are waiting on a blocking call.** The dependency sits idle, but each worker cannot advance any other request while it waits for a **blocking call's** response. This is called **synchronous I/O**. In both cases worker utilization comes out close to 1, and adding workers looks like a remedy either way. The measurement that separates them is not on the workers but on the **dependency**: close to 1, the cause is the first; low, the second. Every explanation made without this measurement is a guess. #### What a Blocking Call Actually Does The block below is not a model but a real measurement running on this machine. Two hundred files are read; duration is not measured, because duration depends on this machine. Both measured quantities are deterministic: the **count of other work that advances** while the read is underway, and the **count of operations concurrently in flight**. ```js // synchronous/real.mjs — a real node run: during a blocking call, no other work can // advance. The measured quantity is not duration, but the count of advancing work and // the count of operations concurrently in flight. import { mkdtempSync, writeFileSync, readFileSync, rmSync } from "node:fs"; import { readFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; const N = 200; const dir = mkdtempSync(join(tmpdir(), "sync09-")); for (let i = 0; i < N; i += 1) writeFileSync(join(dir, `k${i}`), "x".repeat(1024)); const path = (i) => join(dir, `k${i}`); // A: blocking call. Work queued before the loop cannot advance until the loop ends. let advanced = 0, inFlight = 0, maxInFlightA = 0; const pendingWork = Promise.resolve().then(() => { advanced += 1; }); for (let i = 0; i < N; i += 1) { inFlight += 1; maxInFlightA = Math.max(maxInFlightA, inFlight); readFileSync(path(i)); inFlight -= 1; } console.log(`blocking: other work advanced during the loop = ${advanced},` + ` max concurrent in-flight = ${maxInFlightA}, files read = ${N}`); await pendingWork; // B: non-blocking call. The same work advances while the calls are underway. let advancedB = 0, inFlightB = 0, maxInFlightB = 0; const pendingWorkB = Promise.resolve().then(() => { advancedB += 1; }); const result = await Promise.all(Array.from({ length: N }, async (_, i) => { inFlightB += 1; maxInFlightB = Math.max(maxInFlightB, inFlightB); const b = await readFile(path(i)); inFlightB -= 1; return b.length; })); await pendingWorkB; console.log(`non-blocking: other work advanced while calls were underway = ${advancedB},` + ` max concurrent in-flight = ${maxInFlightB}, files read = ${result.length}`); console.log(`concurrent in-flight operation ratio = ${maxInFlightB / maxInFlightA}`); rmSync(dir, { recursive: true, force: true }); ``` ``` blocking: other work advanced during the loop = 0, max concurrent in-flight = 1, files read = 200 non-blocking: other work advanced while calls were underway = 1, max concurrent in-flight = 200, files read = 200 concurrent in-flight operation ratio = 200 ``` Two numbers give the whole mechanism. In the blocking regime, work queued before entering the loop **does not advance at all** until the two hundred reads finish, and the count of operations in flight at once stays at 1. In the non-blocking regime the same work advances while the calls are underway, and in-flight operations rise to 200: **200-fold**. What is measured here is not parallelism — how many jobs the runtime actually executes in parallel is a separate question. What is measured is whether the worker is blocked. The language-level asynchronous model was built in the Asynchronous JavaScript and the Runtime course; it is not retold here. The question here is what this difference does to **capacity**. #### Two Measurements That Separate the Two Causes The run below is a **model**: a round is an abstract step and no real process or thread is set up. **KK13 — call profile.** The concurrency limit is 8 workers, local work is 1 round, a blocking call's wait is 20 rounds. Rationale: a remote dependency's response is two orders of magnitude longer than local work; its sensitivity is given in the last table by varying the wait time. **KK14 — cycle coefficient.** One round counts as 1 ms; this is only to convert throughput to K01's requests/s measure. These assumptions are not added to K01's table. ```js // synchronous/capacity.mjs — MODEL of a worker pool. A round is an abstract step; KK14 // counts one round as 1 ms (cycle coefficient assumption). No real process or thread is // set up. export const K01 = { peakEdge: 513.89 }; export function run({ round = 20_000, limit = 8, work = 1, wait = 20, dependency = 2.0, blocking = true, offer = 3 }) { let idleWorkers = limit, queue = 0, tBacklog = 0, jBacklog = 0; const active = []; // { stage: "work" | "token" | "wait", remaining } const s = { done: 0, working: 0, waiting: 0, tokens: 0, maxInFlight: 0 }; for (let t = 1; t <= round; t += 1) { tBacklog += offer; while (tBacklog >= 1) { tBacklog -= 1; queue += 1; } jBacklog += dependency; let token = Math.floor(jBacklog); jBacklog -= token; while (idleWorkers > 0 && queue > 0) { queue -= 1; idleWorkers -= 1; active.push({ stage: "work", remaining: work }); } for (let i = active.length - 1; i >= 0; i -= 1) { const r = active[i]; if (r.stage === "work") { r.remaining -= 1; s.working += 1; if (r.remaining === 0) r.stage = "token"; } else if (r.stage === "token") { if (token > 0) { token -= 1; s.tokens += 1; r.stage = "wait"; r.remaining = wait; if (!blocking) idleWorkers += 1; // the worker starts the call and is freed } else if (blocking) s.waiting += 1; // with no token available, the worker is held } else { r.remaining -= 1; if (blocking) s.waiting += 1; if (r.remaining === 0) { s.done += 1; if (blocking) idleWorkers += 1; active.splice(i, 1); } } } s.maxInFlight = Math.max(s.maxInFlight, active.length); } const workerRounds = s.working + s.waiting; return { throughput: s.done / round, workerUtilization: workerRounds / (limit * round), waitShare: workerRounds === 0 ? 0 : s.waiting / workerRounds, dependencyUtilization: s.tokens / (dependency * round), inFlight: s.maxInFlight, }; } ``` ```js // synchronous/measure.mjs — two causes of the same symptom, two measurements that tell them apart, and the cost import { K01, run } from "./capacity.mjs"; const WIDTHS = [38, 10, 12, 13, 13, 8]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" ")); const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" ")); const row = (name, r) => write([name, r.throughput.toFixed(4), r.workerUtilization.toFixed(4), r.waitShare.toFixed(4), r.dependencyUtilization.toFixed(4), r.inFlight]); const [LIMIT, WORK, WAIT] = [8, 1, 20]; console.log(`concurrency limit ${LIMIT} workers, local work ${WORK} round, blocking call's wait ${WAIT} rounds`); console.log(`arithmetic ceiling: blocking ${LIMIT}/(${WAIT}+${WORK}) = ${(LIMIT / (WAIT + WORK)).toFixed(4)} requests/round,` + ` non-blocking min(${LIMIT}/${WORK}, dependency) requests/round\n`); write(["case", "throughput", "worker util.", "wait share", "dep. util.", "flight"]); line(); row("cause 1: dependency saturated (D=0.35)", run({ dependency: 0.35 })); row("cause 2: blocking call (D=2.0)", run({ dependency: 2.0 })); console.log(); row("cause 1, non-blocking", run({ dependency: 0.35, blocking: false })); row("cause 2, non-blocking", run({ dependency: 2.0, blocking: false })); const blocking = run({ dependency: 2.0 }); const nonBlocking = run({ dependency: 2.0, blocking: false }); console.log(`\ncause 2's gain from the fix: ${blocking.throughput.toFixed(4)} -> ${nonBlocking.throughput.toFixed(4)} requests/round` + ` = ${(nonBlocking.throughput / blocking.throughput).toFixed(2)}x`); console.log(`workers needed to carry the same throughput with the blocking design = ${nonBlocking.throughput} x (${WAIT}+${WORK})` + ` = ${(nonBlocking.throughput * (WAIT + WORK)).toFixed(0)}, that is ${((WAIT + WORK) / WORK).toFixed(0)}x`); console.log(`in-flight requests growing in exchange: ${blocking.inFlight} -> ${nonBlocking.inFlight} (${(nonBlocking.inFlight / blocking.inFlight).toFixed(2)}x)` + ` and dependency utilization ${blocking.dependencyUtilization.toFixed(4)} -> ${nonBlocking.dependencyUtilization.toFixed(4)}`); // KK14: one round counts as 1 ms; throughput is converted to requests/s and compared with K01's peak rate. const ROUND_MS = 1; const perSecond = (r) => (r.throughput * 1000) / ROUND_MS; console.log(`\none round is ${ROUND_MS} ms (KK14): a single process carries blocking ${perSecond(blocking).toFixed(2)} requests/s,` + ` non-blocking ${perSecond(nonBlocking).toFixed(2)} requests/s`); console.log(`K01's peak edge is ${K01.peakEdge} requests/s -> processes needed ` + `${Math.ceil(K01.peakEdge / perSecond(blocking))} and ${Math.ceil(K01.peakEdge / perSecond(nonBlocking))}`); console.log(`\nthe effect of adding workers in the blocking design (D=2.0):`); write(["case", "throughput", "worker util.", "wait share", "dep. util.", "flight"]); line(); for (const limit of [8, 16, 42, 84]) row(`limit ${limit} workers`, run({ limit, dependency: 2.0 })); console.log(`\nKK13's sensitivity: as wait shortens, the blocking design's cost (D=2.0)`); for (const wait of [20, 5, 2, 1]) { const a = run({ wait, dependency: 2.0 }), b = run({ wait, dependency: 2.0, blocking: false }); console.log(` wait ${String(wait).padStart(2)} round(s) -> blocking ${a.throughput.toFixed(4)},` + ` non-blocking ${b.throughput.toFixed(4)}, ratio ${(b.throughput / a.throughput).toFixed(2)}`); } ``` ``` concurrency limit 8 workers, local work 1 round, blocking call's wait 20 rounds arithmetic ceiling: blocking 8/(20+1) = 0.3810 requests/round, non-blocking min(8/1, dependency) requests/round case throughput worker util. wait share dep. util. flight -------------------------------------- ---------- ------------ ------------- ------------- -------- cause 1: dependency saturated (D=0.35) 0.3496 0.9562 0.9542 0.9999 8 cause 2: blocking call (D=2.0) 0.3635 0.9545 0.9524 0.1819 8 cause 1, non-blocking 0.3496 0.0438 0.0000 0.9999 15 cause 2, non-blocking 1.9979 0.2500 0.0000 1.0000 46 cause 2's gain from the fix: 0.3635 -> 1.9979 requests/round = 5.50x workers needed to carry the same throughput with the blocking design = 1.9979 x (20+1) = 42, that is 21x in-flight requests growing in exchange: 8 -> 46 (5.75x) and dependency utilization 0.1819 -> 1.0000 one round is 1 ms (KK14): a single process carries blocking 363.50 requests/s, non-blocking 1997.90 requests/s K01's peak edge is 513.89 requests/s -> processes needed 2 and 1 the effect of adding workers in the blocking design (D=2.0): case throughput worker util. wait share dep. util. flight -------------------------------------- ---------- ------------ ------------- ------------- -------- limit 8 workers 0.3635 0.9545 0.9524 0.1819 8 limit 16 workers 0.7267 0.9544 0.9524 0.3636 16 limit 42 workers 1.9071 0.9542 0.9524 0.9545 42 limit 84 workers 1.9979 0.9755 0.9756 1.0000 82 KK13's sensitivity: as wait shortens, the blocking design's cost (D=2.0) wait 20 round(s) -> blocking 0.3635, non-blocking 1.9979, ratio 5.50 wait 5 round(s) -> blocking 1.1426, non-blocking 1.9994, ratio 1.75 wait 2 round(s) -> blocking 1.9997, non-blocking 1.9997, ratio 1.00 wait 1 round(s) -> blocking 1.9998, non-blocking 1.9998, ratio 1.00 ``` The in-flight counts and the count of advancing work are **measurement**; the throughput and utilizations coming out of the model's deterministic run are **computed**; the call profile and cycle coefficient are **assumption**. #### The Measurement Tells the Cause The first table's two rows give the same symptom: throughput 0.3496 versus 0.3635, worker utilization 0.9562 versus 0.9545, wait share 0.9542 versus 0.9524. The workers look saturated, and in both cases ninety-five percent of their time passes waiting. The distinction is in the column before the last. In the first cause, dependency utilization is **0.9999** — the dependency is genuinely saturated. In the second, **0.1819** — throughput hits its ceiling while four-fifths of the dependency's capacity sits idle. Worker utilization alone cannot separate the two cases; the dependency's own utilization does. The confirmation is in the second table. When the blocking call is removed, nothing changes for the first cause (0.3496 → 0.3496), while for the second, throughput climbs from 0.3635 to **1.9979** — 5.50 times. The same fix works for one cause and not the other — this is why measurement comes before the fix. The ceiling's source is arithmetic and is written in the first line: in the blocking design, throughput cannot exceed the concurrency limit divided by wait plus work time — 8 / (20 + 1) = 0.3810. The model measures 0.3635; the difference comes from the dependency's token distribution being spread across rounds. The third table shows what adding workers does. When the limit rises from 8 to 16, throughput reaches 0.7267; at 42 it reaches 1.9071 — the blocking design carries the same throughput only with **21 times** the workers. Worker utilization stays around 0.95 across all these rows — utilization never reveals the problem, it only says the workers are busy. #### What Grows in Exchange for the Fix The non-blocking design is not free, and its cost can be counted. **In-flight requests.** The count of requests standing in the system at once rises from 8 to 46: 5.75 times. Every in-flight request holds memory — buffer, context, partial response. The concurrency limit bounded this count in the blocking design; the non-blocking design has no such limit, so admission control must be added separately. The Caching, Queues and Asynchronous Processing course's admission control and concurrency limit lessons fill exactly this gap. **The bottleneck relocates.** Dependency utilization climbs from 0.1819 to 1.0000. Once the caller is fixed, the load lands on the dependency, and the next diagnosis will have to happen there. **Carried requests.** With KK14 counting one round as 1 ms, a single process carries 363.50 requests/s in the blocking design and 1997.90 requests/s in the non-blocking design. The process count needed for K01's 513.89 requests/s peak rate ranges between 2 and 1: the same hardware, a twofold difference. #### The Condition Under Which a Blocking Call Is Correct The last table gives the limit. The cost arises from the ratio of wait to local work. At a 20-round wait the ratio is 5.50; at 5 rounds, 1.75; at 2 rounds, **1.00**. Once the wait drops to the same order as local work, the blocking design costs nothing, because the time the worker holds is already spent working. The second condition follows from the first cause: if the dependency is saturated, the gain from not blocking is zero (0.3496 → 0.3496). If the bottleneck is downstream, complicating the caller only grows the in-flight request count. #### Summary - The symptom is throughput hitting a ceiling and adding workers not helping; it has two causes — the downstream dependency is saturated, or the workers are waiting on a blocking call. - Worker utilization cannot separate the two causes: 0.9562 versus 0.9545, wait share 0.9542 versus 0.9524. The measurement that separates them is the dependency's own utilization — 0.9999 versus 0.1819. - In the blocking design, throughput cannot exceed the concurrency limit divided by wait plus work time: 8 / (20 + 1) = 0.3810 requests/round; carrying the same throughput requires 21 times the workers. - The fix raises throughput from 0.3635 to 1.9979 (5.50 times) for the second cause and changes nothing for the first; with one round counted as 1 ms, a single process carries 1997.90 requests/s instead of 363.50. - In exchange, in-flight requests rise from 8 to 46 (5.75 times) and dependency utilization climbs from 0.1819 to 1.0000: with the limit gone, admission control must be added separately. - A blocking call is correct when the wait is the same order as local work: at a 2-round wait the ratio drops to 1.00. #### Next Step All nine patterns rest on one assumption: that a request is made once. The load coming from the edge is set from outside, and the system only carries it. This assumption falls the moment something breaks. When calls start failing, the caller retries, the retries multiply at every layer of the chain, and the load reaching the dependency now comes not from outside but from the system itself. The next lesson is the topic's final diagnosis, and its question is this: how does such a loop look from the outside. A doubling in the call rate reaching a dependency can be the symptom of either a genuine demand increase or a retry storm; the metric that separates the two is not the call count. ### Retry Storm Source: https://academia.sh/en/courses/performance-and-monitoring/retry-storm Recognizing from the outside a feedback loop that amplifies a failure: separating the same call-rate increase's two causes by edge rate, distinct request identity, and success ratio; the call rate climbing from 4.00 to 25.00 during the storm while distinct identity stays at 4.00; the product of call rate and success ratio staying constant at the capacity limit; and throttling the retry allowance costing 180 dropped requests. All nine patterns rest on one assumption: that a request is made once. The load coming from the edge is set from outside, and the system only carries it. This assumption falls the moment something breaks. The symptom is this: the call rate reaching the dependency rose to six times its usual value. The retry decision, exponential backoff, jitter, and retry budget were established and measured in the Application Architecture: Routing, State and Data, Caching, Queues and Asynchronous Processing, and Resilience and Reliability courses; none of it is recomputed here. The schedule is an **input**. The question here is diagnosis: how does such a rate increase look from the outside, and which metric separates it from a genuine demand increase. #### Same Symptom, Two Causes **First cause: external demand genuinely grew.** The request rate coming from the edge has risen, and the call rate reaching the dependency rises with it. The system is healthy; it is only doing more work. **Second cause: external demand is fixed, and the chain is retrying.** The dependency has broken, calls are failing, and every layer of the chain retries on its own. The allowances multiply, and the load reaching the dependency now comes not from outside but from the system itself. This is called a **retry storm**. Both grow the same number at the dependency. An alert that looks only at call rate cannot tell them apart. #### The Chain The run below is a **model**: a round is an abstract step and no real cluster is set up. **KK15 — chain and schedule.** Three layers, three retries per layer, four rounds between retries, the dependency's usual capacity is 12 calls per round, external demand is 4 requests per round. Rationale: the schedule was decided in earlier courses and is taken here as input; the capacity headroom is three times external demand. The fault is capacity dropping to 1 call between rounds 101–160. Its sensitivity is given through the retry allowance. These assumptions are not added to K01's table. ```js // storm/chain.mjs — MODEL of a three-layer chain. A round is an abstract step. The backoff // schedule and retry budget are not computed here; they are taken as an INPUT decided in // earlier courses (fixed interval, fixed allowance per layer). export const KK15 = { layers: 3, allowance: 3, interval: 4, capacity: 12, externalDemand: 4 }; export function run({ round = 400, externalDemand = KK15.externalDemand, allowance = KK15.allowance, faultStart = 101, faultEnd = 160, faultCapacity = 1 }) { const { layers, interval, capacity } = KK15; const edge = [], calls = [], successful = [], dropped = [], identities = []; const active = []; // { id, counters: [k0,k1,k2], time } let nextId = 1, backlog = 0; for (let t = 1; t <= round; t += 1) { backlog += typeof externalDemand === "function" ? externalDemand(t) : externalDemand; let arrived = 0; while (backlog >= 1) { backlog -= 1; arrived += 1; active.push({ id: nextId++, counters: new Array(layers).fill(0), time: t }); } edge.push(arrived); const thisRound = active.filter((r) => r.time === t); let tokens = t >= faultStart && t <= faultEnd ? faultCapacity : capacity; let ok = 0, drop = 0; const seen = new Set(); for (const r of thisRound) { seen.add(r.id); // the dependency sees the call (even when rejected) if (tokens > 0) { tokens -= 1; ok += 1; active.splice(active.indexOf(r), 1); continue; } let k = layers - 1; // counter odometer: carries upward from the deepest layer while (k >= 0) { r.counters[k] += 1; if (r.counters[k] < allowance) break; r.counters[k] = 0; k -= 1; } if (k < 0) { drop += 1; active.splice(active.indexOf(r), 1); } // every allowance is exhausted else r.time = t + interval; } calls.push(thisRound.length); successful.push(ok); dropped.push(drop); identities.push(seen); } return { edge, calls, successful, dropped, identities }; } // Window summary: distinct identity is counted over the whole window, not within a round. export function window(r, start, end) { const sum = (a) => a.slice(start - 1, end).reduce((x, y) => x + y, 0); const n = end - start + 1; const distinct = new Set(); for (const s of r.identities.slice(start - 1, end)) for (const id of s) distinct.add(id); const c = sum(r.calls); return { edge: sum(r.edge) / n, calls: c / n, distinct: distinct.size / n, amplification: distinct.size === 0 ? 0 : c / distinct.size, success: c === 0 ? 0 : sum(r.successful) / c, dropped: sum(r.dropped), }; } ``` ```js // storm/measure.mjs — two causes of the same symptom and the metrics that tell the storm apart import { KK15, run, window } from "./chain.mjs"; const WIDTHS = [24, 11, 14, 16, 14, 8]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" ")); const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" ")); const row = (name, p) => write([name, p.edge.toFixed(2), p.calls.toFixed(2), p.distinct.toFixed(2), p.amplification.toFixed(2), p.success.toFixed(4)]); console.log(`chain ${KK15.layers} layers, ${KK15.allowance} retries per layer, interval ${KK15.interval} rounds,` + ` dependency's usual capacity ${KK15.capacity} calls/round, external demand ${KK15.externalDemand} requests/round`); console.log(`arithmetic ceiling: ${KK15.allowance}^${KK15.layers} = ${KK15.allowance ** KK15.layers} calls/request;` + ` window rounds 101-160\n`); // Cause 1: external demand genuinely grows, no fault. Cause 2: demand fixed, dependency breaks. const organic = run({ externalDemand: (t) => (t >= 101 && t <= 160 ? 8 : 4), faultCapacity: KK15.capacity }); const storm = run({}); const noRetry = run({ allowance: 1 }); write(["case", "edge/round", "calls/round", "distinct/round", "amplification", "success"]); line(); for (const [name, r] of [["demand grew", organic], ["storm", storm], ["no retry", noRetry]]) { row(`${name}: before`, window(r, 1, 100)); row(`${name}: during`, window(r, 101, 160)); row(`${name}: after`, window(r, 161, 400)); } // Recovery: after the fault ends, how many rounds until the call rate drops back to its usual level. const baseline = window(storm, 1, 100).calls; const recovery = (r) => { for (let t = 161; t <= 400; t += 1) if (r.calls[t - 1] <= baseline * 1.1) return t - 160; return -1; }; // Calls satisfied per round = call rate x success ratio; this product is constant at the capacity limit. const product = (r) => { const p = window(r, 101, 160); return (p.calls * p.success).toFixed(2); }; console.log(`\ncall rate x success ratio (calls satisfied/round): demand grew ${product(organic)},` + ` storm ${product(storm)}, no retry ${product(noRetry)}`); console.log(`usual call rate ${baseline.toFixed(2)}/round; fault ends at round 160`); console.log(`recovery ${recovery(storm)} round(s) (no-retry regime: ${recovery(noRetry)} round(s));` + ` the storm's peak call rate ${Math.max(...storm.calls)}/round = ${(Math.max(...storm.calls) / baseline).toFixed(2)}x` + ` the usual rate`); // The cost of the fix: with the allowance throttled, what is gained and lost in the same fault. console.log(`\nthe cost of throttling the allowance (same fault, rounds 101-400):`); write(["allowance", "calls/round", "amplification", "dropped requests", "recovery", "success"]); line(); for (const allowance of [3, 2, 1]) { const r = run({ allowance }); const p = window(r, 101, 160), base = window(r, 1, 100).calls; let g = -1; for (let t = 161; t <= 400; t += 1) if (r.calls[t - 1] <= base * 1.1) { g = t - 160; break; } write([`${allowance} per layer`, p.calls.toFixed(2), p.amplification.toFixed(2), window(r, 101, 400).dropped, g, p.success.toFixed(4)]); } ``` ``` chain 3 layers, 3 retries per layer, interval 4 rounds, dependency's usual capacity 12 calls/round, external demand 4 requests/round arithmetic ceiling: 3^3 = 27 calls/request; window rounds 101-160 case edge/round calls/round distinct/round amplification success ------------------------ ----------- -------------- ---------------- -------------- -------- demand grew: before 4.00 4.00 4.00 1.00 1.0000 demand grew: during 8.00 8.00 8.00 1.00 1.0000 demand grew: after 4.00 4.00 4.00 1.00 1.0000 storm: before 4.00 4.00 4.00 1.00 1.0000 storm: during 4.00 25.00 4.00 6.25 0.0400 storm: after 4.00 6.50 4.75 1.37 0.7308 no retry: before 4.00 4.00 4.00 1.00 1.0000 no retry: during 4.00 4.00 4.00 1.00 0.2500 no retry: after 4.00 4.00 4.00 1.00 1.0000 call rate x success ratio (calls satisfied/round): demand grew 8.00, storm 1.00, no retry 1.00 usual call rate 4.00/round; fault ends at round 160 recovery 25 round(s) (no-retry regime: 1 round(s)); the storm's peak call rate 49/round = 12.25x the usual rate the cost of throttling the allowance (same fault, rounds 101-400): allowance calls/round amplification dropped requests recovery success ------------------------ ----------- -------------- ---------------- -------------- -------- 3 per layer 25.00 6.25 0 25 0.0400 2 per layer 22.33 5.58 68 17 0.0448 1 per layer 4.00 1.00 180 1 0.2500 ``` All the numbers are **computed**, coming out of a deterministic run; the chain, schedule, and fault window are **assumption**. #### How the Storm Looks From the Outside The two scenarios share something indistinguishable at first glance: the call rate reaching the dependency grows in both. Three metrics make the distinction, and all three can be collected at the dependency. **Edge rate.** When demand grows, the edge rate rises from 4.00 to 8.00; in the storm it stays at 4.00. If a call-rate increase happens without a matching change at the edge, the system itself is generating the extra load. **Distinct request identity.** This is the sharpest of the metrics. When demand grows, the distinct identity the dependency sees rises from 4.00 to 8.00 — there are new requests. In the storm, distinct identity **stays fixed at 4.00** while calls climb to 25.00: the same requests arrive again and again. The ratio of call count to distinct identity is the observed **call amplification**, and here it is **6.25** against 1.00. The arithmetic ceiling is 27; the observed value is lower because retries spread across time. **Success ratio.** When demand grows, success stays at 1.0000. In the storm, as the call rate climbs sixfold, success drops from 1.0000 to **0.0400**. When rate and success move in opposite directions, what is growing is not work but the repetition of the same work. A fourth signal is in time. The fault ends at round 160, but the call rate only drops back to its usual level **25 rounds** later; in the regime without retries this takes 1 round. The storm outlives the fault. #### The Signature of a Capacity Limit One number also reveals the storm's nature. The product of call rate and success ratio is the number of calls the dependency actually satisfies. In the storm this product is 1.00; in the no-retry regime, also 1.00. Twenty-five times more calls **does not change** the number satisfied at all — the dependency's capacity is whatever it is, and the retries only shrink each other's share. In the demand-growth scenario the same product is 8.00 — the edge rate itself. This distinction tells when retrying is harmful. If the dependency is dropping individual calls from a transient error, retries grow the product and the work gets done. If the dependency is at its capacity limit, the product is fixed: every new retry crowds out another request's retry. The call amplification measured in the Resilience and Reliability course turns directly into a failure extender in this second case. #### What Grows in Exchange for the Fix The last table counts the cost of the fix. When the per-layer retry allowance drops from 3 to 2, the call rate falls from 25.00 to 22.33 and recovery from 25 rounds to 17 — and **68 requests are dropped**. When the allowance drops to 1, the storm disappears entirely (calls 4.00, amplification 1.00, recovery 1 round), but dropped requests rise to **180**. The trade-off is clear: throttling the storm means requests that go entirely unanswered during the fault window. With an allowance of 3, not a single request is dropped; its cost is the dependency seeing six times more calls and the fault lasting twenty-five rounds longer. This trade-off cannot be made without measurement — both ends of it are numbers. #### The Condition Under Which Retrying Is Correct Retrying is correct as long as call amplification stays below the dependency's capacity headroom. With the numbers here: the usual call rate is 4.00 and capacity is 12 calls/round, so the headroom is 3.00 times. The observed amplification of 6.25 exceeds this headroom, and this is exactly why the storm forms. Had amplification stayed below the headroom, the retries would have cost nothing. The second condition is the nature of the fault. If the product of call rate and success ratio stays constant regardless of call rate, the dependency is at its capacity limit, and increasing the retry allowance saves no additional requests. If the product grows with the call rate, the error is transient and retrying is cheap. #### Summary - The symptom is the call rate reaching the dependency growing; it has two causes — external demand grew, or the chain is retrying. A measurement that looks only at call rate cannot separate them. - Edge rate rises from 4.00 to 8.00 in the first cause and stays at 4.00 in the storm. The distinct request identity the dependency sees rises to 8.00 in the first cause and stays fixed at 4.00 in the storm. - The observed call amplification is 1.00 against 6.25; success ratio 1.0000 against 0.0400. When rate and success move in opposite directions, what is growing is not work but the repetition of the same work. - The storm outlives the fault: even though capacity recovers at round 160, the call rate does not drop back to usual until 25 rounds later; in the regime without retries this takes 1 round. - The product of call rate and success ratio is 1.00 both in the storm and in the no-retry regime; twenty-five times the calls does not change the number satisfied. This is the signature of a dependency at its capacity limit. - The cost of throttling the allowance is dropped requests: 0 at an allowance of 3, 68 at 2, 180 at 1; in exchange, recovery drops from 25 rounds to 1. #### Next Step This topic tied ten symptoms to ten causes, and in each one what made the distinction was the same kind of thing: a number. Busy database was separated by scanned records, busy front end by separating work by its source on the main thread, chatty I/O by the trade-off between call count and bytes carried in per-request touches, extraneous fetching by end-to-end byte count of unneeded fields, improper instantiation by expensive client setup work, monolithic persistence by the index share serving its own class, no caching by repeated computation, noisy neighbor by work share, synchronous I/O by dependency utilization, and retry storm by distinct request identity. All ten of these diagnoses depend on one condition: **that measurement must exist.** Yet which metric to collect was never designed in this topic. Is distinct request identity counted at the dependency; is work share kept per tenant; is the dependency's own utilization observed; is a computation logged together with its key. None of these exist on their own; each is a design decision, and each has a cost. The next topic builds that layer: which metric categories make a complete monitoring surface, what is worth measuring, which threshold triggers an alert, how load is tested, and how many nodes the measured number converts to. ## Monitoring and Capacity Making failure and slowness visible: building the monitoring categories completely as availability, health, performance, utilization, and security; turning which metric is worth collecting into an instrumentation decision; separating the decision-driving visualization from the alert threshold; load, soak, and spike tests asking separate questions; the move from measurement to resource estimate; and tying design decisions to operating cost with a resource unit. ### Monitoring Categories Source: https://academia.sh/en/courses/performance-and-monitoring/monitoring-categories Testing the completeness of a collected metric set with a coverage measure: separating monitoring from observability, defining the availability, health, performance, utilization, and security categories by the question each one asks, counting which diagnosis from the previous topic becomes impossible when a category is never collected, and showing that a diagnosis is built from a composition of metrics rather than a single metric. The previous topic built ten diagnoses and ran each one through the same three steps: the symptom was written as a number, a second cause producing the same symptom was established, and a measurement separating the two was shown. The topic's closing left the condition all ten lessons share — every one of them works **only if a measurement exists.** Where that measurement comes from was never asked in any lesson: which metric is collected, which is not, and which diagnosis the uncollected one takes off the table were all assumed. This lesson fills that gap and opens with a single question: when does the metric set collected from a system count as **complete**? Completeness here is not a list length but a coverage measure — which category being empty makes which diagnosis impossible. The measure is therefore defined over the previous topic's ten lessons. #### Monitoring and Observability **Monitoring** is continuously collecting answers to questions determined in advance; **observability** is being able to answer a question that was never asked in advance from data already collected. This lesson builds monitoring, because the precondition for being able to answer an unasked question is knowing what gap the asked questions leave. A **metric** corresponds to a question, and its category is that question's type. The five categories ask: - **Availability**: was the request answered? The term should not be confused with interface accessibility and is not abbreviated; the distinction was made in the Introduction to System Design course. Its metrics are request success ratio and minutes of outage; the second directly spends that course's monthly 43.2-minute outage budget and the 28.2-minute failure share within it. - **Health**: which replica can take work? The content of the health endpoint — shallow check, readiness check, deep check — was designed in the Resilience and Reliability course and is not redesigned here; the yes/no it produces is placed here as a monitoring metric, unchanged. - **Performance**: how long does the work take? Response-time percentiles and sub-call durations. - **Utilization**: how much of the resource was spent? Utilization, bytes carried, calls per request, objects in memory, cache hit rate, records scanned, work share per customer. - **Security**: who sent the request, and did they have the right to send it? Rejected unauthorized requests and rate-limit triggers. The reason the performance category cannot be read on its own was already measured in that same course: a latency number cannot be read without saying what load it was taken under. A percentile measured at the peak edge rate of 513.89 requests/s and the same percentile measured at midnight are not the same number. This is why every performance metric wants a utilization metric alongside it. #### Completeness Is a Coverage Measure The computation below maps ten diagnoses to sixteen metrics, then empties each category one at a time and counts how many diagnoses become impossible. The mapping itself is this lesson's assumption (**IZ1**): the previous topic showed a distinguishing measurement for every diagnosis, and the table below picks the metrics that correspond to those measurements and places them into categories. A different mapping gives different numbers. The resulting figures belong to the computation's class. ```js // monitoring/coverage.mjs — coverage measure for the five categories: which metric each of the // ten diagnoses in the Anti-Patterns topic requires, and which diagnoses become impossible if a // category is never collected. const METRIC = { "request success ratio": "availability", "minutes of outage": "availability", "replica pool status": "health", "readiness status": "health", "response-time percentile": "performance", "store query duration": "performance", "calls per request": "utilization", "body bytes carried": "utilization", "processor utilization": "utilization", "pool utilization": "utilization", "objects in memory": "utilization", "cache hit rate": "utilization", "records scanned": "utilization", "work share per customer": "utilization", "rejected unauthorized requests": "security", "rate-limit trigger": "security", }; const DIAGNOSIS = { "busy database": ["store query duration", "processor utilization", "response-time percentile"], "busy front end": ["body bytes carried", "response-time percentile"], "chatty I/O": ["calls per request", "response-time percentile"], "over-fetching": ["body bytes carried", "cache hit rate"], "improper instantiation": ["objects in memory", "processor utilization"], "monolithic persistence": ["records scanned", "store query duration", "processor utilization"], "no caching": ["cache hit rate", "processor utilization", "response-time percentile"], "noisy neighbor": ["work share per customer", "response-time percentile", "rate-limit trigger"], "synchronous I/O": ["pool utilization", "readiness status", "response-time percentile"], "retry storm": ["request success ratio", "calls per request", "replica pool status", "rate-limit trigger"], }; const CATEGORY = [...new Set(Object.values(METRIC))]; const categoryOf = (o) => METRIC[o]; const diagnoses = Object.keys(DIAGNOSIS); console.log(`${CATEGORY.length} categories, ${Object.keys(METRIC).length} metrics, ${diagnoses.length} diagnoses`); console.log(`\n${"category".padEnd(18)}${"metrics".padStart(7)}${"diagnoses fed".padStart(16)}` + `${"category empty: diagnoses dropped".padStart(35)}`); const dropped = {}; for (const k of CATEGORY) { const metrics = Object.keys(METRIC).filter((o) => categoryOf(o) === k); dropped[k] = diagnoses.filter((t) => DIAGNOSIS[t].some((o) => categoryOf(o) === k)); console.log(`${k.padEnd(18)}${String(metrics.length).padStart(7)}` + `${String(dropped[k].length).padStart(16)}` + `${`${dropped[k].length}/${diagnoses.length}`.padStart(35)}`); } console.log(`no category can be left empty: the lightest-loaded category is ` + `${CATEGORY.reduce((a, b) => (dropped[a].length <= dropped[b].length ? a : b))}, and even it ` + `drops ${Math.min(...CATEGORY.map((k) => dropped[k].length))} diagnoses`); const counts = Object.keys(METRIC) .map((o) => [o, diagnoses.filter((t) => DIAGNOSIS[t].includes(o)).length]) .sort((a, b) => b[1] - a[1]); const unique = counts.filter(([, n]) => n === 1).map(([o]) => o); const unused = counts.filter(([, n]) => n === 0).map(([o]) => o); console.log(`\nmost shared metric = ${counts[0][0]} (${counts[0][1]}/${diagnoses.length} diagnoses), ` + `second ${counts[1][0]} (${counts[1][1]}/${diagnoses.length})`); console.log(`metrics appearing in exactly one diagnosis (${unique.length}): ${unique.join(", ")}`); console.log(`metrics appearing in no diagnosis (${unused.length}): ${unused.join(", ")}`); console.log(`\n${"diagnosis".padEnd(26)}${"metrics".padStart(8)}${"categories".padStart(12)}` + `${"distinguishing (unique to it)".padStart(31)}`); for (const t of diagnoses) { const categories = new Set(DIAGNOSIS[t].map(categoryOf)); const distinguishing = DIAGNOSIS[t].filter((o) => unique.includes(o)); console.log(`${t.padEnd(26)}${String(DIAGNOSIS[t].length).padStart(8)}${String(categories.size).padStart(12)}` + `${(distinguishing[0] ?? "-").padStart(31)}`); } const pairs = []; for (let i = 0; i < diagnoses.length; i += 1) for (let j = i + 1; j < diagnoses.length; j += 1) { const [a, b] = [diagnoses[i], diagnoses[j]]; const shared = DIAGNOSIS[a].filter((o) => DIAGNOSIS[b].includes(o)); pairs.push({ a, b, shared }); } const maxShared = Math.max(...pairs.map((c) => c.shared.length)); console.log(`\ndiagnosis pairs sharing the most metrics (${maxShared} shared):`); for (const { a, b, shared } of pairs.filter((c) => c.shared.length === maxShared)) console.log(` ${a} / ${b} -> shared ${shared.join(" + ")}; distinguished by ` + `${DIAGNOSIS[a].filter((o) => !shared.includes(o)).join(" + ")} / ` + `${DIAGNOSIS[b].filter((o) => !shared.includes(o)).join(" + ")}`); ``` ``` 5 categories, 16 metrics, 10 diagnoses category metrics diagnoses fed category empty: diagnoses dropped availability 2 1 1/10 health 2 2 2/10 performance 2 7 7/10 utilization 8 10 10/10 security 2 2 2/10 no category can be left empty: the lightest-loaded category is availability, and even it drops 1 diagnoses most shared metric = response-time percentile (6/10 diagnoses), second processor utilization (4/10) metrics appearing in exactly one diagnosis (7): request success ratio, replica pool status, readiness status, pool utilization, objects in memory, records scanned, work share per customer metrics appearing in no diagnosis (2): minutes of outage, rejected unauthorized requests diagnosis metrics categories distinguishing (unique to it) busy database 3 2 - busy front end 2 2 - chatty I/O 2 2 - over-fetching 2 1 - improper instantiation 2 1 objects in memory monolithic persistence 3 2 records scanned no caching 3 2 - noisy neighbor 3 3 work share per customer synchronous I/O 3 3 pool utilization retry storm 4 4 request success ratio diagnosis pairs sharing the most metrics (2 shared): busy database / monolithic persistence -> shared store query duration + processor utilization; distinguished by response-time percentile / records scanned busy database / no caching -> shared processor utilization + response-time percentile; distinguished by store query duration / cache hit rate ``` #### If a Category Is Left Empty The first table gives the coverage measure. **The utilization category feeds ten diagnoses out of ten:** in a system that collects only availability and response time, none of the ten diagnoses can be made. This is not surprising, but the consequence is severe — a symptom's cause requires knowing how much of the resource was spent, yet utilization is the category most often neglected, because anyone can see the service is up without ever looking at it. Performance feeds seven diagnoses. The remaining three categories each feed one or two diagnoses, and **that small number does not make them optional**: if availability is never collected, retry storm cannot be separated from external demand that genuinely grew; if health is never collected, synchronous I/O cannot be distinguished from a storm; if security is never collected, noisy neighbor cannot be distinguished from a storm. The dropped diagnoses are not compensated for by another category; this is why the measure is a coverage, not a total. Two metrics appear in no diagnosis: minutes of outage and rejected unauthorized requests. This is not a gap; it is a consequence of how the category is defined. **A category is defined by the question it answers, not by the diagnoses it enables.** Minutes of outage does not distinguish a single antipattern, but it is the one number that spends the monthly 43.2-minute budget, and no alert threshold can be defended without that budget. A metric set pruned to the diagnosis list loses the questions that fall outside diagnosis. The mapping's sensitivity shows up here too: in IZ1, processor utilization feeds four diagnoses and is busy database's only utilization metric, so losing that one metric drops this diagnosis's entire utilization leg. A different mapping changes the column's numbers, but none leaves any of the five categories carrying no load at all. #### Diagnosis Is a Metric Composition The second block gives the lesson's second result. Response-time percentile appears in six of the ten diagnoses, processor utilization in four. **The most collected metric is the least distinguishing metric.** A response-time chart climbing is the shared symptom of six separate causes and points to none of them on its own; this is the numeric counterpart of the previous topic's line that "slow is not a diagnosis." Seven metrics appear in exactly one diagnosis, and these are signature metrics: objects in memory, work share per customer, records scanned, pool utilization, readiness status, replica pool status, and request success ratio. Five diagnoses have a signature metric, five do not; the latter are separated only by **composition**. The last two lines show how. Busy database and no caching share two metrics — processor utilization and response-time percentile — and a third metric makes the distinction: did store query duration grow, or did cache hit rate fall? In the same way, busy database and monolithic persistence share store query duration and processor utilization; the distinction is made by records scanned, because the symptom of piling onto a single store is a query scanning many times over its own data class. This also shapes the form of the instrumentation decision: a metric is collected not on its own but for the pair it distinguishes. A metric's value comes not from "how many diagnoses it appears in" but from "which two causes it separates." #### Summary - Monitoring collects answers to questions determined in advance; observability is being able to answer a question that was never asked in advance from data already collected. - The five categories are defined by the question each one asks: availability, health, performance, utilization, and security; the content of the health endpoint was designed in the previous course and is placed here only as a category. - Completeness is a coverage measure: the utilization category feeds ten of the ten diagnoses, performance seven; availability feeds one, health and security two each, and none can be left empty. - Minutes of outage and rejected unauthorized requests appear in no diagnosis but are still collected; a category is defined by the question it answers, not by the diagnoses it enables. - The most shared metric is the least distinguishing metric: response-time percentile appears in six diagnoses, so on its own it points to none of them. - Seven of the sixteen metrics appear in exactly one diagnosis and are that diagnosis's signature; five diagnoses have no signature metric and are separated only by composition: busy database and no caching share two metrics, and the distinction is made by store query duration and cache hit rate. #### Next Step This lesson put sixteen metrics into a list and tested the list's completeness with coverage, but never accounted for the list having a cost. Every metric is produced, carried, stored, and read; each one is added to a code path, and that code path runs on every one of the 513.89 requests at the peak edge. As the metric count grows, so does the number of records collected, and at some point the load the monitoring system produces becomes comparable to the load of the system it monitors. The next lesson counts that cost: the records a metric produces per request, the bytes it stores per day, and the code path it adds are weighed on one side against the diagnosis it enables on the other; it measures what information sampling and aggregation lose while making things cheaper. ### Instrumentation Design Source: https://academia.sh/en/courses/performance-and-monitoring/instrumentation-design Weighing both sides of whether a metric is worth collecting: computing its cost in records produced, bytes stored, and code path added; measuring on the same population the volume sampling and aggregation gain and the information they lose; comparing the percentile error of two similarly-priced forms; and choosing the aggregation form by the diagnosis it enables. The previous lesson placed sixteen metrics into five categories and tested the list's completeness with coverage. The test was one-sided: every metric was evaluated only by the diagnosis it enables, and none of them had its cost counted. Yet every metric is produced, carried, stored, and added to a code path; the code path it is added to runs on every one of the 513.89 requests at the peak edge. This lesson makes the decision two-sided. The answer to whether a metric is worth collecting comes from comparing its cost against the diagnosis it enables. Both are converted to a number. #### The Metric's Three Cost Items **Instrumentation** is the calls added to code to produce a metric. Its cost has three items. **Records produced**: how many metric records are born per unit of time. **Bytes stored**: how much space those records take up, and for how long. **Code path added**: how many instrumentation calls run per request. The third item differs from the first two and cannot be counted directly — incrementing a counter is not the same work as building an event record and queuing it. In this lesson, code path is represented by **records produced per request**: a form that produces one record per request runs more code path than one that produces one record per second. The computation below's first column is therefore both the records item and the code path item. Monthly volume is written in **GB-months**: one gigabyte held for one month. No currency is used, because a price depends on a provider and a date; a resource unit does not. This rule holds for the rest of the course. ```js // monitoring/cost.mjs — the cost of five aggregation forms: records produced, bytes stored, and // the ratio to the system's own write rate. Inputs are K01's figures and this lesson's assumptions. const PEAK_EDGE = 513.89; // K01: peak edge requests/s const PEAK_WRITE = 97.22; // K01: peak write requests/s (the system's own persistent record) const EVENT_BYTES = 180; // IZ2 (assumption): one raw metric record (name, value, time, three tags) const BUCKET_BYTES = 40; // IZ3 (assumption): one aggregated bucket record const BUCKETS = 20; // IZ4 (assumption): histogram bucket count const RETENTION = 30; // IZ5 (assumption): raw record retention (days) const b = (x, n = 2) => x.toFixed(n); const FORM = [ ["raw event", PEAK_EDGE, EVENT_BYTES], ["1/10 sample", PEAK_EDGE / 10, EVENT_BYTES], ["1/100 sample", PEAK_EDGE / 100, EVENT_BYTES], ["per-second histogram", BUCKETS, BUCKET_BYTES], ["per-second counter+sum", 2, BUCKET_BYTES], ]; console.log(`${"aggregation form".padEnd(23)}${"records/s".padStart(10)}${"bytes/s".padStart(10)}` + `${"daily GB".padStart(11)}${"GB-month (30 days)".padStart(20)}${"system write ratio".padStart(23)}`); for (const [name, rate, bytes] of FORM) { const dailyGB = (rate * bytes * 86_400) / 1e9; console.log(`${name.padEnd(23)}${b(rate).padStart(10)}${b(rate * bytes, 0).padStart(10)}` + `${b(dailyGB, 3).padStart(11)}${b(dailyGB * RETENTION, 2).padStart(20)}` + `${(b(rate / PEAK_WRITE) + "x").padStart(23)}`); } const raw = PEAK_EDGE * EVENT_BYTES * 86_400 / 1e9; const hist = BUCKETS * BUCKET_BYTES * 86_400 / 1e9; console.log(`\nraw event / per-second histogram = ${b(raw / hist, 0)}x volume`); console.log(`sampling grows with request rate, aggregation does not: if the request rate rises to 2x, ` + `raw becomes ${b(2 * raw, 3)} GB/day, histogram stays ${b(hist, 3)} GB/day`); console.log(`if 10 metrics are collected in the same form, volume rises to 10x: raw becomes ${b(10 * raw, 2)} GB/day`); ``` ``` aggregation form records/s bytes/s daily GB GB-month (30 days) system write ratio raw event 513.89 92500 7.992 239.76 5.29x 1/10 sample 51.39 9250 0.799 23.98 0.53x 1/100 sample 5.14 925 0.080 2.40 0.05x per-second histogram 20.00 800 0.069 2.07 0.21x per-second counter+sum 2.00 80 0.007 0.21 0.02x raw event / per-second histogram = 116x volume sampling grows with request rate, aggregation does not: if the request rate rises to 2x, raw becomes 15.984 GB/day, histogram stays 0.069 GB/day if 10 metrics are collected in the same form, volume rises to 10x: raw becomes 79.92 GB/day ``` The last column is the measure's harshest form. Dropping one metric record onto every request produces **5.29 times** as many records as the system's own persistent write: the shipment tracking service writes 97.22 state events per second while monitoring writes 513.89 records. The previous lesson's list had sixteen metrics; if all of them were collected this way, the monitoring load would stop being a small line item next to the system's own load. The last line says this directly: ten metrics in raw-event form produce 79.92 GB per day. The two forms depend on two separate kinds of growth, and the difference decides the design. **Sampling grows with request rate**, because what is sampled is requests; when the request rate doubles, raw volume rises from 7.992 to 15.984 GB. **Aggregation does not grow with request rate**, because the number of records produced is set by the bucket count and the aggregation interval; at the same doubling, the histogram stays at 0.069 GB. In a scaling system, this is the question of whether the monitoring cost grows along with the scale. #### Gains Are Measured in Loss Every form that lowers the cost also loses something, and the decision cannot be made until that loss is named. To measure the loss, all five forms are applied to a single population at once. The setup below is a **model**: latencies are produced with integer arithmetic and a visible seed, so the run repeats independent of machine. The model does not predict a real distribution; what it shows is what the forms do to the same data. The quantity measured is **tail latency**, the slowest end of the distribution — it has nothing to do with the queue as a line of waiting jobs; here, the tail is the distribution's tail. Percentiles were defined in the Introduction to System Design course and are not redefined here. ```js // monitoring/loss.mjs — information-loss MODEL of sampling and aggregation. Latencies are // produced with integer arithmetic (seed visible), so the run repeats independent of machine. const N = 200_000; // model population: 400 seconds x 500 requests/second const WINDOW = 500; // requests per second (close to K01's 513.89 peak rate) const SEED = 20260730; let s = SEED; const rand = (n) => Math.floor((((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32)) * n); // Latencies are integers in units of 0.1 ms: body, mid-tail, far-tail, farthest-tail. const LAYER = [[900, 30, 30], [90, 60, 140], [9, 200, 700], [1, 900, 3100]]; const data = []; for (const [share, base, width] of LAYER) for (let i = 0; i < (N * share) / 1000; i += 1) data.push(base + rand(width)); for (let i = data.length - 1; i > 0; i -= 1) { // seeded shuffle const j = rand(i + 1); [data[i], data[j]] = [data[j], data[i]]; } const percentile = (arr, p) => { const d = [...arr].sort((a, b) => a - b); return d[Math.min(d.length - 1, Math.floor((p / 100) * d.length))]; }; const ms = (x) => (x / 10).toFixed(1); const maxOf = (d) => d.reduce((a, c) => (c > a ? c : a), 0); const EDGE = [30, 40, 50, 60, 80, 100, 130, 170, 220, 290, 380, 500, 650, 850, 1100, 1500, 2000, 2600, 3400, Infinity]; const bucket = new Array(EDGE.length).fill(0); for (const v of data) bucket[EDGE.findIndex((k) => v < k)] += 1; const bucketPercentile = (p) => { let total = 0; const target = (p / 100) * data.length; for (let i = 0; i < bucket.length; i += 1) { total += bucket[i]; if (total >= target) return EDGE[i]; } return Infinity; }; const bucketMax = EDGE[bucket.findLastIndex((k) => k > 0)]; const averages = []; for (let i = 0; i < data.length; i += WINDOW) averages.push(data.slice(i, i + WINDOW).reduce((a, c) => a + c, 0) / WINDOW); const everyKth = (k) => data.filter((_, i) => i % k === 0); const exact = { p50: percentile(data, 50), p99: percentile(data, 99), p999: percentile(data, 99.9), max: maxOf(data) }; console.log(`seed ${SEED}, ${N} requests, ${data.length / WINDOW} seconds, ${EDGE.length} buckets`); console.log(`true value: p50 ${ms(exact.p50)} ms, p99 ${ms(exact.p99)} ms, ` + `p99.9 ${ms(exact.p999)} ms, maximum ${ms(exact.max)} ms`); console.log(`\n${"aggregation form".padEnd(23)}${"records stored".padStart(15)}${"p50".padStart(8)}` + `${"p99".padStart(8)}${"p99.9".padStart(9)}${"maximum".padStart(10)}${"p99 error".padStart(12)}`); const print = (name, records, p50, p99, p999, max) => console.log(`${name.padEnd(23)}${records.toLocaleString("en-US").padStart(15)}${ms(p50).padStart(8)}` + `${ms(p99).padStart(8)}${ms(p999).padStart(9)}${ms(max).padStart(10)}` + `${(((p99 - exact.p99) / exact.p99) * 100).toFixed(1).padStart(11)}%`); print("raw event", data.length, exact.p50, exact.p99, exact.p999, exact.max); for (const k of [10, 100]) { const o = everyKth(k); print(`1/${k} sample`, o.length, percentile(o, 50), percentile(o, 99), percentile(o, 99.9), maxOf(o)); } print("per-second histogram", (data.length / WINDOW) * EDGE.length, bucketPercentile(50), bucketPercentile(99), bucketPercentile(99.9), bucketMax); print("per-second counter+sum", (data.length / WINDOW) * 2, percentile(averages, 50), percentile(averages, 99), percentile(averages, 99.9), maxOf(averages)); const overallAvg = data.reduce((a, c) => a + c, 0) / data.length; const avgOfAvg = averages.reduce((a, c) => a + c, 0) / averages.length; console.log(`\nthe average is preserved exactly in both forms: raw ${ms(overallAvg)} ms, ` + `the average of per-second averages ${ms(avgOfAvg)} ms`); const slow = data.filter((v) => v >= 200).length; console.log(`requests over 200 ms: ${slow} in raw data, ${everyKth(100).filter((v) => v >= 200).length} ` + `in the 1/100 sample (expected ${(slow / 100).toFixed(1)})`); ``` ``` seed 20260730, 200000 requests, 400 seconds, 20 buckets true value: p50 4.6 ms, p99 20.0 ms, p99.9 90.3 ms, maximum 399.5 ms aggregation form records stored p50 p99 p99.9 maximum p99 error raw event 200,000 4.6 20.0 90.3 399.5 0.0% 1/10 sample 20,000 4.6 20.4 91.3 397.6 2.0% 1/100 sample 2,000 4.6 38.6 351.8 397.6 93.0% per-second histogram 8,000 5.0 22.0 110.0 Infinity 10.0% per-second counter+sum 800 5.8 7.9 8.8 8.8 -60.5% the average is preserved exactly in both forms: raw 5.9 ms, the average of per-second averages 5.9 ms requests over 200 ms: 2000 in raw data, 22 in the 1/100 sample (expected 20.0) ``` #### Same Cost, Different Loss Reading the two tables side by side gives the lesson's main result. **The 1/100 sample costs 2.40 GB-months, the per-second histogram 2.07 GB-months** — their costs are close. Their losses are not: the sample's p99 error is 93.0 percent, the histogram's 10.0 percent. In the sample, p99.9 becomes entirely meaningless, because at 2,000 records that percentile rests on only two records; it reads 351.8 ms instead of the true 90.3 ms. **When a form exists that loses far less information at the same cost, choosing the form that loses more is not a decision — it is an oversight.** What sampling loses stands out plainly in the last line: the raw data has 2,000 requests over 200 ms, and the 1/100 sample sees 22 of them. The number is close to the expected 20.0, so the sample is not biased — what is lost is precision. The more extreme a percentile, the fewer records carry it, and sampling erases the most extreme information first. The 1/10 sample still holds 20,000 records and its p99 error is 2.0 percent; this shows that the loss does not grow linearly with the sampling ratio. Aggregation's loss takes a different form. The histogram counts every request, drops none, but rounds the value to the bucket boundary: p99 reads 22.0 instead of 20.0, and the error comes from the bucket width. The maximum column, meanwhile, shows `Infinity` — because the top bucket has no upper bound, the histogram cannot bound the maximum latency. Bringing it back means storing one more number per second: the maximum. One number costs 40 bytes. The per-second counter and sum, meanwhile, erase the distribution entirely. The average is preserved exactly in both forms (5.9 ms), but the per-second averages' p99 is 7.9 ms and the largest value it sees is 8.8 ms; the true maximum is 399.5 ms. When five hundred requests are averaged, one slow request is swallowed by four hundred ninety-nine fast ones. **A metric that reports average latency does not measure tail latency; this form's cheapness comes from precisely that information being thrown away.** #### The Decision Rule Three results give the rule for the per-metric decision. First: an aggregation form is chosen not by its cost but by **the information it preserves per unit of cost**. Second: the form is chosen by the diagnosis the metric will enable. The histogram answers "how slow" cheaply but cannot answer "which request was slow," because it has no record identity; the sample does the opposite — it holds few but complete records, giving a request's trace along with its tags. Of the previous lesson's ten diagnoses, the ones separated by metric composition want complete records; those that only need to see a threshold crossed are satisfied with a histogram. Third is the scaling rule: raw events and samples grow with request rate, aggregation does not. A sample that costs 2.40 GB-months today at 513.89 requests/s costs 4.80 GB-months once traffic doubles; under the same condition, the histogram stays at 2.07 GB-months. A monitoring design cannot be made independent of the system's scaling plan. #### Summary - A metric's cost has three items: records produced, bytes stored, and code path added; the third is represented by records produced per request. Volume is written in GB-months, not currency. - A single metric that drops a record onto every request produces 5.29 times as many records as the system's own persistent write (513.89 against 97.22 records/s); ten metrics collected the same way come to 79.92 GB per day. - Sampling grows with request rate, aggregation does not: when the request rate doubles, raw volume rises from 7.992 to 15.984 GB/day, the histogram stays at 0.069 GB/day. - The 1/100 sample and the per-second histogram are close in cost (2.40 and 2.07 GB-months), but their p99 errors are 93.0 percent and 10.0 percent; the form that loses less at the same cost is chosen. - Sampling erases the most extreme information first (22 of 2,000 slow requests are seen); aggregation rounds the value to the bucket boundary and, because the top bucket is unbounded, cannot bound the maximum value at all. - Storing an average erases the distribution: the largest value the per-second averages see is 8.8 ms while the true maximum is 399.5 ms; the average itself is preserved as 5.9 ms in both forms. #### Next Step These two lessons decided what to collect and in what form. The collected number still tells no one anything: it will be plotted somewhere, or it will wake someone up. The two are not the same thing, and they do not work with the same threshold. A plot answers the question of someone already looking at it; an alert calls someone who is not looking to look, and it has a cost when it calls wrongly. The next lesson separates the two, ties where the alert threshold comes from to the Introduction to System Design course's monthly 43.2-minute outage budget, and tests the threshold choice with two numbers: how many false alerts it produces and how many real events it misses. ### Dashboards and Alerts Source: https://academia.sh/en/courses/performance-and-monitoring/dashboards-and-alerts Turning a collected metric into a decision: showing that a dashboard and an alert answer separate questions; scanning threshold and window pairs over a month to count false alerts and missed events; measuring that raising the threshold misses events and that widening the window only cuts short-lived noise; and deriving the threshold from K01's monthly outage budget as a burn rate. The previous two lessons decided what to collect, in what form, and at what cost. The collected number still tells no one anything. It becomes a decision one of two ways: it is plotted somewhere and read by someone already looking, or it crosses a threshold and calls someone who is not. Both paths use the same number, and **they do not work with the same threshold**. This lesson separates the two, then tests the calling path's one parameter — the threshold — with a scan, and ties where it comes from to the Introduction to System Design course's monthly outage budget. #### Two Separate Questions A **dashboard** answers the question of someone already looking. Its question is: is this normal right now, and if not, in which direction? This is why every panel is written next to its **expected range**; a line chart without a range only shows a shape going up or down and carries no decision. An **alert** calls someone who is not looking. Its question is different: should something be done now? One rule follows from this — **every alert is tied to an action.** If there is no work to do in response, the collected metric goes into the dashboard, not an alert. This distinction is not about tool choice; it is about producing separate answers to two separate questions. The first screen's content follows from the previous lesson's category list: one metric per category, together with the load measured. A performance panel cannot be read without a utilization panel beside it, because the same response time does not say the same thing at 513.89 requests/s and at midnight. **A panel's expected range and an alert threshold are not the same number:** the range answers "is this normal," the threshold answers "should I wake someone up," and the second is always wider. #### A Threshold Is a Matter of Scanning A threshold is measured by two errors. A **false alert** is one triggered when no real event is present, and its cost is direct: each one takes a person's attention, and piling up lowers every alert's credibility. A **missed alert** is one that does not trigger during a real event, and its cost is paid out of the budget. The setup below is a **model**: for each of a month's 43,200 minutes, an error count is produced with seeded integer arithmetic, six real events are placed on a known schedule (IZ6), and noise is defined by a separate assumption (IZ7). The model does not predict a real failure distribution; what it tests is what different threshold and window pairs do on the same data. ```js // alert/scan.mjs — MODEL of an alert threshold scan. For each minute of a month, an error count // is produced with seeded integer arithmetic; for each threshold-window pair, false alerts, // missed events, and detection delay are counted. const MINUTES = 43_200; // K01: a 30-day month const REQUESTS_PER_MIN = 30_833; // K01: 513.89 requests/s x 60 const SEED = 20260731; let s = SEED; const rand = (n) => Math.floor((((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32)) * n); // IZ6 (assumption): six real events in a month [start minute, duration, severity (per thousand)] const EVENT = [[3000, 240, 8], [8000, 30, 15], [15000, 3, 30], [21000, 20, 120], [30000, 4, 800], [38000, 45, 400]]; // IZ7 (assumption): noise model — baseline error and a one-minute spike every 240 minutes const error = new Array(MINUTES); for (let t = 0; t < MINUTES; t += 1) { let n = rand(40); if (rand(240) === 0) n += rand(1000); error[t] = n; } for (const [start, duration, severity] of EVENT) for (let t = start; t < start + duration; t += 1) error[t] += Math.round((REQUESTS_PER_MIN * severity) / 1000); const ratio = error.map((h) => h / REQUESTS_PER_MIN); const eventAt = (t) => EVENT.findIndex(([start, duration]) => t >= start && t < start + duration); const lostMinutes = EVENT.reduce((a, [, duration, severity]) => a + (duration * severity) / 1000, 0); console.log(`seed ${SEED}, ${MINUTES} minutes, ${EVENT.length} real events, ` + `total ${EVENT.reduce((a, o) => a + o[1], 0)} minutes of degradation`); console.log(`what the events eat from the budget: ${lostMinutes.toFixed(2)} minute-equivalents ` + `(K01's failure share is 28.2 minutes)`); const noise = ratio.filter((_, t) => eventAt(t) < 0); console.log(`noise: highest minute ${(noise.reduce((a, c) => (c > a ? c : a), 0) * 100).toFixed(2)}%, ` + `minutes exceeding the 0.5% threshold ${noise.filter((o) => o >= 0.005).length}, ` + `minutes exceeding the 2% threshold ${noise.filter((o) => o >= 0.02).length}`); function scan(threshold, window) { const r = { alerts: 0, falseAlerts: 0, caught: new Set(), delay: [] }; let active = false; for (let t = window - 1; t < MINUTES; t += 1) { let all = true; for (let k = 0; k < window; k += 1) if (ratio[t - k] < threshold) all = false; if (all === false) { active = false; continue; } if (active) continue; active = true; r.alerts += 1; const i = eventAt(t); if (i < 0) r.falseAlerts += 1; else if (r.caught.has(i) === false) { r.caught.add(i); r.delay.push(t - EVENT[i][0]); } } return r; } const THRESHOLD = [0.005, 0.01, 0.02, 0.05]; const WINDOW = [1, 3, 5]; console.log(`\n${"threshold".padStart(11)}${"window".padStart(9)}${"alerts".padStart(7)}` + `${"false alerts".padStart(14)}${"events caught".padStart(16)}${"missed".padStart(11)}` + `${"avg delay".padStart(14)}${"budget lost (min)".padStart(19)}`); for (const e of THRESHOLD) for (const w of WINDOW) { const r = scan(e, w); const missedBudget = EVENT.filter((_, i) => r.caught.has(i) === false) .reduce((a, [, duration, severity]) => a + (duration * severity) / 1000, 0); const avg = r.delay.length ? r.delay.reduce((a, c) => a + c, 0) / r.delay.length : 0; console.log(`${((e * 100).toFixed(1) + "%").padStart(11)}${(w + " min").padStart(9)}` + `${String(r.alerts).padStart(7)}${String(r.falseAlerts).padStart(14)}` + `${`${r.caught.size}/${EVENT.length}`.padStart(16)}` + `${String(EVENT.length - r.caught.size).padStart(11)}` + `${(avg.toFixed(1) + " min").padStart(14)}${missedBudget.toFixed(2).padStart(19)}`); } ``` ``` seed 20260731, 43200 minutes, 6 real events, total 342 minutes of degradation what the events eat from the budget: 26.06 minute-equivalents (K01's failure share is 28.2 minutes) noise: highest minute 3.34%, minutes exceeding the 0.5% threshold 149, minutes exceeding the 2% threshold 69 threshold window alerts false alerts events caught missed avg delay budget lost (min) 0.5% 1 min 155 149 6/6 0 0.0 min 0.00 0.5% 3 min 6 0 6/6 0 2.0 min 0.00 0.5% 5 min 4 0 4/6 2 4.0 min 3.29 1.0% 1 min 131 125 6/6 0 2.0 min 0.00 1.0% 3 min 5 0 5/6 1 2.0 min 1.92 1.0% 5 min 3 0 3/6 3 4.0 min 5.21 2.0% 1 min 74 69 5/6 1 2.4 min 0.45 2.0% 3 min 4 0 4/6 2 2.0 min 2.37 2.0% 5 min 2 0 2/6 4 4.0 min 5.66 5.0% 1 min 3 0 3/6 3 0.0 min 2.46 5.0% 3 min 3 0 3/6 3 2.0 min 2.46 5.0% 5 min 2 0 2/6 4 4.0 min 5.66 ``` #### How to Shrink Both Errors at Once The table is read on two axes, and the two are not the same thing. **Raising the threshold cuts both false alerts and real events.** At the 0.5 percent threshold and a one-minute window, 155 alerts trigger in a month, 149 of them false — five times a day, for nothing. When the threshold is raised to 5 percent, false alerts drop to zero, but events caught fall from 6 to 3 and the missed events eat 2.46 minutes from the budget. This is the cost of cutting false alerts with the threshold: a threshold that rises above the noise also rises above the mild events. **Widening the window cuts only short-lived noise.** At the same 0.5 percent threshold, when the window is extended to three minutes, false alerts drop from 149 to 0, all six events are still caught, and the budget lost stays at 0.00 minutes. The cost is only a two-minute detection delay. The reason is written into the model: noise lasts one minute, while real events last at least three. **The window is chosen longer than the noise's duration and shorter than the shortest real event.** The five-minute window shows the limit: at the same threshold, false alerts are again 0, but the two events lasting three and four minutes are no longer seen, and 3.29 minutes leak from the budget. Missing begins the moment the window passes the shortest event. One row also shows a misleading success. At the 1.0 percent threshold and a one-minute window, all six of six events appear to be caught, yet the first event's 0.8 percent severity stays below the threshold. It is caught because a noise spike happens to land inside that event's 240 minutes. This is coincidence, not diagnosis, and the scan table reads correctly only once the reason is known. #### Where the Threshold Comes From The scan finds the best pair, but it does not say **which number** the threshold should be; 0.5 percent and 2 percent both look defensible. The number's source is not the metric itself but the service's commitment. The Introduction to System Design course established the monthly budget: the 99.9 percent target is 43.2 minutes, of which 15.0 minutes go to planned work and 28.2 minutes to failure. The percentage arithmetic is not repeated here; these three numbers are inputs. Dividing the failure share by a month gives **a continuously permitted error rate**, and the observed rate divided by that rate is the **burn rate**. ```js // alert/budget.mjs — deriving the alert threshold from K01's outage budget: burn rate, the // threshold's error-rate equivalent, and the budget spent by the time the alert triggers. All of // it is arithmetic. const MONTH_MIN = 43_200; // K01: a 30-day month (minutes) const MONTHLY_BUDGET = 43.2; // K01: monthly outage budget for 99.9% (minutes) const FAILURE_SHARE = 28.2; // K01: the budget's failure share (minutes) const allowed = FAILURE_SHARE / MONTH_MIN; // request-based permitted error rate const b = (x, n = 2) => x.toFixed(n); console.log(`monthly budget ${MONTHLY_BUDGET} min, failure share ${FAILURE_SHARE} min, month ${MONTH_MIN} min`); console.log(`the continuously permitted error rate = ${b(allowed * 100, 4)}% (burn rate 1.0)`); const WINDOW = [["5 min", 5], ["1 hr", 60], ["6 hr", 360]]; console.log(`\n${"burn rate".padStart(13)}${"error rate threshold".padStart(22)}` + `${"share exhausted in".padStart(20)}` + `${WINDOW.map(([a]) => `${a} window`.padStart(16)).join("")}`); for (const rate of [1, 3, 6, 14.4]) { const errRate = allowed * rate; const row = WINDOW.map(([, min]) => `${b((100 * min * errRate) / FAILURE_SHARE)}%`.padStart(16)).join(""); console.log(`${b(rate, 1).padStart(13)}${(b(errRate * 100, 3) + "%").padStart(22)}` + `${(b(30 / rate, 2) + " days").padStart(20)}${row}`); } console.log(`window columns: how much of the failure share has been spent when the alert triggers`); const SCAN = [0.005, 0.01, 0.02, 0.05]; console.log(`\nthe previous scan's thresholds in budget terms:`); for (const e of SCAN) console.log(` ${b(e * 100, 1)}% threshold -> burn rate ${b(e / allowed, 1)}, ` + `at this rate the failure share runs out in ${b(FAILURE_SHARE / (e * 1440), 2)} days`); // IZ6 (assumption): the previous block's six events [duration (min), severity (per thousand)] const EVENT = [[240, 8], [30, 15], [3, 30], [20, 120], [4, 800], [45, 400]]; console.log(`\n${"event".padStart(5)}${"duration".padStart(10)}${"severity".padStart(9)}` + `${"burn rate".padStart(14)}${"consumed (min)".padStart(16)}${"share consumed".padStart(15)}` + `${"exhausted at this rate".padStart(23)}`); let total = 0; EVENT.forEach(([duration, severity], i) => { const rate = severity / 1000, consumed = duration * rate; total += consumed; console.log(`${String(i + 1).padStart(5)}${(duration + " min").padStart(10)}` + `${(b(rate * 100, 1) + "%").padStart(9)}${b(rate / allowed, 1).padStart(14)}` + `${b(consumed).padStart(16)}${(b((100 * consumed) / FAILURE_SHARE, 1) + "%").padStart(15)}` + `${(b(FAILURE_SHARE / (rate * 1440), 2) + " days").padStart(23)}`); }); console.log(`total ${b(total)} min, ${b((100 * total) / FAILURE_SHARE, 1)}% of the failure share; ` + `remaining share ${b(FAILURE_SHARE - total)} min`); ``` ``` monthly budget 43.2 min, failure share 28.2 min, month 43200 min the continuously permitted error rate = 0.0653% (burn rate 1.0) burn rate error rate threshold share exhausted in 5 min window 1 hr window 6 hr window 1.0 0.065% 30.00 days 0.01% 0.14% 0.83% 3.0 0.196% 10.00 days 0.03% 0.42% 2.50% 6.0 0.392% 5.00 days 0.07% 0.83% 5.00% 14.4 0.940% 2.08 days 0.17% 2.00% 12.00% window columns: how much of the failure share has been spent when the alert triggers the previous scan's thresholds in budget terms: 0.5% threshold -> burn rate 7.7, at this rate the failure share runs out in 3.92 days 1.0% threshold -> burn rate 15.3, at this rate the failure share runs out in 1.96 days 2.0% threshold -> burn rate 30.6, at this rate the failure share runs out in 0.98 days 5.0% threshold -> burn rate 76.6, at this rate the failure share runs out in 0.39 days event duration severity burn rate consumed (min) share consumed exhausted at this rate 1 240 min 0.8% 12.3 1.92 6.8% 2.45 days 2 30 min 1.5% 23.0 0.45 1.6% 1.31 days 3 3 min 3.0% 46.0 0.09 0.3% 0.65 days 4 20 min 12.0% 183.8 2.40 8.5% 0.16 days 5 4 min 80.0% 1225.5 3.20 11.3% 0.02 days 6 45 min 40.0% 612.8 18.00 63.8% 0.05 days total 26.06 min, 92.4% of the failure share; remaining share 2.14 min ``` The first table converts the threshold into a measure of time. A burn rate of 1.0 finishes the budget exactly at month's end; 14.4 finishes it within two days. Choosing a threshold means answering the question "at what rate do I want to be woken up," and the window column shows its cost: waiting six hours at a burn rate of 14.4 means 12 percent of the failure share is already spent by the time the alert arrives. A short window wakes someone early and is exposed to noise; a long window is quiet and expensive. **This is the trade-off measured in the scan, written out in terms of the budget.** The second block converts the scan's four thresholds into the same language and removes the arbitrariness: a 0.5 percent threshold means a burn rate of 7.7, and at that rate the failure share runs out in 3.92 days. A 5 percent threshold is a rate of 76.6 and 0.39 days — waiting for this threshold lets the month's budget be exhausted in nine hours. Once the threshold is written this way, the disagreement stops being one of preference. The third table gives the cost of missing, line by line. The six events ate 92.4 percent of the failure share, leaving 2.14 minutes. The most expensive event is not the most severe one but the one whose severity times duration is largest: the 40 percent event lasting forty-five minutes alone took 63.8 percent of the share. By contrast, the 0.8 percent-severity event — the one missed by every threshold of 1 percent and above in the scan — ate 6.8 percent of the share. **An event not producing an alert does not mean it was cheap;** a mild, long degradation can eat more budget than a short, sharp event. This is why a budget-based alert setup wants not a single threshold but at least two: one with a short window and a high rate, another with a long window and a low rate. #### Summary - A dashboard answers "is this normal" for someone looking; an alert answers "should I do something now" for someone not looking; a panel's expected range and an alert threshold are not the same number. - Every alert is tied to an action; if there is no work to do in response, the metric goes into the dashboard. - Raising the threshold changes both errors at once: at 0.5 percent and a one-minute window, 149 of 155 monthly alerts are false; at 5 percent, false alerts drop to 0, but three events are missed and 2.46 minutes leave the budget. - Widening the window cuts only short-lived noise: at 0.5 percent and a three-minute window, false alerts drop from 149 to 0, all six of six events are caught, and the cost is a 2.0-minute detection delay. If the window passes the shortest real event (5 minutes), two events are missed and 3.29 minutes leave the budget. - The threshold is derived from the budget: a 28.2-minute failure share equals a continuous 0.0653 percent error rate, and the observed rate divided by that rate is the burn rate; a 0.5 percent threshold means a rate of 7.7, a share that runs out in 3.92 days. - A missed event's cost is duration times severity: the 45-minute, 40 percent event eats 63.8 percent of the share, the 240-minute, 0.8 percent event eats 6.8 percent. #### Next Step Everything measured so far was a number that arose on its own, under the system's real traffic. These numbers only describe **what has happened**: they say what happened under today's load, not what will happen under tomorrow's. The alert threshold is tuned to today's traffic; which metric will cross its threshold first when traffic doubles, at what point the system will saturate, and whether it can recover after a sudden spike cannot be read from the collected data, because that condition has never occurred. The next lesson fills this gap: instead of waiting for load, it generates it. Three separate tests ask three separate questions, and each one surfaces a different symptom. ### Load Testing Source: https://academia.sh/en/courses/performance-and-monitoring/load-testing Producing load instead of waiting for it: measuring that load, soak, and spike tests ask three separate questions and surface three separate symptoms, writing an open-loop load generator, showing that a long run at the same load reveals in tail latency a degradation invisible in the median, and counting that a spike's cost is larger than the spike itself. The previous three lessons collected the numbers the system produces on its own, formatted them, and tied them to a threshold. Every collected number describes **what has already happened**. It says what happened at today's load; it does not say what will happen at tomorrow's load, which metric will cross its threshold first when traffic triples, or whether today's number will stay the same after the system runs at the same load for hours. These three questions have not been measured, because they have never happened. This lesson produces load instead of waiting for it. Three separate tests are set up, and the lesson's claim is this: **the three are not the long and short forms of the same test — they ask three separate questions and surface three separate symptoms.** #### Three Questions **The load test** asks whether thresholds are met at the targeted load. Its duration is short, its load is fixed, and its answer is binary: met or not met. **The soak test** applies the same load for a long duration and asks whether anything degrades over time. Keeping the load fixed matters specifically: the only thing that changes is duration, so any degradation found is the result not of the load but of something **accumulating** in the system — memory held and never released, a growing array, a ledger never cleared. **The spike test** multiplies the load for a short time and asks two things: what happens during the spike, and how long the system takes to recover after the spike ends. The second question matters more than the first, because a spike is transient, but requests that have nothing to do with the spike also pay a cost throughout the recovery period. The rig below writes its own load generator. The generator is **open-loop**: it sends requests at the target rate and does not wait for the server to keep up. A closed-loop generator — one that waits for each response before sending the next — reduces its load on its own once the server slows down, and can never produce a spike at all. On the server side, the tracking endpoint includes a **maintenance pass** that scans the accumulated records on every hundredth request; this pass is the source of the accumulation the soak test will look for. The measured durations are tied to this machine; the lesson's claims rest on ratios and counters. ```js // load/server.mjs — the tracking endpoint: every request spends a wait and a processor share, // every hundredth request runs a maintenance pass that scans the accumulated records. The // /metrics endpoint returns the server's counters. import { createServer } from "node:http"; const WAIT = 3; // rig parameter: wait standing in for the record lookup (ms) const CPU = 0.85; // rig parameter: processor share per request (ms) const MAINT_INTERVAL = 100; // rig parameter: how many requests between maintenance passes const MAINT_FACTOR = 600; // rig parameter: how many times a maintenance pass scans the accumulated record const records = []; let requests = 0, maint = 0, maintSteps = 0; function work(ms) { const end = performance.now() + ms; let t = 0; while (performance.now() < end) t += 1; return t; } const port = Number(process.argv[2]); if (Number.isInteger(port) === false) console.log("usage: node load/server.mjs "); else createServer(async (req, res) => { const url = new URL(req.url, "http://local"); if (url.pathname === "/metrics") { res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ requests, records: records.length, maint, maintSteps, factor: MAINT_FACTOR })); return; } requests += 1; const seq = requests; // its sequence is fixed on entry, not on exit records.push({ no: url.searchParams.get("no") ?? "-", step: seq % 7 }); await new Promise((c) => setTimeout(c, WAIT)); work(CPU); if (seq % MAINT_INTERVAL === 0) { // maintenance pass: the work grows with the accumulated records maint += 1; let t = 0; for (let k = 0; k < MAINT_FACTOR; k += 1) for (const record of records) t += record.step; maintSteps += MAINT_FACTOR * records.length; } res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify({ state: "at transfer hub", step: seq % 7 })); }).listen(port); ``` ```js // load/generator.mjs — open-loop load generator: sends requests at the target rate, does not // wait for the server to keep up. All three tests run with the same generator, each against a // freshly started server. import { Agent, request } from "node:http"; const port = Number(process.argv[2]); const test = process.argv[3]; const BASE = 514; // target rate (requests/s) — K01: peak edge 513.89 requests/s const SPIKE = 3; // K01: peak factor 3 const THRESHOLD = 25; // YT1 (assumption): tracking response's performance threshold (ms) const agent = new Agent({ keepAlive: true, maxSockets: 8192 }); const pct = (d, p) => { const s = [...d].sort((a, b) => a - b); return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; }; const fix2 = (x) => x.toFixed(2); const pad = (x, n) => String(x).padStart(n); const slice = (k, a, b) => k.filter((r) => r.sn >= a && r.sn < b).map((r) => r.ms); function oneRequest(no, records, sn) { const start = performance.now(); return new Promise((resolve) => { const r = request({ port, path: `/tracking?no=${no}`, agent }, (y) => { y.resume(); y.on("end", () => { records.push({ ms: performance.now() - start, sn }); resolve(); }); }); r.on("error", () => { records.push({ ms: performance.now() - start, sn, error: 1 }); resolve(); }); r.end(); }); } const metrics = () => new Promise((resolve) => { request({ port, path: "/metrics", agent }, (y) => { let body = ""; y.on("data", (p) => { body += p; }); y.on("end", () => resolve(JSON.parse(body))); }).end(); }); async function run(duration, rate) { // rate(t): target rate as a function of elapsed seconds const records = [], inFlight = []; let debt = 0, i = 0, lastT = 0; const start = performance.now(); while (performance.now() - start < duration * 1000) { const t = (performance.now() - start) / 1000; debt += rate(t) * (t - lastT); // based on real elapsed time: absorbs timer drift lastT = t; while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(`G${i++}`, records, Math.floor(t))); } await new Promise((c) => setTimeout(c, 5)); } await Promise.all(inFlight); return { records, elapsed: (performance.now() - start) / 1000 }; } if (Number.isInteger(port) === false || ["load", "soak", "spike"].includes(test) === false) { console.log("usage: node load/generator.mjs load|soak|spike"); } else if (test === "load") { await oneRequest("warmup", [], 0); const r = await run(3, () => BASE); const m = r.records.map((k) => k.ms); const over = m.filter((x) => x > THRESHOLD).length; console.log(`LOAD TEST: target ${BASE} requests/s, 3 s, threshold ${THRESHOLD} ms`); console.log(` completed ${m.length}, reached rate ${fix2(m.length / r.elapsed)} requests/s ` + `(${fix2((100 * m.length) / (BASE * r.elapsed))}% of target)`); console.log(` median ${fix2(pct(m, 50))} ms, p95 ${fix2(pct(m, 95))} ms, p99 ${fix2(pct(m, 99))} ms`); console.log(` over threshold ${over} requests (${fix2((100 * over) / m.length)}%)`); } else if (test === "soak") { await oneRequest("warmup", [], 0); const o1 = await metrics(); const r = await run(12, () => BASE); const o2 = await metrics(); console.log(`SOAK TEST: same ${BASE} requests/s, 12 s`); console.log(` quarter requests median(ms) p95(ms) p99(ms) over threshold`); const c = []; for (let i = 0; i < 4; i += 1) { const d = slice(r.records, i * 3, (i + 1) * 3); c.push(d); console.log(` ${pad(i + 1, 7)}${pad(d.length, 11)}${pad(fix2(pct(d, 50)), 13)}` + `${pad(fix2(pct(d, 95)), 10)}${pad(fix2(pct(d, 99)), 10)}${pad(d.filter((x) => x > THRESHOLD).length, 17)}`); } console.log(` last/first quarter: median ${fix2(pct(c[3], 50) / pct(c[0], 50))}x, ` + `p95 ${fix2(pct(c[3], 95) / pct(c[0], 95))}x, p99 ${fix2(pct(c[3], 99) / pct(c[0], 99))}x`); const passes = o2.maint - o1.maint; console.log(` server counters: accumulated records ${o1.records} -> ${o2.records}, maintenance passes ${passes}`); console.log(` steps per maintenance pass: first ${(o1.factor * 100).toLocaleString("en-US")}, ` + `last ${(o2.factor * 100 * passes).toLocaleString("en-US")} (${passes}x)`); } else { await oneRequest("warmup", [], 0); const r = await run(9, (t) => (t >= 3 && t < 4.5 ? BASE * SPIKE : BASE)); console.log(`SPIKE TEST: baseline ${BASE} requests/s, ${BASE * SPIKE} requests/s between 3.0-4.5 s, 9 s`); console.log(` second completed median(ms) p95(ms) over threshold`); const baseline = []; for (let sn = 0; sn < 9; sn += 1) { const d = slice(r.records, sn, sn + 1); if (sn < 3) baseline.push(pct(d, 95)); console.log(` ${pad(sn, 6)}${pad(d.length, 12)}${pad(fix2(pct(d, 50)), 13)}` + `${pad(fix2(pct(d, 95)), 10)}${pad(d.filter((x) => x > THRESHOLD).length, 17)}`); } let recovery = null; for (let sn = 5; sn < 9; sn += 1) if (recovery === null && pct(slice(r.records, sn, sn + 1), 95) <= THRESHOLD) recovery = sn - 4.5; const extra = Math.round(BASE * (SPIKE - 1) * 1.5); const over = r.records.filter((k) => k.ms > THRESHOLD).length; console.log(` baseline p95 ${fix2(pct(baseline, 100))} ms; extra requests sent during spike ${extra}, ` + `ended in error ${r.records.filter((k) => k.error).length}`); console.log(` p95 return below threshold: ${recovery === null ? "did not return" : fix2(recovery) + " s after spike ended"}`); console.log(` total requests over threshold ${over} = ${fix2(over / extra)}x the extra requests`); } agent.destroy(); ``` ```bash # measure.sh — three tests, each run against a freshly started server for T in load soak spike; do node load/server.mjs 8851 & SERVER=$! sleep 1 node load/generator.mjs 8851 $T kill $SERVER wait $SERVER 2>/dev/null echo done ``` ``` LOAD TEST: target 514 requests/s, 3 s, threshold 25 ms completed 1539, reached rate 512.72 requests/s (99.75% of target) median 7.24 ms, p95 9.92 ms, p99 13.05 ms over threshold 0 requests (0.00%) SOAK TEST: same 514 requests/s, 12 s quarter requests median(ms) p95(ms) p99(ms) over threshold 1 1538 7.06 9.04 9.65 0 2 1544 6.97 9.00 10.12 0 3 1542 7.30 15.95 22.37 4 4 1541 7.33 20.04 25.99 22 last/first quarter: median 1.04x, p95 2.22x, p99 2.69x server counters: accumulated records 1 -> 6166, maintenance passes 61 steps per maintenance pass: first 60,000, last 3,660,000 (61x) SPIKE TEST: baseline 514 requests/s, 1542 requests/s between 3.0-4.5 s, 9 s second completed median(ms) p95(ms) over threshold 0 512 7.22 10.01 0 1 514 7.35 12.71 0 2 514 7.21 11.43 0 3 1545 101.74 1508.92 1400 4 1024 111.74 1028.35 976 5 516 14.07 108.66 224 6 514 7.31 16.52 4 7 512 7.61 20.21 12 8 516 7.32 22.93 15 baseline p95 12.71 ms; extra requests sent during spike 1542, ended in error 0 p95 return below threshold: 1.50 s after spike ended total requests over threshold 2631 = 1.71x the extra requests ``` The output's **duration columns** — median, p95, p99, and the multiples derived from them — are tied to this machine and its load at that moment; they come out differently on another machine, and differently again if repeated on the same machine. The **counter columns**, by contrast, do not depend on the machine's speed: the number of requests sent, the accumulated records, the number of maintenance passes, and the steps per maintenance pass are all determined by the product of the target rate and the duration. The claims in the next three sections rest on this second set and on the **direction** of the duration numbers; values read from a single run are marked *in this run*. #### The Load Test Passed The first test reached its target: the generator stayed within one percent of the requested rate (in this run, 512.72 requests/s, 99.75% of target). Reporting the reached rate looks unnecessary, and it is not — if the generator had not reached its target, the measured latency would be the latency of a different load, and the table could not be read. The percentiles stayed clearly below the threshold, and in this run no request exceeded the 25-millisecond threshold. The answer is binary, and it is positive. This test's guarantee, on its own, is narrow. What it says is this: a system in this state meets the threshold at this load, for this duration. Two more tests remove two of those three conditions. #### Soak Test: The Median Lies The second test applied the same load for four times the duration, and the table splits in two down the middle. **The median stays put across all four quarters** (in this run, the last quarter is 1.04x the first). Looking only at that number, someone would say the system has not changed. Yet over the same quarters p95 and p99 have grown by close to two times (2.22x and 2.69x in this run), and threshold-exceeding requests go from zero in the first quarter to dozens in the last (22 in this run). **At the same load, in the same code, a threshold violation was born purely because the duration got longer.** The multiples' exact values move from run to run; their direction does not. And the number carrying the claim is not those multiples anyway — it is the server's own counters. The cause of the degradation is not a guess; it is written down: accumulated records climbed to 6,166, sixty-one maintenance passes ran, and a maintenance pass's step count went from 60,000 to 3,660,000 — exactly **61x**. This ratio is independent of the run: no matter how fast the machine is, the sixty-first pass does sixty-one times the work of the first, because the scanned array grows by one hundred records every hundred requests. The median's not budging comes from the same number: a maintenance pass slows down only one request out of a hundred — one percent of requests. The median is the fiftieth-percentile request, and nothing happens there. **The symptom of accumulation shows up not in the median, but in tail latency.** #### Spike Test: The Cost Is Bigger Than the Spike The third test multiplied the load by three for a second and a half — the peak factor from the Introduction to System Design course was also three. In the spike second, p95 grew by **two orders of magnitude** (from 12.71 milliseconds to 1,508.92 milliseconds in this run). After the spike ended, p95 took a second and a half to return below the threshold; the measurement has second-level resolution, and the work that had accumulated during that time was worked off. The test's real number is in the last line. During the spike window, 1,542 extra requests were sent — this number is a computed value, it follows from the generator's parameters, and it does not depend on the run. The number of threshold-exceeding requests is **larger** than that (2,631 in this run, that is, 1.71x). The ratio's exact value moves from run to run, but its being greater than one does not, because its cause is structural: the excess work entering the queue also makes the ordinary requests behind it wait, so the set of requests whose latency degrades is wider than the set of requests that spiked. A spike's cost is measured not by the number of requests that spiked, but by the **total** number of requests whose latency degraded. One column also stays empty: the number of requests that ended in error is zero. The system did not reject the excess load; it queued it. The back-pressure and load-shedding patterns built in the Resilience and Reliability course change exactly this behavior; neither is present here, so the capacity shortfall was paid for in latency. This is the symptom the spike test surfaces, and neither the load test nor the soak test can see it. #### Summary - The three tests ask three separate questions: the load test measures today's threshold, the soak test measures accumulation over time, and the spike test measures the degradation during a spike and the recovery afterward. - The load generator must be open-loop: a generator that waits for each response reduces its own load once the server slows down and can never produce a spike at all. The reached rate is reported on every run (99.75% of target in this run). - The load test met the threshold: within one percent of the target rate, p99 stayed well below the threshold and no request exceeded it (13.05 ms and 0 requests in this run). - The soak test found degradation at the same load: the median stayed put while p95 and p99 grew by close to two times, and threshold-exceeding requests went from zero to dozens (1.04x / 2.22x / 2.69x, 22 requests in this run). - The source of the degradation is in the server's counters: a maintenance pass's step count grew 61x; because the pass slows down only one request in a hundred, the symptom shows up in tail latency, not the median. - In the spike test, 1,542 extra requests caused more requests than that to exceed the threshold (2,631, 1.71x, in this run); the ratio being greater than one is structural. p95 took a second and a half to return below the threshold, and no request was rejected — the excess was paid for in latency. #### Next Step The three tests showed how far a single process can carry a system's load today, but none of them asked "how many are needed." The number the load test gives is not a limit but a confirmation: 514 requests/s is met. The limit itself was measured in the Introduction to System Design course — a process's saturation throughput — and that course also showed that latency does not grow linearly as utilization rises. The next lesson takes these two inputs and turns them into a forecast: it derives how many nodes the measured utilization requires, counts the capacity that remains after a node is lost, and computes how the accumulation the soak test showed changes that node count, and how often a growing demand raises it. The one decision in between is a ratio, and it has a name: headroom. ### Capacity Planning Source: https://academia.sh/en/courses/performance-and-monitoring/capacity-planning Turning measured utilization into a resource forecast: defining headroom as a utilization ceiling, computing node count as the greater of the load constraint and the failure constraint, counting how the redundancy premium shrinks with scale, the effect on node count of accumulation lowering effective capacity, and showing why improving performance does not lower the node count. The previous lesson used three tests to measure what a single process does at today's load, and never asked one question: how many are needed. The number the load test gives is not a limit but a confirmation — 514 requests/s is met. The limit itself is not remeasured in this lesson; the Introduction to System Design course measured a process's **saturation throughput** and built the queuing model showing that latency does not grow linearly as utilization rises. Both are inputs here. This lesson turns a measurement into a forecast. The input is a rate and an upper bound; the output is one number: how many nodes. The one decision in between is a ratio. #### Headroom and Two Constraints **Headroom** is the portion of a resource deliberately left unused, and it is written as a **utilization ceiling**: 30% headroom means utilization must not pass 70%. The rationale for headroom is in K01's queuing model — as utilization rises, latency grows faster than linearly, so headroom is not waste but a latency decision. Two separate constraints determine the node count, and **the larger one wins**. **The load constraint**: the number of nodes needed for the peak rate to be met below the utilization ceiling. **The failure constraint**: the number needed for the remaining nodes to stay below that same ceiling once one node drops — which means the remaining nodes must meet the load constraint, so there is always one extra node. The calculation below has these inputs: peak rate 513.89 requests/s (K01, computed); saturation throughput per node 1,176.47 requests/s — since the previous lesson's server spends 0.85 ms of processor time per request, this follows from that arithmetic on a single thread, it is not a measurement; K01's measured 4,268 requests/s at the bare edge is used in the sensitivity line. Monthly demand growth (IZ8) and accumulation's effect on effective capacity (IZ9) are assumptions. ```js // capacity/nodes.mjs — from measured saturation throughput to node count. Headroom is a // utilization ceiling; node count is the greater of the load constraint and the failure // constraint. All of it is arithmetic. const PEAK = 513.89; // K01 (computed): peak edge requests/s const BATCH = 833.33; // K01 (computed): end-of-day job's scan rate (records/s) const RECORD_MS = 1.20; // K01 (computed): batch job time per record, single worker const MU = 1 / 0.00085; // computed: previous lesson's server spends 0.85 ms processor per request const MU_BARE = 4268; // K01 (measurement): bare tracking endpoint's saturation throughput, on that machine const DEGRADED = 0.7; // IZ9 (assumption): accumulation drops a node's effective capacity by 30% const GROWTH = 0.06; // IZ8 (assumption): monthly demand growth const b = (x, n = 2) => x.toFixed(n); const loadConstraint = (peak, mu, ceiling) => Math.ceil(peak / (mu * ceiling)); const failureConstraint = (peak, mu, ceiling) => loadConstraint(peak, mu, ceiling) + 1; const latencyMs = (mu, lambda) => 1000 / (mu - lambda); // K01's queuing MODEL, as an input console.log(`saturation throughput per node ${b(MU)} requests/s (computed)`); console.log(`${"peak requests/s".padStart(16)}${"utilization ceiling".padStart(21)}${"load constraint".padStart(17)}` + `${"failure constraint".padStart(20)}${"chosen n".padStart(10)}${"utilization".padStart(13)}` + `${"utilization at loss".padStart(21)}${"latency at loss".padStart(17)}`); for (const peak of [PEAK, PEAK * 3]) for (const ceiling of [0.5, 0.6, 0.7, 0.8, 0.9]) { const n = Math.max(loadConstraint(peak, MU, ceiling), failureConstraint(peak, MU, ceiling)); const atLoss = peak / (n - 1); console.log(`${b(peak).padStart(16)}${(b(ceiling * 100, 0) + "%").padStart(21)}` + `${String(loadConstraint(peak, MU, ceiling)).padStart(17)}${String(failureConstraint(peak, MU, ceiling)).padStart(20)}` + `${String(n).padStart(10)}${(b((peak / (n * MU)) * 100, 1) + "%").padStart(13)}` + `${(b((atLoss / MU) * 100, 1) + "%").padStart(21)}${(b(latencyMs(MU, atLoss)) + " ms").padStart(17)}`); } console.log(`\nredundancy premium shrinks with scale (utilization ceiling 70%):`); console.log(`${"scale".padStart(6)}${"peak requests/s".padStart(17)}${"load constraint".padStart(17)}` + `${"chosen n".padStart(10)}${"redundancy premium".padStart(20)}${"monthly node-hours".padStart(20)}`); for (const k of [1, 2, 3, 4, 6, 10]) { const peak = PEAK * k; const y = loadConstraint(peak, MU, 0.7); const n = Math.max(y, failureConstraint(peak, MU, 0.7)); console.log(`${(k + "x").padStart(6)}${b(peak).padStart(17)}${String(y).padStart(17)}` + `${String(n).padStart(10)}${(b((100 * (n - y)) / y, 1) + "%").padStart(20)}` + `${(n * 720).toLocaleString("en-US").padStart(20)}`); } console.log(`\ndemand growth (monthly ${b(GROWTH * 100, 0)}%, utilization ceiling 70%):`); console.log(`${"month".padStart(6)}${"peak requests/s".padStart(17)}${"chosen n".padStart(10)}` + `${"utilization".padStart(13)}${"n with degraded node".padStart(23)}`); for (const month of [0, 6, 12, 18, 24]) { const peak = PEAK * (1 + GROWTH) ** month; const n = Math.max(loadConstraint(peak, MU, 0.7), failureConstraint(peak, MU, 0.7)); const nDeg = Math.max(loadConstraint(peak, MU * DEGRADED, 0.7), failureConstraint(peak, MU * DEGRADED, 0.7)); console.log(`${String(month).padStart(6)}${b(peak).padStart(17)}${String(n).padStart(10)}` + `${(b((peak / (n * MU)) * 100, 1) + "%").padStart(13)}${String(nDeg).padStart(23)}`); } console.log(`\nsensitivity: if saturation throughput per node were ${MU_BARE} requests/s (K01's bare edge)`); console.log(` at today's peak n = ${Math.max(loadConstraint(PEAK, MU_BARE, 0.7), failureConstraint(PEAK, MU_BARE, 0.7))}` + `, at ${b(MU)} requests/s n = ${Math.max(loadConstraint(PEAK, MU, 0.7), failureConstraint(PEAK, MU, 0.7))}` + ` (${b(MU_BARE / MU)}x the throughput, same node count)`); const worker = Math.max(loadConstraint(BATCH, 1000 / RECORD_MS, 0.7), 1); console.log(`the batch job is a separate capacity question: ${BATCH} records/s needed, one worker does ` + `${b(1000 / RECORD_MS)} records/s -> at a 70% ceiling ${worker} workers, the window narrows to ` + `${b((BATCH * 4) / (worker * (1000 / RECORD_MS)), 2)} hours`); ``` ``` saturation throughput per node 1176.47 requests/s (computed) peak requests/s utilization ceiling load constraint failure constraint chosen n utilization utilization at loss latency at loss 513.89 50% 1 2 2 21.8% 43.7% 1.51 ms 513.89 60% 1 2 2 21.8% 43.7% 1.51 ms 513.89 70% 1 2 2 21.8% 43.7% 1.51 ms 513.89 80% 1 2 2 21.8% 43.7% 1.51 ms 513.89 90% 1 2 2 21.8% 43.7% 1.51 ms 1541.67 50% 3 4 4 32.8% 43.7% 1.51 ms 1541.67 60% 3 4 4 32.8% 43.7% 1.51 ms 1541.67 70% 2 3 3 43.7% 65.5% 2.47 ms 1541.67 80% 2 3 3 43.7% 65.5% 2.47 ms 1541.67 90% 2 3 3 43.7% 65.5% 2.47 ms redundancy premium shrinks with scale (utilization ceiling 70%): scale peak requests/s load constraint chosen n redundancy premium monthly node-hours 1x 513.89 1 2 100.0% 1,440 2x 1027.78 2 3 50.0% 2,160 3x 1541.67 2 3 50.0% 2,160 4x 2055.56 3 4 33.3% 2,880 6x 3083.34 4 5 25.0% 3,600 10x 5138.90 7 8 14.3% 5,760 demand growth (monthly 6%, utilization ceiling 70%): month peak requests/s chosen n utilization n with degraded node 0 513.89 2 21.8% 2 6 728.96 2 31.0% 3 12 1034.05 3 29.3% 3 18 1466.82 3 41.6% 4 24 2080.71 4 44.2% 5 sensitivity: if saturation throughput per node were 4268 requests/s (K01's bare edge) at today's peak n = 2, at 1176.47 requests/s n = 2 (3.63x the throughput, same node count) the batch job is a separate capacity question: 833.33 records/s needed, one worker does 833.33 records/s -> at a 70% ceiling 2 workers, the window narrows to 2.00 hours ``` #### Headroom Has No Effect at Today's Load The first table's first five rows give an unexpected result. Even as the utilization ceiling is raised from 50% to 90%, the node count stays at 2, utilization stays at 21.8%, and utilization at the loss of one node stays at 43.7%. **At today's peak rate, the choice of headroom changes nothing**, because the load constraint gives 1 node at every ceiling, and the constraint that decides is the failure constraint. This is the most often overlooked side of capacity planning. The question "how many nodes are needed" has two independent answers; one comes from load, the other from failure behavior. A single node meets today's load with room to spare, but when that single node drops, capacity is zero, and K01's outage budget cannot exceed that one node's availability. What makes the node count two is not the load — it is this constraint. The next five rows show when headroom becomes binding. When the peak rate triples — K01's peak factor was also three — a ceiling of 70% and above needs 3 nodes, while 50% and 60% need 4. What the fourth node buys is in the last two columns: at the loss of one node, utilization is 43.7% instead of 65.5%, and the latency the queuing model gives is 1.51 ms instead of 2.47 ms. **Headroom looks like a node-count decision, and it is a failure-day latency decision.** #### Redundancy Premium, Growth, and Accumulation The second table gives the failure constraint's bill alongside scale. At today's scale, the extra node doubles the fleet: a redundancy premium of 100%. At ten times the scale, the same rule adds only 14.3%. **Redundancy is disproportionately expensive in small systems**, and this is one of scale's few defensible advantages. The right column writes the resource on a monthly basis: 1,440 node-hours today, 5,760 node-hours at ten times the scale. This course uses a resource unit in place of a currency. The third table shows that capacity is not a number but a **time series**. At 6% monthly growth, the node count climbs to 3 by the twelfth month and to 4 by the twenty-fourth; utilization drops with each addition and climbs back up. The last column accounts for the previous lesson's soak-test finding: if a node's effective capacity drops 30% because of accumulation (IZ9), the third node is needed right away instead of at month six, and the count reaches five nodes by month twenty-four. **A soak test's finding, if it had not been run, gets written into the capacity plan as an extra node.** #### Performance Does Not Lower the Node Count The sensitivity line is the lesson's harshest result. If saturation throughput per node were 4,268 requests/s — K01's measurement at the bare edge — instead of 1,176.47, that is, **an edge 3.63x as fast**, the required node count would still be 2. At today's scale, a performance improvement's resource payoff is zero, because the deciding constraint is failure, not performance. Translating a performance effort into capacity only makes sense once the load constraint is binding; in this table, that only happens after the fourth row. The last line is a reminder that capacity is asked per flow. End-of-day billing needs 833.33 records per second, and a single worker's rate is exactly 833.33 records/s — that is, zero headroom. At a 70% ceiling, two workers are needed, and in exchange, the four-hour window narrows to two hours. The tracking query's node count never affects this calculation: the two flows are part of the same system, and each asks its own capacity question. #### Summary - Headroom is a utilization ceiling; its rationale is K01's queuing model, and it is not waste but a latency decision. - Node count is the greater of the load constraint and the failure constraint; the failure constraint always demands one extra node, because the remaining nodes must also meet the same ceiling. - At today's peak of 513.89 requests/s, the choice of headroom has no effect at all: every ceiling gives 2 nodes, utilization 21.8%. When the peak triples, a 50% ceiling gives 4 nodes and a 70% ceiling gives 3, and the difference is a failure-day latency (1.51 ms instead of 2.47 ms). - The redundancy premium shrinks with scale: at today's scale, the extra node grows the fleet by 100%; at ten times the scale, by 14.3%; the resource goes from 1,440 node-hours a month to 5,760. - Capacity is a time series: at 6% monthly growth, the node count reaches 3 by month 12 and 4 by month 24; if accumulation drops effective capacity by 30%, that same calendar becomes 3 and 5. - Improving performance 3.63x does not change the node count at today's scale; translating performance into capacity only makes sense once the load constraint is binding. The batch job is a separate question: at a 70% ceiling, two workers narrow the window from four hours to two. #### Next Step This lesson produced one number: how many nodes. Alongside that number is a resource unit — 1,440 node-hours a month today, 5,760 at ten times the scale. But **what those nodes cost** was never asked. A node-hour is not a single line item; a request carries data over the network, a record gets stored, a metric gets produced and stored, a cache sits in memory. More importantly, every design decision made across all the courses up to this point moved one of these line items, and none of them was counted as operating cost: what does raising the cache hit ratio reduce, what does partitioning the data multiply, what line item does adding a replica multiply and by how much, in which line item does monitoring's own cost show up. The next lesson takes up this question and answers it in resource units. ### Cost Awareness Source: https://academia.sh/en/courses/performance-and-monitoring/cost-awareness Counting design decisions' operating cost in resource units: converting monthly resource consumption into a unit cost by dividing it by work volume, separating volume growth from unit-cost growth — which give the same total — using resource per unit of work, measuring how much of the purchased capacity does work, and showing that no fix removes cost, only shifts it between line items. The previous lesson produced one number: how many nodes. Alongside it, it wrote a resource unit too — 1,440 node-hours a month today, 5,760 at ten times the scale. But what those nodes cost, and how every course's design decisions up to this point changed operating cost, was never asked. What does raising the cache hit ratio reduce, what does splitting the data multiply, which line item does adding a replica grow, where does monitoring's own cost show up. This lesson takes up that question and applies the course's diagnostic rule one last time: a symptom is a number, two causes are set up that give the same symptom, and the measurement that separates them is shown. #### Cost Is Written in Resource Units In this course, operating cost is **not written in currency**. A price depends on a provider, a contract, and a date; read five years later, it is wrong. A resource unit does not depend on those: **processor-seconds**, **GB-months**, **node-hours**, **GB transferred**, touches per request. When a design decision's operating cost is written in these units, the comparison stays valid even if the price changes. **Unit cost** in this course is denominated in resources: the resource one unit of work — one request or one record — consumes. All the inputs to the calculation below are the previous lessons' numbers, and none of them is recomputed; no new assumption is added either. ```js // cost/unit.mjs — design decisions' operating cost, in resource units. All inputs are the // previous lessons' numbers and none of them are recomputed. No currency is used. const QUERY_DAY = 12_000_000; // K01: daily tracking queries const EVENT_DAY = 2_800_000; // K01: daily state events const RESPONSE_BYTES = 480; // K01 (V5): tracking response body const EVENT_BYTES = 220; // K01 (V6): one state event record const BATCH_GB = 10.80; // K01: what the end-of-day job reads const STORED_GB = 712.48; // K01: data stored over 730 days const CPU_S = 0.00085; // this course's load testing lesson: processor share per request (s) const PEAK = 513.89; // K01: peak edge requests/s const MONITOR_RAW = 239.76; // this course's instrumentation lesson: raw event, GB-months const MONITOR_HIST = 2.07; // same lesson: per-second histogram, GB-months const DAYS = 30, HOURS = 720; const b = (x, n = 2) => x.toFixed(n); const nodes = (peak) => Math.ceil(peak / ((1 / CPU_S) * 0.7)) + 1; // 05's rule const requestsMonth = (QUERY_DAY + EVENT_DAY) * DAYS; const lineItems = (k = 1) => ({ "processor-seconds": requestsMonth * k * CPU_S, "node-hours": nodes(PEAK * k) * HOURS, "GB transferred": ((QUERY_DAY * RESPONSE_BYTES + EVENT_DAY * EVENT_BYTES) * DAYS * k) / 1e9 + BATCH_GB * DAYS * k, "storage GB-months": STORED_GB * k, "monitoring GB-months": MONITOR_HIST, }); const baseline = lineItems(); console.log(`monthly work volume ${requestsMonth.toLocaleString("en-US")} requests (K01: daily 12,000,000 queries + 2,800,000 events)`); console.log(`${"line item".padEnd(22)}${"monthly amount".padStart(15)}${"per request".padStart(16)}`); for (const [name, value] of Object.entries(baseline)) console.log(`${name.padEnd(22)}${b(value).padStart(15)}${value / requestsMonth < 1e-4 ? (value / requestsMonth).toExponential(2).padStart(16) : b(value / requestsMonth, 6).padStart(16)}`); const purchased = nodes(PEAK) * HOURS * 3600; console.log(`\npurchased node-seconds ${purchased.toLocaleString("en-US")}, ` + `working processor-seconds ${Math.round(baseline["processor-seconds"]).toLocaleString("en-US")}`); console.log(`resource efficiency ${b((100 * baseline["processor-seconds"]) / purchased)}% — the rest is headroom, ` + `redundancy, and off-peak hours`); console.log(`\nas scale grows, which line item grows by how much:`); console.log(`${"scale".padStart(6)}${"nodes".padStart(7)}${Object.keys(baseline).map((a) => a.padStart(22)).join("")}`); for (const k of [1, 2, 4, 10]) { const y = lineItems(k); console.log(`${(k + "x").padStart(6)}${String(nodes(PEAK * k)).padStart(7)}` + `${Object.keys(baseline).map((a) => `${b(y[a] / baseline[a])}x`.padStart(22)).join("")}`); } console.log(`two line items that do not grow with volume lower the unit cost: at 10x volume, node-hours grow ` + `${b(nodes(PEAK * 10) / nodes(PEAK))}x, monitoring 1.00x`); console.log(`\nsame total growth, two separate causes (the measurement that separates them: processor-seconds per request):`); console.log(`${"month".padStart(5)}${"A: requests".padStart(16)}${"A: processor-s".padStart(16)}` + `${"A: per request".padStart(17)}${"B: requests".padStart(16)}${"B: processor-s".padStart(16)}${"B: per request".padStart(17)}`); for (const month of [0, 6, 12]) { const growth = 1.06 ** month; // A: volume grows, unit cost fixed const aRequests = requestsMonth * growth, aCpu = aRequests * CPU_S; const bRequests = requestsMonth, bCpu = bRequests * CPU_S * growth; // B: volume fixed, unit cost grows console.log(`${String(month).padStart(5)}${Math.round(aRequests).toLocaleString("en-US").padStart(16)}` + `${Math.round(aCpu).toLocaleString("en-US").padStart(16)}${(aCpu / aRequests).toExponential(3).padStart(17)}` + `${Math.round(bRequests).toLocaleString("en-US").padStart(16)}` + `${Math.round(bCpu).toLocaleString("en-US").padStart(16)}${(bCpu / bRequests).toExponential(3).padStart(17)}`); } console.log(`\nresource-profile shift — no decision removes cost, each one shifts it between line items:`); const DECISIONS = [ ["edge cache (K01 V9: hit ratio 0.90)", `reads reaching the store shrink ${b(416.67 / 41.67)}x`, "held memory and the staleness window grow (quantified in K04)"], ["splitting off monolithic persistence", "period scan reads 801x fewer records", `store writes ${b(194.44 / 97.22)}x, stores in operation 1 -> 3`], ["one more node (redundancy)", "a single node's failure no longer causes an outage", `node-hours ${b(1440 / 720)}x (720 -> 1,440)`], ["per-second histogram instead of raw event", `monitoring holds ${b(MONITOR_RAW / MONITOR_HIST)}x fewer GB-months`, "record identity is lost: p99 error rises to 10.0%"], ]; for (const [name, shrinks, grows] of DECISIONS) console.log(` ${name}\n shrinks: ${shrinks}\n grows : ${grows}`); ``` ``` monthly work volume 444,000,000 requests (K01: daily 12,000,000 queries + 2,800,000 events) line item monthly amount per request processor-seconds 377400.00 0.000850 node-hours 1440.00 3.24e-6 GB transferred 515.28 1.16e-6 storage GB-months 712.48 1.60e-6 monitoring GB-months 2.07 4.66e-9 purchased node-seconds 5,184,000, working processor-seconds 377,400 resource efficiency 7.28% — the rest is headroom, redundancy, and off-peak hours as scale grows, which line item grows by how much: scale nodes processor-seconds node-hours GB transferred storage GB-months monitoring GB-months 1x 2 1.00x 1.00x 1.00x 1.00x 1.00x 2x 3 2.00x 1.50x 2.00x 2.00x 1.00x 4x 4 4.00x 2.00x 4.00x 4.00x 1.00x 10x 8 10.00x 4.00x 10.00x 10.00x 1.00x two line items that do not grow with volume lower the unit cost: at 10x volume, node-hours grow 4.00x, monitoring 1.00x same total growth, two separate causes (the measurement that separates them: processor-seconds per request): month A: requests A: processor-s A: per request B: requests B: processor-s B: per request 0 444,000,000 377,400 8.500e-4 444,000,000 377,400 8.500e-4 6 629,822,486 535,349 8.500e-4 444,000,000 535,349 1.206e-3 12 893,415,233 759,403 8.500e-4 444,000,000 759,403 1.710e-3 resource-profile shift — no decision removes cost, each one shifts it between line items: edge cache (K01 V9: hit ratio 0.90) shrinks: reads reaching the store shrink 10.00x grows : held memory and the staleness window grow (quantified in K04) splitting off monolithic persistence shrinks: period scan reads 801x fewer records grows : store writes 2.00x, stores in operation 1 -> 3 one more node (redundancy) shrinks: a single node's failure no longer causes an outage grows : node-hours 2.00x (720 -> 1,440) per-second histogram instead of raw event shrinks: monitoring holds 115.83x fewer GB-months grows : record identity is lost: p99 error rises to 10.0% ``` #### Purchased Resource and Working Resource The second block gives the lesson's most uncomfortable number. 5,184,000 node-seconds are purchased per month, and 377,400 of them do work: **resource efficiency is 7.28%.** The rest has not gone to waste; it was deliberately purchased by three separate decisions. Headroom is held so utilization does not pass the ceiling, and its rationale is the queuing model. Redundancy is held so service continues when a node drops. The third is arithmetic: capacity is bought for the peak, while work is done on average, and K01's peak factor is three. Read as a waste measure, this number is misread. The correct reading is this: **this is the resource-denominated price of the availability target and the latency target.** The only way to raise the 7.28% is to give up one of those two targets; as long as the targets stand, the ratio is a consequence, not a flaw. Cost awareness is not trying to lower this ratio — it is knowing what was bought. #### Symptom: Resource Grows Faster Than Volume Growth in monthly resource consumption is a symptom, and on its own it says nothing. Two causes give the same total. **A:** volume has grown, unit cost is fixed. **B:** volume is fixed, unit cost has grown. The third table places the two side by side, and the total columns are identical at month twelve: 759,403 processor-seconds. The measurement that separates them is **resource per unit of work**. In A, processor-seconds per request stays fixed at 8.500e-4 across the twelve months; in B, it goes from 8.500e-4 to 1.710e-3 — doubling. Someone looking only at total resource cannot separate the two cases, and will most likely give both the same answer: add a node. In A, that is the right answer. In B, adding a node covers up the symptom and leaves the cause in place. This course measured a real source of B. The soak test showed a process running at the same load having its steps per maintenance pass grow 61x; accumulation grows the work per request independent of volume. **Unit cost's time series finds, in production, what a soak test finds** — and without running a test at all. #### Unit Cost Falls With Scale The second table shows the line items do not grow at the same rate as scale. Processor-seconds, GB transferred, and storage are linear with volume: ten times the work, ten times the resource. Node-hours are not — at ten times the volume, they grow only 4.00x, because the previous lesson's failure constraint demands one extra node at small scale, and that extra node's share erodes with scale. The monitoring line item does not grow at all: the per-second histogram holds 2.07 GB-months regardless of request rate. The conclusion is this: **unit cost is not a number independent of scale.** Node-hours per request get 2.5 times cheaper at ten times the scale, monitoring per request gets ten times cheaper, processor-seconds per request never change. In an operating-cost discussion, the sentence "this feature costs this much per request" cannot be read without saying at what scale it was measured — the same rule as the course's first lesson, this time on the cost side. #### Resource-Profile Shift The last block takes four design decisions and shows that none of them **removes** cost — each only shifts it between line items. An edge cache shrinks reads reaching the store by 10.00x and holds memory in exchange. Splitting off monolithic persistence shrinks the period scan by 801x and raises store writes 2.00x, moving the number of stores in operation from 1 to 3. Adding one more node keeps a single node's failure from producing an outage and raises node-hours 2.00x. Collecting a histogram instead of the raw event makes monitoring 115.83 times cheaper and erases record identity. The shape shared by all four is the cost-side counterpart of the course's diagnostic rule: when a fix is being justified, **which line item shrinks and which one grows** are written down together. A justification that writes down only what shrank is an incomplete justification, because the resource profile has changed, and the new profile's bottleneck is somewhere else. #### Summary - Operating cost is written in resource units, not currency — processor-seconds, GB-months, node-hours, GB transferred — because price depends on the provider and the date, and a resource unit does not. - 5,184,000 node-seconds are purchased per month, and 377,400 of them do work: resource efficiency is 7.28%. The rest is the resource-denominated price of headroom, redundancy, and the peak factor, not waste. - Two causes give the same resource growth, and the total cannot separate them (both are 759,403 processor-seconds at month twelve); the measurement that separates them is resource per unit of work: fixed at 8.500e-4 in A, 1.710e-3 in B. - Unit cost changes with scale: at ten times the volume, processor-seconds grow 10.00x, node-hours 4.00x, monitoring 1.00x; cost per request cannot be read without saying at what scale it was measured. - No fix removes cost, it shifts it between line items: the histogram makes monitoring 115.83 times cheaper and erases record identity, splitting shrinks the scan 801x and raises writes 2.00x. #### Course Wrap-Up This course did one job across sixteen lessons: tying a symptom to a cause. The table below brings together each lesson's symptom, its distinguishing measurement, and what the fix grows in exchange. | Lesson | Symptom | Distinguishing measurement | What grows in exchange for the fix | |---|---|---|---| | Busy Database | App idle while the store is busy | Boundary ratio: work moves up at 1.00, down well above it | App row work 0 → 240,000 records (3333.33 per second) | | Busy Front End | Visible time grows while server metrics stay flat | Work per record: 0.95% at load, 16.1% in code | Response 3,871 → 3,896 bytes and the cache key splitting | | Chatty I/O | Every call is fast, useful payload small, open still slow | Envelope share 0.4591 / unused-payload share 0.6266 | Merging grows unused payload, splitting grows round count (6 rounds) | | Extraneous Fetching | Edge hit rate falls below 0.90 while requests hold steady | Working set's multiplier: bytes 5.19x / variety 4.00x | Narrowing diversifies: hit rate 0.8007 → 0.6315 | | Improper Instantiation | Throughput does not rise with concurrency | Creation ratio: reduce at 1.0000, increase at 0.0007 | Held object-rounds: pool 24 holds 2.26x the resource of pool 5 | | Monolithic Persistence | Per-write work grows | Share serving its own class: index 0.33, scan 801x | Store writes 97.22 → 194.44/s, store count 1 → 3, atomicity lost | | No Caching | Application layer approaches saturation | De-duplication by computation key: read 1.00 / computation 125.00 | Memory 960 → 119,365 entries and a wrongness surface (0.5000) | | Noisy Neighbor | Small tenants' wait grows 0.24 → 0.98 rounds | Request share fixed at 0.6143 while work share 0.1702 → 0.2415 | S1's own delay 0.47 → 2.96 rounds, 200 counters held | | Synchronous I/O | Throughput hits a ceiling, adding workers does not help | Dependency's own utilization: 0.9999 / 0.1819 | In-flight requests 8 → 46 (5.75x); admission control is needed | | Retry Storm | Call rate reaching the dependency grows | Distinct request identity fixed at 4.00, success ratio 0.0400 | Throttling the allowance produces dropped requests: 180 at an allowance of 1 | | Monitoring Categories | All ten diagnoses assume a metric that does not exist | Category coverage: utilization 10/10, performance 7/10 | The list of metrics to collect reaches sixteen items | | Instrumentation Design | Monitoring's own write is 5.29x the system's own write | Information preserved per unit of cost: 93.0% error against 10.0% | Aggregation erases record identity, cannot bound the maximum | | Dashboards and Alerts | An alert either fires constantly or never fires | Threshold-window scan and burn rate: 0.5% threshold, rate 7.7 | As the window widens, a 2.0 min delay; at 5 min, 3.29 min of budget | | Load Testing | Today's measurement is silent about tomorrow's load | Three separate tests: median stays put, p95 and p99 close to double | A spike's cost is bigger than the spike: 1,542 extra requests | | Capacity Planning | How many nodes are needed cannot be read from measurement | Load constraint vs. failure constraint: today, n = 2 at every ceiling | Redundancy premium 100%; 720 → 1,440 node-hours | | Cost Awareness | Monthly resource grows faster than work volume | Resource per unit of work: 8.500e-4 fixed / 1.710e-3 growing | Every fix shifts a line item: 115.83x fewer GB-months, deleted identity | The table's third column is the course's rule: **a symptom is not a diagnosis.** None of the sentences "slow," "expensive," "does not scale" points to a cause; the diagnosis is the measurement that separates the two causes giving the same symptom. In none of the sixteen lessons was the symptom sufficient on its own, and in several of them the most-watched metric turned out to be the least distinguishing one — worker utilization could not separate the two causes (0.9562 against 0.9545), total resource could not separate the two causes (both 759,403), and the response-time percentile showed up in six of the ten diagnoses at once. The fourth column carries the second rule: **no pattern is wrong under every condition.** Every lesson bounded with a number the condition under which the same shape is correct. Keeping the computation in the store is justified to the extent that it reduces the record count crossing the boundary — in F1, it cuts that count 100x. Carrying an extra field is justified if it is smaller than the 282-byte envelope of the call it eliminates. No caching is the correct design in a stream where the repeat is 1.01; there, memoization would drop to a 0.71x ratio instead of gaining anything. A shared resource is correct while the noise ratio stays close to one and no single request exceeds the per-round capacity. A blocking call is correct while the wait is the same order of magnitude as local work. An antipattern is not a prohibition — it is a choice that turns wrong **at a scale**, and the measurement in the third column is what gives that scale. Six courses built a design vocabulary piece by piece: how traffic gets served, how the application gets split, how data gets distributed, how failure gets contained, and how all of it gets made visible. Each course answered its own question with its own measurement. The one thing that was never done is this: using these pieces **together, in a single problem**, and narrating the choices with their reasoning from beginning to end. In a real design discussion, questions do not line up by course boundary; they are all asked at once, and the trade-offs among them are defended together, not one at a time. The next course, **Case Studies**, does that work.