--- title: 'Capacity and Cost Monitoring' source: 'https://academia.sh/en/courses/devops-observability/capacity-and-cost-monitoring' course: 'Observability and Operations' language: en updated: '2026-09-13T05:08:21+00:00' license: 'CC BY-SA 4.0' objectives: - 'Sets up the cost question set and counts what share of the bill each question is answered for' - 'Measures the cost of idle resources against the risk of saturation in the same table: the target utilization rate is swept' - 'Derives the unit cost alongside scale, and names the line item where economies of scale fail' --- # Capacity and Cost Monitoring Tying the operating cost to the question set: 30% of the daily 753 fictional units stays unowned, so only four of eight cost questions are fully answered; monitoring data's own share is 16%; a 90% target utilization saturates on eight of 30 days and produces 360 minutes of outage, while at 75% saturation drops to zero and the avoided minute costs 5.1 units; as scale rises eightfold, infrastructure drops from 507 to 344 units per million requests, but monitoring rises from 100 to 244 and its share climbs to 41%. The previous lesson turned the error budget into a rule: the line froze, the queue piled up, and the shape of the exit either doubled the outage or cut it in half. All of those numbers were denominated in minutes and changes. The question never asked was: what does all of this **cost** to run. The capacity decision itself was measured in M19/K06. Here capacity is treated as an **operating cost**: resource usage turns into a price, that price is applied to a question set, and the collected monitoring data's own cost is counted as a separate line item. In this table, observability is both the one measuring and the one being measured. ## The Cost Question Set The example is once more the fictional **regional measurement network**. The items, prices, and demand are fictional; **the currency is fictional too**, and it is called `u`; it corresponds to no real price list. **IM31:** There are nine cost items; each has a fixed and a variable daily charge, and a scale exponent written for it. The period is 30 days. **IM32:** Some of the items carry an **owner tag**; the shared data store and cross-region transfer are unowned and cannot be apportioned to services. **IM33:** Billing data is delayed by three days; the last three days have no detail at all. **IM34:** Daily demand is a series averaging 1.0 (the generator is our own writing, seed 20260826); capacity is derived from the target utilization rate — the capacity calculation itself was measured in M19/K06, here it is an input. **IM35:** Every day demand exceeds capacity counts as **saturation** and produces 45 minutes of outage. **IM36:** When scale grows `k`-fold, instance count grows `k`-fold and the label cardinality multiplier grows `√k`-fold; the monitoring series is the product of the two, so the scale exponent is 1.5. Items, demand, and the bill are a **model** internal to the course; there is no real bill. ```js // cost/item.mjs — the operating cost model: there is no real bill; items, prices, and demand // are a data structure. The currency is fictional and called `u`. The seed is visible. export const PERIOD = 30, SEED = 20260826, DELAY = 3, SATURATION_MIN = 45; // item: name, fixed u/day, variable u/day, scale exponent, is it owned, is it a monitoring item export const ITEM = [ ["service instances", 40, 134, 1.00, true, false], ["memory and local disk", 0, 86, 1.00, true, false], ["block storage", 12, 58, 1.00, true, false], ["nightly batch job cluster", 55, 20, 0.70, true, false], ["shared data store", 90, 60, 0.80, false, false], ["cross-region transfer", 0, 74, 1.00, false, false], ["time series store", 8, 46, 1.50, true, true], ["log store", 6, 38, 1.50, true, true], ["trace store", 4, 22, 1.50, true, true], ]; export const REQUEST = 1240000; // daily request count at scale 1 export const CAPACITY_UNIT = 278; // daily u for capacity-bound items export const unitCost = (k, x) => x[1] + x[2] * Math.pow(k, x[3]); export const daily = (k, f = () => true) => ITEM.filter(f).reduce((t, x) => t + unitCost(k, x), 0); export function generator(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // daily demand: a series averaging 1.0; the peak-to-floor ratio comes from the assumption export const demand = (() => { const z = generator(SEED); return Array.from({ length: PERIOD }, () => 0.78 + 0.44 * z()); })(); // the cost of a target utilization rate: idle capacity alongside saturation days export function utilization(target) { const avg = demand.reduce((a, b) => a + b, 0) / PERIOD; const capacity = avg / target; const saturation = demand.filter((t) => t > capacity).length; const cost = CAPACITY_UNIT * capacity * PERIOD; return { target, capacity, cost, idle: cost * (1 - target), saturation, outage: saturation * SATURATION_MIN, peak: Math.max(...demand) }; } ``` ```js // cost/measure.mjs — the cost question set, monitoring data's own share, idle resource // alongside saturation, and which line item's unit cost fails to drop as scale grows. import { ITEM, PERIOD, DELAY, SATURATION_MIN, REQUEST, unitCost, daily, utilization } from "./item.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); const G1 = daily(1), owned = daily(1, (x) => x[4]), monitoring = daily(1, (x) => x[5]); const share = (a) => `${(100 * a / G1).toFixed(0)}%`; console.log(`${PERIOD} days, ${ITEM.length} cost items, daily ${G1.toFixed(0)} u ` + `(fictional currency); billing is ${DELAY} days delayed`); // question set: [question, answer, covered cost share, reason unanswered] const QUESTION = [ ["what does the total cost", "yes", 1, "-"], ["what does each service cost", "partial", owned / G1, "unowned resource"], ["where did yesterday's spike come from", "no", 0, "delayed billing data"], ["which resource sits idle", "yes", 1, "-"], ["what does one request cost", "yes", 1, "-"], ["what does one service's request cost", "no", owned / G1, "a shared item cannot be apportioned"], ["what does monitoring data cost", "yes", 1, "-"], ["what happens to price if scale doubles", "partial", 1, "cannot be read from the bill"], ]; console.log("\n1. cost question set"); const A = [-40, 9, 11, 37]; write(A, "question", "answer", "covered", "reason unanswered"); for (const [s, y, p, n] of QUESTION) write(A, s, y, `${(100 * p).toFixed(0)}%`, n); const full = QUESTION.filter((s) => s[1] === "yes").length; console.log(` ${full}/${QUESTION.length} questions are fully answered; ` + `${QUESTION.filter((s) => s[1] === "partial").length} partial, ` + `${QUESTION.filter((s) => s[1] === "no").length} unanswered`); console.log("\n2. item breakdown and monitoring's own share"); const B = [-27, 11, 8, 10, 12]; write(B, "item", "daily u", "share", "owned", "type"); for (const x of ITEM) write(B, x[0], unitCost(1, x).toFixed(0), share(unitCost(1, x)), x[4] ? "owned" : "none", x[5] ? "monitoring" : "infra"); write(B, "total", G1.toFixed(0), "100%", share(owned), share(monitoring)); console.log(` (last row: owned share, monitoring share) monitoring ${monitoring.toFixed(0)} u/day`); console.log("\n3. target utilization rate: idle resource alongside saturation"); const C = [-9, 11, 12, 11, 12, 11, 14]; write(C, "target", "capacity", "period u", "idle u", "saturation", "outage", "idle u/min"); const K = [0.30, 0.45, 0.60, 0.75, 0.90].map(utilization); const baseline = K[K.length - 1]; for (const r of K) { const avoided = baseline.outage - r.outage, diff = r.idle - baseline.idle; write(C, `${(100 * r.target).toFixed(0)}%`, r.capacity.toFixed(2), r.cost.toFixed(0), r.idle.toFixed(0), `${r.saturation}/${PERIOD}`, `${r.outage} min`, avoided > 0 ? (diff / avoided).toFixed(1) : "-"); } console.log(` demand peak ${baseline.peak.toFixed(2)}; a saturated day produces ${SATURATION_MIN} min of outage`); console.log("\n4. unit cost as scale grows"); const D = [-7, 12, 14, 13, 14, 18]; write(D, "scale", "daily u", "u/M requests", "infra/M", "monitoring/M", "monitoring share"); for (const k of [1, 2, 4, 8]) { const t = daily(k), mo = daily(k, (x) => x[5]), M = (REQUEST * k) / 1e6; write(D, `${k}x`, t.toFixed(0), (t / M).toFixed(1), ((t - mo) / M).toFixed(1), (mo / M).toFixed(1), `${(100 * mo / t).toFixed(0)}%`); } const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfour tables and the scale sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`); ``` ``` 30 days, 9 cost items, daily 753 u (fictional currency); billing is 3 days delayed 1. cost question set question answer covered reason unanswered what does the total cost yes 100% - what does each service cost partial 70% unowned resource where did yesterday's spike come from no 0% delayed billing data which resource sits idle yes 100% - what does one request cost yes 100% - what does one service's request cost no 70% a shared item cannot be apportioned what does monitoring data cost yes 100% - what happens to price if scale doubles partial 100% cannot be read from the bill 4/8 questions are fully answered; 2 partial, 2 unanswered 2. item breakdown and monitoring's own share item daily u share owned type service instances 174 23% owned infra memory and local disk 86 11% owned infra block storage 70 9% owned infra nightly batch job cluster 75 10% owned infra shared data store 150 20% none infra cross-region transfer 74 10% none infra time series store 54 7% owned monitoring log store 44 6% owned monitoring trace store 26 3% owned monitoring total 753 100% 70% 16% (last row: owned share, monitoring share) monitoring 124 u/day 3. target utilization rate: idle resource alongside saturation target capacity period u idle u saturation outage idle u/min 30% 3.33 27785 19449 0/30 0 min 51.5 45% 2.22 18523 10188 0/30 0 min 25.7 60% 1.67 13892 5557 0/30 0 min 12.9 75% 1.33 11114 2778 0/30 0 min 5.1 90% 1.11 9262 926 8/30 360 min - demand peak 1.21; a saturated day produces 45 min of outage 4. unit cost as scale grows scale daily u u/M requests infra/M monitoring/M monitoring share 1x 753 607.3 507.3 100.0 16% 2x 1356 546.7 418.5 128.2 23% 4x 2706 545.5 370.9 174.6 32% 8x 5832 587.9 344.3 243.6 41% four tables and the scale sweep within the 2000 ms band: yes ``` The numbers are of the **measurement** class; their inputs are the assumptions above. ## Thirty Percent of the Bill Is Unowned The first table ties eight cost questions to a record and writes what share of the bill each question is answered for: four in full, two partial, two unanswered. The reasons are named separately. **Unowned resource:** the shared data store (20%) and cross-region transfer (10%) carry no owner tag; "what does each service cost" is answered for only **70%** of the bill. **A shared item cannot be apportioned:** the same 30% shuts down "what does one service's request cost" entirely — total unit cost can be calculated, per-service cost cannot. **Delayed billing data:** because the detail comes three days late, "where did yesterday's spike come from" is unanswered with **0%** coverage, and in those three days the same spike is paid for again every day. ## Monitoring Carries Its Own Bill The second table breaks down the items. The daily total is 753 units; **124 units** of that (16%) is monitoring data's own cost — the time series store 54, the log store 44, the trace store 26. All three are on the measuring side, and none of them serves a single request. Measuring the measuring side is the course's own rule applied to itself. Two numbers read alongside this: removing the single label that opened no question recovered 89% of the full series, and panels that answered no question scanned 95% of the query cost. **Unanswered data shows up in this table as a line item.** ## Idle Resource Alongside Saturation The third table puts both costs on the same row. As the target utilization rate rises, capacity shrinks: period cost drops from 27,785 to 9,262 units, and the cost of idle capacity from 19,449 to 926. Against that, at a 90% target, demand exceeds capacity on **eight** of thirty days, and 360 minutes of outage result; demand peak is 1.21, capacity is 1.11. The last column is the decision itself: the per-minute price of avoiding saturation. At a 75% target, saturation is zero, and the extra idle capacity costs **5.1 units per minute avoided**. The same zero saturation costs 12.9 units at a 60% target, 25.7 at 45%, 51.5 at 30%. **Idle resource is insurance, and the price of the same coverage varies tenfold;** pulling the target below 75% does not buy a single additional minute of saturation avoided. One side of this table is unmeasurable: how many units an outage minute corresponds to is nowhere written down. The resource's price is read from the bill; the outage's price is not. ## The Line Item Where Economies of Scale Fail The fourth table scales up eightfold. Infrastructure behaves as expected: per million requests it drops from 507.3 to **344.3 units**, because the fixed floors amortize and the shared data store and the batch job cluster have a scale exponent under one. Monitoring goes the other way. Per million requests it rises from 100.0 to **243.6 units**, and its share climbs from 16% to **41%**. The reason is written plainly in the assumption: as instance count grows `k`-fold, label cardinality grows too, series count is the product of the two, and the scale exponent settles at 1.5. The metric pipeline grows **faster** than the system it is measuring. Total unit cost is the sum of these two curves, and the turning point is in the table: it drops from 607.3 to 546.7, then to 545.5, and climbs back to **587.9** at eight times scale. Economies of scale run out at fourfold growth; past that, growing the system further does not lower unit cost, because the monitoring line item takes the savings back. **The line item where economies of scale fail is observability itself.** ## Summary - Four of eight cost questions are answered in full, two partially, two remain unanswered; three reasons are named — unowned resource (30%), a shared item that cannot be apportioned, and billing data delayed by three days. - 124 of the daily 753 units (16%) is monitoring data's own cost; the measuring side is a separate line item on the bill, and it serves no request at all. - At a 90% target utilization, 8 of 30 days saturate and 360 minutes of outage result; at 75%, saturation drops to zero and the avoided minute costs 5.1 units, while at 30% the same coverage costs 51.5 units. - As scale grows eightfold, infrastructure drops from 507.3 to 344.3 units per million requests, monitoring rises from 100.0 to 243.6 and its share climbs to 41%; total unit cost bottoms out at fourfold scale (545.5) and returns to 587.9 at eightfold. ## Course Wrap-Up | Lesson | Question answered | Question unanswered and why | Cost of the data | |---|---|---|---| | Metric Collection Models | 9/12 (push) | Q12, transaction ID not carried | 1.6 MB/question | | Time Series Data | 11/12 | Q12, not solved by a label | 634.82 GB of unanswered label | | Query and Aggregation | 2/4 measures exact | 729 requests, erased by the summary | 3.6M record scan | | Log Collection | 9/12 (1x load) | 3 questions, the truncated error line | 75 MB/question (2x load) | | Log Parsing | 4/6 correct | 2 questions, an unmaintained rule | 39,957 bytes/correct answer | | Retention and Cost | 11/12 | Q12, in no tier | 2.76 GB/question | | Distributed Tracing Infrastructure | 9/11 (trace) | 1 question, instrumentation depth | 52.7 MB/question | | Instrumentation Standards | 6/10 (convention) | 4 questions, an incompatible service | 20 field changes | | Dashboard Design | 11/12 (after redesign) | 1 question, an uncollected dimension | 814/859 unanswered series | | Alerting Rules | 7/8 incidents (three rules) | 1 incident, the metric itself | 21 false positives | | Alert Fatigue | 13/20 incidents | 7 incidents, the delay down the queue | 191 notifications, 93% empty | | Synthetic Monitoring | 10/10 (both together) | 3 failures, a fixed-input probe | 480 probes/day | | On-Call Rotation | 5/8 | 3 questions, unmeasured recovery | 118.7 recovery hours | | Incident Classification | 4/7 | 3 questions, before the incident closes | 11,040 impact-minutes | | Runbooks | 5/8 | 3 questions, the runbook's validity | 384 minutes of delay | | Post-Incident Review | 7/8 | 1 question, the counterfactual | 10,971 bytes/question | | Error Budget Management | 5/6 | 1 question, the counterfactual | 5,952 bytes/question | | Capacity and Cost Monitoring | 4/8 full | 2 questions, unowned resource | 124 u/day monitoring | The course's rule reads the same way across all eighteen rows: **data is worth as much as the question it answers; data that answers no question is cost, and a question left unanswered is risk.** The first column is never full in any lesson; the fourth column carries a number in every lesson. Four reasons repeat in the third column. **An uncollected dimension** appears in ten rows: the field was never written at all, and neither extending retention nor adding a label opens it — transaction ID, subscriber group, recovery window, unowned resource are all the same gap. **Deleted detail** is in three rows: the data was collected, then lost in some layer — the threshold question the summary erases, the truncated error line, the field an unmaintained rule leaves blank. **A late-arriving answer** is in three rows: the data exists but is not there at the moment it is asked. **The counterfactual** is in two rows, and no record opens it: the outage a closed item prevented, the incident a freeze prevented. The first three can be closed by a design decision; the fourth cannot, and counting what was not prevented takes its place. The system is visible now, and incidents are managed. The next course, **M22/K07 Servers, Networking and Hardening**, goes underneath what is visible: everything these eighteen lessons measured was running on top of a machine and a network, and both are still sitting at their default settings. An open port, an unwritten rule, an unchanged default account — how wide is that surface?