Cost Estimation

The previous lesson counted a validation run’s own cost and the leaked cost of the leftovers it left behind. Both were counted after the fact — once the money had already been spent. The same question can be asked before applying too, because the plan is already in hand: which resource will be added, which will be grown, which will be deleted is written right there. A plan diff is at the same time a bill diff.

Capacity and cost awareness were measured at system scale earlier; the unit here is the change itself: if this contribution is merged, how much does the monthly cost grow? This lesson produces an estimate per change from the plan, compares it against what actually happens, and counts three things: which item can be estimated, which cannot, how the error is distributed. Then a threshold rule — “stop if the estimate exceeds this much” — is run over a known set of changes, and the rule’s two failure modes are both counted. The through-line is the regional measurement network; the price list and the amounts are both fictional.

Estimable and Non-Estimable Items

A cost item is one of two kinds. A fixed-unit-price item’s amount is read directly from the plan: three machines, 120 gigabytes of disk, 500 gigabytes of storage. For these items, the estimate is a multiplication and does not change over the month. A usage-dependent item’s amount, though, is not in the plan; the plan says the resource will exist but does not say how many requests it will take, how many gigabytes of data it will send out, how many messages it will carry. For these items, the estimate rests on the expected usage written in the record — that is, it is an estimate built on top of an estimate.

// estimate.mjs — produces a monthly cost estimate per change from the plan (model)
import { writeFileSync } from "node:fs";

// IV19: the price list is fictional and has nothing to do with any provider's pricing; unit is "fu" (fictional unit)
const FIXED = { "machine-medium": 30, "machine-large": 60, "disk-gb": 0.5, "store-gb": 0.8,
  "archive-gb": 0.1 };                       // monthly fixed unit price per item
// usage-dependent item: [monthly unit price, uncertainty multiplier r] — real amount is between 1/r and r
const USAGE = { "request-million": [4, 4], "egress-gb": [0.9, 4], "message-million": [2, 2],
  "read-million": [3, 2] };

// IV20: twelve changes are read from the plan; each item's amount is the plan's resource
// difference, and for usage-dependent items it is the expected usage written in the record (fictional)
const CHANGE = [
  ["G01", "three collector machines are added", [["machine-medium", 3], ["disk-gb", 120],
    ["egress-gb", 60]]],
  ["G02", "data store is grown by 500 GB", [["store-gb", 500], ["read-million", 20]]],
  ["G03", "archive stream is added", [["archive-gb", 2000], ["read-million", 30]]],
  ["G04", "measurement queue is added", [["message-million", 120]]],
  ["G05", "collector size is made large", [["machine-large", 3], ["machine-medium", -3]]],
  ["G06", "new region: network and two machines", [["machine-medium", 2], ["disk-gb", 80], ["egress-gb", 300]]],
  ["G07", "collector disk goes from 40 to 200 GB", [["disk-gb", 480], ["read-million", 10]]],
  ["G08", "backup window goes from 7 days to 30", [["store-gb", 1400]]],
  ["G09", "external endpoint is opened", [["request-million", 20], ["egress-gb", 800]]],
  ["G10", "archive class is moved to cold", [["archive-gb", -1500]]],
  ["G11", "collector count goes from 3 to 9", [["machine-medium", 6], ["disk-gb", 240],
    ["egress-gb", 120]]],
  ["G12", "collection frequency doubles", [["message-million", 40], ["read-million", 60]]],
];

const EST = CHANGE.map(([code, what, items]) => {
  let fixed = 0;
  const usage = [];
  for (const [item, amount] of items) {
    if (FIXED[item] !== undefined) fixed += FIXED[item] * amount;
    else usage.push([item, USAGE[item][0] * amount, USAGE[item][1]]);
  }
  const u = usage.reduce((s, [, t]) => s + t, 0);
  return { code, what, fixed, usage, estimate: fixed + u, fixedItems: items.length - usage.length };
});
writeFileSync("estimate.json", JSON.stringify(EST));

console.log(`${"code".padEnd(6)}${"change".padEnd(40)}${"fixed items".padStart(13)}` +
  `${"usage items".padStart(16)}${"fixed fu".padStart(10)}${"usage fu".padStart(13)}` +
  `${"estimate fu".padStart(13)}`);
for (const d of EST) {
  const u = d.estimate - d.fixed;
  console.log(`${d.code.padEnd(6)}${d.what.padEnd(40)}${String(d.fixedItems).padStart(13)}` +
    `${String(d.usage.length).padStart(16)}${d.fixed.toFixed(1).padStart(10)}` +
    `${u.toFixed(1).padStart(13)}${d.estimate.toFixed(1).padStart(13)}`);
}
const fixedTotal = EST.reduce((s, d) => s + d.fixed, 0);
const usageTotal = EST.reduce((s, d) => s + (d.estimate - d.fixed), 0);
const itemCount = EST.reduce((s, d) => s + d.fixedItems + d.usage.length, 0);
console.log(`\n${EST.length} changes, ${itemCount} items: ` +
  `${EST.reduce((s, d) => s + d.fixedItems, 0)} fixed-unit-price (estimable), ` +
  `${EST.reduce((s, d) => s + d.usage.length, 0)} usage-dependent (not estimable)`);
console.log(`total estimate ${(fixedTotal + usageTotal).toFixed(1)} fu; ` +
  `fixed share ${fixedTotal.toFixed(1)} fu (${(100 * fixedTotal / (fixedTotal + usageTotal)).toFixed(0)}%), ` +
  `usage share ${usageTotal.toFixed(1)} fu (${(100 * usageTotal / (fixedTotal + usageTotal)).toFixed(0)}%)`);
console.log(`changes carrying a usage-dependent item: ${EST.filter((d) => d.usage.length).length}/${EST.length}`);

code  change                                    fixed items     usage items  fixed fu     usage fu  estimate fu
G01   three collector machines are added                  2               1     150.0         54.0        204.0
G02   data store is grown by 500 GB                       1               1     400.0         60.0        460.0
G03   archive stream is added                             1               1     200.0         90.0        290.0
G04   measurement queue is added                          0               1       0.0        240.0        240.0
G05   collector size is made large                        2               0      90.0          0.0         90.0
G06   new region: network and two machines                2               1     100.0        270.0        370.0
G07   collector disk goes from 40 to 200 GB               1               1     240.0         30.0        270.0
G08   backup window goes from 7 days to 30                1               0    1120.0          0.0       1120.0
G09   external endpoint is opened                         0               2       0.0        800.0        800.0
G10   archive class is moved to cold                      1               0    -150.0          0.0       -150.0
G11   collector count goes from 3 to 9                    2               1     300.0        108.0        408.0
G12   collection frequency doubles                        0               2       0.0        260.0        260.0

12 changes, 24 items: 13 fixed-unit-price (estimable), 11 usage-dependent (not estimable)
total estimate 4362.0 fu; fixed share 2450.0 fu (56%), usage share 1912.0 fu (44%)
changes carrying a usage-dependent item: 9/12

Thirteen of the twenty-four items are fixed-unit-price, eleven are usage-dependent. By item count, the majority looks estimable, but the split is different by amount: fifty-six percent of the total is fixed, forty-four percent usage-dependent. More important is the distribution — nine of the twelve changes carry at least one usage-dependent item. Three changes (G05, G08, G10) consist only of fixed items, and their estimate is not a calculation but a certainty.

Comparing Against Actuals

IV21 — a fixed-unit-price item does not change over the month, its error is exactly zero; a usage-dependent item’s actual amount is between 1/r and r times the expected amount. IV22 — twelve changes are applied within the same month, and the actual amount is read at month end (fictional). The uncertainty multiplier varies by item: external requests and egress data are the most volatile items (r = 4), queue messages and store reads are calmer (r = 2).

// actual.mjs — the actual cost, the distribution of estimate error, and the threshold rule's result (model)
import { readFileSync } from "node:fs";

// IV21: a fixed-unit-price item does not change over the month, its error is exactly zero; a
// usage-dependent item's actual amount is between 1/r and r times the expected amount (r is the item's multiplier)
const SEED = 20260401;
let state = SEED;
const random = () => (state = (state * 48271) % 2147483647) / 2147483647;

const ACTUAL = JSON.parse(readFileSync("estimate.json", "utf8")).map((d) => {
  const actual = d.fixed + d.usage.reduce((s, [, amount, r]) =>
    s + amount * Math.pow(r, 2 * random() - 1), 0);
  return { ...d, actual, error: actual - d.estimate };
});

console.log(`${"code".padEnd(6)}${"estimate fu".padStart(13)}${"actual fu".padStart(16)}` +
  `${"error fu".padStart(10)}${"error %".padStart(9)}${"source".padStart(20)}`);
for (const d of ACTUAL) {
  const pct = d.estimate === 0 ? 0 : (100 * d.error) / Math.abs(d.estimate);
  console.log(`${d.code.padEnd(6)}${d.estimate.toFixed(1).padStart(13)}${d.actual.toFixed(1).padStart(16)}` +
    `${d.error.toFixed(1).padStart(10)}${pct.toFixed(0).padStart(9)}` +
    `${(d.usage.length ? `${d.usage.length} usage item${d.usage.length === 1 ? "" : "s"}` : "fixed only").padStart(20)}`);
}

const exact = ACTUAL.filter((d) => d.error === 0);
const off = ACTUAL.filter((d) => d.error !== 0);
const ratio = (d) => d.actual / d.estimate;
console.log(`\n${exact.length} changes have exactly zero error (they carry only fixed-unit-price ` +
  `items); ${off.length} changes are off`);
console.log(`deviation ranges from ${Math.min(...off.map(ratio)).toFixed(2)}x to ` +
  `${Math.max(...off.map(ratio)).toFixed(2)}x; ` +
  `total estimate ${ACTUAL.reduce((s, d) => s + d.estimate, 0).toFixed(1)} fu, ` +
  `actual ${ACTUAL.reduce((s, d) => s + d.actual, 0).toFixed(1)} fu ` +
  `(${(100 * (ACTUAL.reduce((s, d) => s + d.error, 0) / ACTUAL.reduce((s, d) => s + d.estimate, 0))).toFixed(0)}% overall)`);

// IV23: the threshold rule looks at the plan's estimate: if the estimate exceeds the threshold, the change is held and goes to manual approval
console.log(`\n${"threshold fu".padStart(14)}${"held".padStart(7)}${"held for nothing".padStart(18)}` +
  `${"costly miss".padStart(14)}${"correctly held".padStart(16)}${"budget to approval".padStart(20)}`);
for (const E of [200, 300, 500, 800]) {
  const held = ACTUAL.filter((d) => d.estimate > E);
  const forNothing = held.filter((d) => d.actual <= E);
  const missed = ACTUAL.filter((d) => d.estimate <= E && d.actual > E);
  console.log(`${String(E).padStart(14)}${String(held.length).padStart(7)}` +
    `${String(forNothing.length).padStart(18)}${String(missed.length).padStart(14)}` +
    `${String(held.length - forNothing.length).padStart(16)}` +
    `${held.reduce((s, d) => s + d.estimate, 0).toFixed(0).padStart(20)}`);
}
for (const E of [200, 300]) {
  const ad = (f) => ACTUAL.filter(f).map((d) => d.code).join(", ") || "none";
  console.log(`\nthreshold ${E} fu: held ${ad((d) => d.estimate > E)}; ` +
    `held for nothing ${ad((d) => d.estimate > E && d.actual <= E)}; ` +
    `costly miss ${ad((d) => d.estimate <= E && d.actual > E)}`);
}

code    estimate fu       actual fu  error fu  error %              source
G01           204.0           192.3     -11.7       -6        1 usage item
G02           460.0           461.5       1.5        0        1 usage item
G03           290.0           362.5      72.5       25        1 usage item
G04           240.0           121.6    -118.4      -49        1 usage item
G05            90.0            90.0       0.0        0          fixed only
G06           370.0           535.2     165.2       45        1 usage item
G07           270.0           288.3      18.3        7        1 usage item
G08          1120.0          1120.0       0.0        0          fixed only
G09           800.0           958.4     158.4       20       2 usage items
G10          -150.0          -150.0       0.0        0          fixed only
G11           408.0           368.8     -39.2      -10        1 usage item
G12           260.0           316.2      56.2       22       2 usage items

3 changes have exactly zero error (they carry only fixed-unit-price items); 9 changes are off
deviation ranges from 0.51x to 1.45x; total estimate 4362.0 fu, actual 4664.7 fu (7% overall)

  threshold fu   held  held for nothing   costly miss  correctly held  budget to approval
           200     10                 2             0               8                4422
           300      5                 0             2               5                3158
           500      2                 0             1               2                1920
           800      1                 0             1               1                1120

threshold 200 fu: held G01, G02, G03, G04, G06, G07, G08, G09, G11, G12; held for nothing G01, G04; costly miss none

threshold 300 fu: held G02, G06, G08, G09, G11; held for nothing none; costly miss G03, G12

The first reading of the error table is misleading. In total, the estimate is 4,362, the actual is 4,664.7 fictional units — a seven percent deviation, which looks acceptable. But that seven percent comes from individual changes’ deviations canceling each other out: G04 comes in about half low (0.51x), G06 comes in forty-five percent high (1.45x), G03 twenty-five percent high. Overall accuracy gives the decision-maker nothing, because decisions are made per change. Seven percent is a good number for a budget report; for a gate, the deviation’s 0.51x–1.45x range is the number that matters.

The second reading shows where the error comes from. Three changes have exactly zero error, and that is not luck but a structural outcome: those three carry only fixed-unit-price items. All nine of the rest are off, and the size of the deviation grows with the item’s uncertainty multiplier. The way to bring down estimate error is not a better estimation tool; it is knowing the share that is usage-dependent. If a change’s estimate does not come with the number “what percent of this amount is usage-dependent,” the estimate alone does not say how reliable it is.

The one negative item in the table follows the same rule: moving the archive class to cold is estimated as a monthly saving of 150 units, and it saves exactly 150 units. In a fixed-unit-price item, savings are as certain as spend; in a usage-dependent item, a promised reduction carries the same uncertainty and may not materialize.

Threshold Rule

The moment an estimate is tied to a gate, it starts producing decisions: if the estimate exceeds the threshold, the change is held and goes to manual approval. The rule has two failure modes, and both are counted in the table. Held for nothing is a change whose estimate exceeds the threshold but whose actual does not; its cost is a waiting contribution and spent approval time. Costly miss is a change whose estimate stays under the threshold but whose actual comes in above it; its cost is a surprise in the budget.

A two-hundred-unit threshold holds ten of the twelve changes. At this threshold, no costly change slips through — but two of the held changes (G01 and G04) actually stay under the threshold, and the budget going to approval climbs to 4,422 units, nearly the entire total. At this point the gate stops being a gate: everything is held, and approval turns into a ceremony. At a three-hundred-unit threshold, the held count drops to five, and all five of those actually exceed the threshold — but this time two changes (G03 and G12) pass because their estimate stays under it, and they actually come in above it.

At an eight-hundred-unit threshold, only one change is held and one slips through. No threshold zeroes out both at once, because a threshold looks at a single number, and that number is itself uncertain. The way to improve the rule is not to choose a better threshold; it is to change the number the threshold looks at: if the threshold looks at the estimate’s usage-dependent share instead of the estimate itself, a change like G08, which carries only fixed items, can pass automatically despite its high amount, because its estimate is certain.

There is one more limit. These estimates’ input is the plan, and the plan does not fully predict the apply either: an item that never appears in the plan does not enter the estimate either, and an item that appears in the plan but does not materialize inflates the estimate. All the numbers here assume the plan is correct; the gap between the record and reality is a second error stacked on top of the estimate error.

Summary

• The estimate for twelve changes was produced from the plan diff: 13 of 24 items are fixed-unit-price (estimable), 11 are usage-dependent (not estimable); 56% of the amount is fixed, 44% usage-dependent, and 9 of 12 changes carry at least one usage-dependent item.
• The 3 changes that carry only fixed items have exactly zero error; the remaining 9 deviate between 0.51x and 1.45x.
• The overall deviation is 7%, and that number is deceptive: individual deviations cancel each other out, while decisions are made per change.
• The threshold rule’s two failure modes work in opposite directions: a 200-unit threshold holds 10 of 12 changes (2 for nothing, budget to approval 4,422 units) and misses no costly change; at 300 units, 5 are held and 2 costly changes pass; at 800 units, 1 is held and 1 costly change passes.
• The way to bring down estimate error is not a better tool, it is reporting the estimate’s usage-dependent share; if the threshold also looks at that share, a high-amount change with a certain estimate is not held for nothing.

Next Step

This topic’s lessons so far have checked the record, run it through policy, tested it, and priced it. All four looked at the same thing: the resource itself — machine, disk, network, queue, store. But when a machine is created, its inside is empty; which package will be installed on it, which file will sit with which permission, which service will stay open is not written in the provisioning layer’s record. The next lesson takes up the far side of that boundary: the configuration management layer that takes over once the machine is created, the difference between agent-based and agentless approaches, and where that layer’s own record overlaps with the provisioning record and where it diverges.