Estimation and Evaluation

The previous lesson measured how well the document’s claim stayed equal to the code. Everything measured up to this point had one thing in common: it described something already past — a decision made, a boundary drawn, code running. One more thing an architect commits to in writing looks at the future instead — how long this work will take.

An estimate is also a document, and this course’s measure applies to it too: which question it answers, which it cannot, what its deviation is. Rough size estimation was set up in the Introduction to System Design course; the size computed there was data and request volume. The size here is work size, its unit is person-days, and its verification is different from a volume estimate’s — a volume estimate is verified approximately, a work estimate is compared against what actually happened, item by item.

Estimating in Two Forms

The block below models and defines a quarter’s work list, the band form of the estimates, and the outcome model, then runs a single outcome. The outcome model consists of two numbers, and both are visible: systematic optimism and item-specific variability.

// estimate.mjs — defines the work items, the band rule, and the outcome model; when run
// directly, measures a single outcome. The next two blocks import this file.

// VM21: the regional library network system's work list for the next quarter; estimates
// in person-days.
export const WORK = [
  ["branch definition flow", 5], ["catalog wrapper endpoint", 8], ["loan record migration", 13],
  ["membership verification", 5], ["notification channel", 3], ["fee calculation rules", 8],
  ["reservation queue", 13], ["report screen", 5], ["failover rehearsal", 8],
  ["bulk upload window", 3], ["audit script", 2], ["view document", 5],
];
// The band is derived from the single number: low bound three-quarters, high bound one
// and a half times. Whoever gives the estimate is saying the band will cover nine out of
// ten outcomes.
export const band = (t) => [Math.round(t * 0.75), Math.round(t * 1.5)];

// VM22: the outcome model. Outcome = estimate x 1.25 x exp(N(0, 0.45)). 1.25 stands for
// systematic optimism (work not visible in the estimate), exp(N(0, 0.45)) stands for
// item-specific independent variability. Both numbers are inputs to the model; every
// result below depends on them.
export const OPTIMISM = 1.25;
export const SPREAD = 0.45;

export const generator = (seed) => () => {          // the generator is hand-written, the seed is visible
  seed = (seed + 0x6d2b79f5) | 0;
  let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
export const normal = (r) => {                    // Box-Muller: one normal value from two uniform ones
  const u = Math.max(r(), 1e-12);
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * r());
};
export const multiplier = (r) => OPTIMISM * Math.exp(SPREAD * normal(r));

if (process.argv[1]?.endsWith("estimate.mjs")) {
  const SEED = 60406;
  const r = generator(SEED);
  console.log(`seed ${SEED}; optimism ${OPTIMISM}, spread ${SPREAD}`);
  console.log(`${"work item".padEnd(26)}${"estimate".padStart(9)}${"band".padStart(10)}` +
    `${"actual".padStart(12)}${"ratio".padStart(7)}${"in band".padStart(10)}`);
  let totalEstimate = 0, totalActual = 0, within = 0;
  for (const [name, t] of WORK) {
    const [low, high] = band(t);
    const a = Math.round(t * multiplier(r) * 10) / 10;
    const inBand = a >= low && a <= high;
    totalEstimate += t; totalActual += a; within += inBand ? 1 : 0;
    console.log(`${name.padEnd(26)}${String(t).padStart(9)}${`${low}-${high}`.padStart(10)}` +
      `${a.toFixed(1).padStart(12)}${(a / t).toFixed(2).padStart(7)}` +
      `${(inBand ? "yes" : "no").padStart(10)}`);
  }
  const [pLow, pHigh] = band(totalEstimate);
  console.log(`total estimate ${totalEstimate} days, actual ${totalActual.toFixed(1)} days, ` +
    `ratio ${(totalActual / totalEstimate).toFixed(2)}`);
  console.log(`the band covered ${within} of ${WORK.length} items; for the project total the band ` +
    `is ${pLow}-${pHigh} days and the actual is ${totalActual >= pLow && totalActual <= pHigh ? "in band" : "out of band"}`);
}

seed 60406; optimism 1.25, spread 0.45
work item                  estimate      band      actual  ratio   in band
branch definition flow            5       4-8         8.6   1.72        no
catalog wrapper endpoint          8      6-12         8.0   1.00       yes
loan record migration            13     10-20        12.2   0.94       yes
membership verification           5       4-8         5.6   1.12       yes
notification channel              3       2-5         1.5   0.50        no
fee calculation rules             8      6-12         8.3   1.04       yes
reservation queue                13     10-20        17.6   1.35       yes
report screen                     5       4-8        10.1   2.02        no
failover rehearsal                8      6-12         5.0   0.63        no
bulk upload window                3       2-5         2.0   0.67       yes
audit script                      2       2-3         3.9   1.95        no
view document                     5       4-8         1.2   0.24        no
total estimate 78 days, actual 84.0 days, ratio 1.08
the band covered 6 of 12 items; for the project total the band is 59-117 days and the actual is in band

In a single run, the project total is 84 days against an estimate of 78: an eight percent deviation, inside the band. The same estimate looks much worse at the item level — six of twelve items land outside the band, two at a ratio of 1.95 and 2.02. The total’s good look comes from item deviations canceling out. A single run is not evidence; the same list has to run many times.

How the Deviation Is Distributed

// distribution.mjs — imports estimate.mjs; measures the deviation distribution and the band's coverage rate
import { WORK, band, generator, multiplier } from "./estimate.mjs";

const SEED = 31517;
const RUNS = 500;
const r = generator(SEED);
const totalEstimate = WORK.reduce((s, [, t]) => s + t, 0);

const itemRatio = [], projectRatio = [];
let itemWithin = 0, projectWithin = 0, under = 0;
for (let k = 0; k < RUNS; k += 1) {
  let totalActual = 0;
  for (const [, t] of WORK) {
    const a = t * multiplier(r);
    const [low, high] = band(t);
    itemRatio.push(a / t);
    if (a >= low && a <= high) itemWithin += 1;
    if (a < t) under += 1;
    totalActual += a;
  }
  projectRatio.push(totalActual / totalEstimate);
  const [pLow, pHigh] = band(totalEstimate);
  if (totalActual >= pLow && totalActual <= pHigh) projectWithin += 1;
}

const percentile = (d, p) => [...d].sort((a, b) => a - b)[Math.floor((d.length - 1) * p)];
const row = (name, d) => `${name.padEnd(16)}` +
  [0.1, 0.25, 0.5, 0.75, 0.9].map((p) => percentile(d, p).toFixed(2).padStart(8)).join("") +
  `${(d.reduce((s, x) => s + x, 0) / d.length).toFixed(2).padStart(10)}`;

console.log(`seed ${SEED}, ${RUNS} runs; ${itemRatio.length} item outcomes`);
console.log(`${"actual/estimate".padEnd(16)}${"p10".padStart(8)}${"p25".padStart(8)}` +
  `${"median".padStart(8)}${"p75".padStart(8)}${"p90".padStart(8)}${"mean".padStart(10)}`);
console.log(row("per item", itemRatio));
console.log(row("project total", projectRatio));
console.log(`\nband coverage: per item ${((itemWithin / itemRatio.length) * 100).toFixed(0)}%, ` +
  `project total ${((projectWithin / RUNS) * 100).toFixed(0)}% ` +
  `(the estimator said nine in ten, i.e. 90%)`);
console.log(`items landing under the estimate: ${((under / itemRatio.length) * 100).toFixed(0)}%; ` +
  `landing over: ${(100 - (under / itemRatio.length) * 100).toFixed(0)}%`);

seed 31517, 500 runs; 6000 item outcomes
actual/estimate      p10     p25  median     p75     p90      mean
per item            0.71    0.92    1.25    1.68    2.23      1.38
project total       1.13    1.23    1.36    1.51    1.67      1.38

band coverage: per item 54%, project total 72% (the estimator said nine in ten, i.e. 90%)
items landing under the estimate: 31%; landing over: 69%

Three results are read separately. First, the deviation distribution is wide and not symmetric: per item, the range between p10 and p90 runs from 0.71 to 2.23, more than a threefold difference between the tenth-percentile ends. 69 percent of items exceed the estimate.

Second, the item median and the project total’s center are not the same number. The per-item median is 1.25, while the project total’s median is 1.36 and its mean is 1.38. The reason comes from the distribution’s shape: the multiplier is right-skewed, so its mean sits above its median, and once twelve items are summed, the total converges toward the mean. Summing makes the mean visible, not the median. Someone who looks at a single item and says “it usually holds to the estimate” hits a systematic overrun at the project total.

Third, the band’s coverage rate is far below what was claimed. The estimator said the band would cover nine in ten; the measured coverage is 54 percent per item, 72 percent at the project total. The band is not wrong, its claim is: giving a range does not tie it to a confidence level. A band’s coverage is known only by comparing it against what actually happens; until then, a band differs from a single number only in form.

Splitting the Estimate

// split.mjs — imports estimate.mjs; measures the effect of splitting an estimate into pieces on the deviation
import { WORK, band, generator, normal, OPTIMISM, SPREAD } from "./estimate.mjs";

// VM23: when an item is split into k pieces, two things change. One, a (1 - 1/k) share of
// the previously invisible work becomes visible and gets added to the estimate. Two, each
// piece's variability is drawn independently, so the k pieces' outcomes average out. The
// actual work comes from the same model in both cases: estimate x 1.25 x variability.
const splitEstimate = (t, k) => t * (1 + (OPTIMISM - 1) * (1 - 1 / k));
const splitActual = (t, k, r) => {
  let total = 0;
  for (let i = 0; i < k; i += 1) total += (t * OPTIMISM / k) * Math.exp(SPREAD * normal(r));
  return total;
};

const SEED = 74113;
const RUNS = 500;
const totalEstimate = WORK.reduce((s, [, t]) => s + t, 0);
const percentile = (d, p) => [...d].sort((a, b) => a - b)[Math.floor((d.length - 1) * p)];

console.log(`seed ${SEED}, ${RUNS} runs, ${WORK.length} items`);
console.log(`${"pieces".padStart(6)}${"item estimate".padStart(14)}${"p10".padStart(7)}` +
  `${"median".padStart(8)}${"p90".padStart(7)}${"p90/p10".padStart(9)}` +
  `${"in band".padStart(10)}${"project median".padStart(16)}`);
for (const k of [1, 2, 4, 8]) {
  const r = generator(SEED);
  const ratio = [], projectRatio = [];
  let within = 0;
  for (let n = 0; n < RUNS; n += 1) {
    let totalActual = 0, totalEst = 0;
    for (const [, t] of WORK) {
      const e = splitEstimate(t, k);
      const a = splitActual(t, k, r);
      const [low, high] = band(e);
      ratio.push(a / e);
      if (a >= low && a <= high) within += 1;
      totalActual += a; totalEst += e;
    }
    projectRatio.push(totalActual / totalEst);
  }
  const p10 = percentile(ratio, 0.1), p90 = percentile(ratio, 0.9);
  console.log(`${String(k).padStart(6)}` +
    `${(splitEstimate(totalEstimate, k) / totalEstimate).toFixed(3).padStart(14)}` +
    `${p10.toFixed(2).padStart(7)}${percentile(ratio, 0.5).toFixed(2).padStart(8)}` +
    `${p90.toFixed(2).padStart(7)}${(p90 / p10).toFixed(2).padStart(9)}` +
    `${`${((within / ratio.length) * 100).toFixed(0)}%`.padStart(10)}` +
    `${percentile(projectRatio, 0.5).toFixed(2).padStart(16)}`);
}
console.log(`\nsplitting moves two numbers in opposite directions: it grows the estimate as`);
console.log(`previously invisible work becomes visible (the item estimate column), and`);
console.log(`averaging independent pieces narrows the spread (the p90/p10 column).`);

seed 74113, 500 runs, 12 items
pieces item estimate    p10  median    p90  p90/p10   in band  project median
     1         1.000   0.70    1.24   2.19     3.14       54%            1.35
     2         1.125   0.77    1.16   1.76     2.29       65%            1.21
     4         1.188   0.83    1.13   1.52     1.82       84%            1.16
     8         1.219   0.90    1.12   1.38     1.52       93%            1.13

splitting moves two numbers in opposite directions: it grows the estimate as
previously invisible work becomes visible (the item estimate column), and
averaging independent pieces narrows the spread (the p90/p10 column).

Splitting’s effect comes through two separate, measurable channels. The first grows the estimate: an item split into eight pieces has an estimate 1.219 times the unsplit one’s, because looking piece by piece brings a portion of previously invisible work onto the list. The second narrows the spread: independent pieces’ outcomes average out and balance each other, and the p90/p10 ratio drops from 3.14 to 1.52.

Together, the two channels pull the median deviation from 1.24 to 1.12, the project median from 1.35 to 1.13, and raise the band’s coverage from 54 to 93 percent. Most of the gain comes in the first splits: one piece to four raises coverage by 30 points, four to eight raises it by 9. Splitting’s return grows at a diminishing rate, and every piece then has to be estimated separately.

This lesson looked for answers to a set of three questions. “How long will this work take” is answered by a single number. “How confident” is answered only by a band, but only once its coverage rate has been measured — an unmeasured band looks like it answers this question without actually answering it. The third question, “what is this estimate based on,” stays unanswered in both. This is the same gap as the fourth lesson’s threshold source: the number gets written down, where it came from does not.

Summary

• Work-size estimation’s deviation is measured against what actually happens, item by item; in the model, the per-item p10-p90 range is 0.71-2.23 and 69 percent of items exceed the estimate.
• The item median and the project total’s center are separate numbers: the item median is 1.25, while the project total’s median is 1.36 and its mean is 1.38 — summing makes the mean visible, not the median.
• The band’s coverage rate is independent of what is claimed: a band said to cover nine in ten covers 54 percent per item, 72 percent at the project total; an unmeasured band differs from a single number only in form.
• Splitting works through two channels: it raises the estimate to 1.219x by adding a portion of invisible work, and it lowers the p90/p10 ratio from 3.14 to 1.52 by averaging independent pieces.
• Splitting into eight pulls the median deviation from 1.24 to 1.12, the project median from 1.35 to 1.13, and carries coverage to 93 percent; the return grows at a diminishing rate, with a 9-point gain after one-to-four.

Course Wrap-Up

This course ran on one rule: a document’s value is measured by how many of the questions someone who comes later asks, it answers. Each of the eleven lessons built its own question set, ran that set against a record, a view, a diagram, a scenario, or an estimate, and counted the unanswered along with the answered.

Lesson	Question set	Answered / unanswered	Maintenance cost or deviation
Architecture Decision Records	39 questions asked of 4 decisions	18/39 unrecorded, 39/39 with a structured record	4 records, 101 lines, 170 minutes; 15% of the time went to a section answering no question
The Proposal and Evaluation Process	question set asked of 8 decisions	correct answers 13, 32, and 40; verbal approval gave 12 wrong and 15 missing answers	440 and 420 person-minutes per flipped decision; the one-round scheme spent 900 person-minutes without changing direction
Trade-Off Analysis	question set asked of 5 analyses	2/5 reproducible as recorded, 5/5 once gaps were fixed; 1 verifiable	completion cost 110 minutes, 36.7 minutes per gained analysis
The Risk Register	2,348 realized risks	808 risks (34.4%) never in the record; correct answers rose from 16 to 40.9, wrong answers fell from 6 to 0.4	1.26 risks per period under yearly review; 602 person-minutes per risk caught early under monthly review
Technical Debt Management	9 debt items and a year’s change set	5 inadvertent items cost 6,820 minutes, 4 deliberate items 3,640 minutes; not noticed at all within a year at the 48-repeat threshold	the same work takes 63.7% longer in the debt-laden module; noticing delay runs from 2.5 months to 7.4 months
Architectural Views	24 questions	18 answered / 6 unanswered	280 writes across 200 changes, 80 of them repeats; 15.56 writes per answer
The Layered Diagram Approach	14 questions	14 answered at their own level / D03 unanswered at the context level, D11 at the first three levels	112 nodes in maintenance; 62 nodes read instead of 278 at a single level, the gap paid back every 4.24 rounds
Selecting the Diagram Type by Question	18 questions	16 answered / 2 unanswered	241 item updates across 150 changes; maintenance per answer between 2.8 and 24.3
Quality Attribute Scenarios	5 questions asked of 12 requirements, 60 total	29 answered / 31 unanswered	writing cost 1.99x; 9 requirements with no written threshold produce 5.31 disagreements per delivery
Keeping Documentation Current	7 questions	6 correct under the generated audit / 4 correct and 2 wrong under manual update	drift grows from 2 to 10; the audit costs 10 updates and 6 false alarms
Estimation and Evaluation	3 questions	2 answered / 1 unanswered (“what is this estimate based on”)	item median deviation 1.25; band coverage 54%, 93% with splitting

The table shows a pattern. No lesson closed its question set completely, and that is not a flaw: the unanswered questions came from the same two classes every time — either they asked to combine information from two separate views, or they asked where a number came from. The second class ran from the start of the course to the end; “where did this number come from” in the decision record, “the threshold’s source” in the quality attribute scenario, “what is this estimate based on” in estimation are three names for one gap.

The second pattern is in maintenance cost: every lesson counted a document form’s price, and the price never came out zero. The cost of keeping a document current is paid not at the moment of writing but at every change after; that is why a document’s value is meaningful not just by how many questions it answers, but by that number’s ratio to its maintenance cost.

All these records, views, scenarios, and estimates stayed inside a single system: the regional library network’s software system, with its own modules, boundaries, and document. Yet no system stands alone. It stands inside an institution, other systems stand alongside it, and it exchanges data with them. The next course — Enterprise Context and Integration — opens with this question: where does a system’s boundary run once its connections to what lies outside it are taken into account.