--- title: 'Trade-Off Analysis' source: 'https://academia.sh/en/courses/architectural-documentation/trade-off-analysis' course: 'Architectural Decisions and Documentation' language: en updated: '2026-08-23T07:01:04+00:00' license: 'CC BY-SA 4.0' --- # Trade-Off Analysis Measuring the recorded state of the table where alternatives are scored along quality axes: whether a third person can arrive at the same result from the recorded table is counted, a missing criterion, an unwritten weight, and an unsourced score are separated as three distinct flaws, each flaw is fixed one at a time to measure the rate of reproducible decisions, and the cost of completion in minutes is computed per gained analysis. The previous lesson measured an evaluation round flipping a decision, and left behind a one-line rationale of the form "changed because of this information." Behind that line stood multiple alternatives, multiple qualities, and every alternative's standing on every quality. A **trade-off analysis** is the name of that table. How the table gets built, how the weights determine the winner, and how resilient the decision is under a sweep of the weight space were measured in the previous course. They are not repeated here. This lesson's question is different: once the analysis is done and recorded, can a third person reading that record arrive at the same result. That is the quantity being measured — **reproducibility**. ## The Recorded Table and the Third Person For an analysis to be reproducible, the record must contain four things: alternatives, criteria, scores, and weights. In practice, three flaws show up, and each breaks it a different way. **Missing criterion.** A quality that influenced the decision never entered the table at all. The table looks complete, because the missing column leaves no blank spot behind — it is not there at all. **Unwritten weight.** The criteria are written, the weights are not. The reader is forced to make an assumption, and the assumption they will make is equal weight; as measured in the previous course, equal weight is also a weight set — it just was not stated as a choice. **Unsourced score.** The score is written, but where it came from is not. This flaw does not change the winner — it blocks verification: the reader arrives at the same result, but cannot test whether the result they arrived at is correct. The model below holds five analysis records. Alongside each record sits the real information that never made it into the record, as the model's input: the missing criterion's name and scores, and the weight vector the decision actually rested on (**DR10**). The third person does not see any of this; they only read the record. The scenario is fictional. ```js // tradeoff/table.mjs — trade-off analyses as they made it into the record; the scenario is fictional // The recorded table, the missing criterion, and the actual weight vector are the model's input (DR10). export const ANALYSES = [ { name: "search-index", options: ["in-catalog", "separate-service", "external-service"], criteria: ["response-time", "operating-load", "cost"], scores: [[2, 2, 2], [5, 4, 3], [4, 5, 2]], weights: null, unsourced: 2, missing: { name: "data-freshness", scores: [5, 2, 2] }, actual: [0.20, 0.15, 0.15, 0.50], recorded: "in-catalog" }, { name: "report-generation", options: ["central", "branch", "nightly-batch"], criteria: ["freshness", "server-load", "development-time"], scores: [[4, 2, 4], [5, 4, 2], [2, 5, 5]], weights: [0.50, 0.30, 0.20], unsourced: 1, missing: null, actual: [0.50, 0.30, 0.20], recorded: "branch" }, { name: "member-identity", options: ["central-service", "branch-table", "daily-copy"], criteria: ["consistency", "outage-tolerance", "operating-load"], scores: [[5, 2, 3], [2, 5, 4], [3, 4, 2]], weights: null, unsourced: 0, missing: null, actual: [0.55, 0.25, 0.20], recorded: "central-service" }, { name: "penalty-engine", options: ["loan-service", "catalog-module", "branch-setting"], criteria: ["modifiability", "setup-load"], scores: [[5, 3], [2, 5], [3, 4]], weights: [0.60, 0.40], unsourced: 2, missing: { name: "branch-exception", scores: [2, 3, 5] }, actual: [0.35, 0.25, 0.40], recorded: "branch-setting" }, { name: "backup-path", options: ["night-window", "continuous-stream", "weekly-full"], criteria: ["data-loss-window", "operating-load", "cost"], scores: [[4, 4, 4], [5, 2, 2], [2, 5, 5]], weights: [0.45, 0.30, 0.25], unsourced: 3, missing: null, actual: [0.45, 0.30, 0.25], recorded: "night-window" }, ]; export const winner = (options, scores, w) => options .map((name, i) => ({ name, total: scores[i].reduce((t, p, j) => t + p * w[j], 0) })) .sort((a, b) => b.total - a.total)[0].name; export const equal = (n) => Array(n).fill(1 / n); // The table the reader has: no missing criterion, and an unwritten weight is assumed equal. export function asRead(a, fix = {}) { const criteria = fix.addCriterion && a.missing ? [...a.criteria, a.missing.name] : [...a.criteria]; const scores = a.options.map((_, i) => fix.addCriterion && a.missing ? [...a.scores[i], a.missing.scores[i]] : [...a.scores[i]]); let w = fix.writeWeight ? a.actual.slice(0, criteria.length) : a.weights ? a.weights.slice(0, criteria.length) : equal(criteria.length); while (w.length < criteria.length) w = [...w, 1 / criteria.length]; // default for the added criterion const s = w.reduce((t, x) => t + x, 0); return { criteria, scores, w: w.map((x) => x / s) }; } if (import.meta.url.endsWith(process.argv[1].split("/").pop())) { const k = (fn) => ANALYSES.filter(fn).length; console.log(`${ANALYSES.length} analyses; flaws: missing criterion ${k((a) => a.missing)}, ` + `unwritten weight ${k((a) => !a.weights)}, unsourced score ` + `${ANALYSES.reduce((t, a) => t + a.unsourced, 0)}\n`); console.log("reproduction with the recorded table:"); for (const a of ANALYSES) { const r = asRead(a); const flaws = [a.missing && "missing criterion", !a.weights && "unwritten weight", a.unsourced && `${a.unsourced} unsourced score`].filter(Boolean); console.log(` ${a.name.padEnd(17)} recorded ${a.recorded.padEnd(17)} read ` + `${winner(a.options, r.scores, r.w).padEnd(17)} flaws: ${flaws.join(", ") || "-"}`); } const FIXES = [["as recorded", {}], ["once weights are written", { writeWeight: true }], ["once the missing criterion is added", { addCriterion: true }], ["both", { writeWeight: true, addCriterion: true }], ["once sources are written too", { writeWeight: true, addCriterion: true, writeSource: true }]]; console.log("\neffect of each fix (reproducible / also verifiable):"); for (const [name, fix] of FIXES) { const reproducible = ANALYSES.filter((a) => { const r = asRead(a, fix); return winner(a.options, r.scores, r.w) === a.recorded; }); const verifiable = reproducible.filter((a) => fix.writeSource || a.unsourced === 0); console.log(` ${name.padEnd(36)} ${reproducible.length}/${ANALYSES.length} ${verifiable.length}/${ANALYSES.length}`); } } ``` ``` 5 analyses; flaws: missing criterion 2, unwritten weight 2, unsourced score 8 reproduction with the recorded table: search-index recorded in-catalog read separate-service flaws: missing criterion, unwritten weight, 2 unsourced score report-generation recorded branch read branch flaws: 1 unsourced score member-identity recorded central-service read branch-table flaws: unwritten weight penalty-engine recorded branch-setting read loan-service flaws: missing criterion, 2 unsourced score backup-path recorded night-window read night-window flaws: 3 unsourced score effect of each fix (reproducible / also verifiable): as recorded 2/5 0/5 once weights are written 3/5 1/5 once the missing criterion is added 3/5 0/5 both 5/5 1/5 once sources are written too 5/5 5/5 ``` Two of the five analyses are reproducible as recorded. In three, the reader arrives at a winner different from the one written in the record: the search-index record says in-catalog but the table gives separate-service, the member-identity record says central-service but the table gives branch-table. This is not the reader misunderstanding the decision — the recorded table genuinely produces that result. The decision itself is correct; the record is wrong. The fix table separates the weight of the three flaws. Writing the weights raises the number of reproducible analyses from 2 to 3; adding the missing criterion also from 2 to 3. Neither fix alone is enough, because search-index carries both flaws at once: once the weight is written, the missing column is still absent; once the column is added, the weight is still assumed equal. Once both are fixed, all five of the five analyses become reproducible. The flaws do not add — they collide. The third flaw's effect is in the second column. Even once all five analyses are reproducible, only one is verifiable: in the remaining four, the scores' source is not written, so the reader arrives at the same winner but cannot test whether the scores are correct. The gap between these two states looks small and is large. A reproducible analysis shows that a decision is **consistent**; a verifiable analysis shows that a decision is **well-founded**. The backup-path record gives the correct winner as recorded, but none of its three scores says where it came from — it is possible for all three to be wrong and the winner to come out the same regardless. ## Question Set and Completion The same five records are run against a question set. As separated in the previous lesson, an answer can be correct, missing, or wrong; the distinction matters especially here, because a table carrying a missing criterion answers "which criteria were used" **wrong**: the list is incomplete but looks complete. ```js // tradeoff/questions.mjs — the question set applied to the analysis records; the cost of completion is measured import { ANALYSES, winner, asRead } from "./table.mjs"; const reproducible = (a, fix) => { const r = asRead(a, fix); return winner(a.options, r.scores, r.w) === a.recorded; }; const QUESTIONS = [ ["O1", "which criteria were used", (a, c) => (!c && a.missing ? "wrong" : "correct")], ["O2", "what were the criteria's weights", (a, c) => (!c && !a.weights ? "missing" : "correct")], ["O3", "where did this score come from", (a, c) => (!c && a.unsourced ? "missing" : "correct")], ["O4", "does the same table give the same result", (a, c) => (reproducible(a, c ? { writeWeight: true, addCriterion: true } : {}) ? "correct" : "wrong")], ["O5", "which alternatives were evaluated", () => "correct"], ]; const state = {}; for (const complete of [false, true]) { const count = { correct: 0, missing: 0, wrong: 0 }; for (const a of ANALYSES) for (const [code, , rule] of QUESTIONS) { const status = rule(a, complete); count[status] += 1; (state[code] ??= { false: [], true: [] })[complete].push(status); } console.log(`${complete ? "after completion " : "as recorded "}: ` + `correct ${count.correct}, missing ${count.missing}, wrong ${count.wrong} ` + `(${QUESTIONS.length} questions x ${ANALYSES.length} analyses)`); } console.log("\nnot-correct answers per question as recorded (w = wrong, m = missing):"); for (const [code, text] of QUESTIONS) { const s = state[code][false]; const w = s.filter((x) => x === "wrong").length, m = s.filter((x) => x === "missing").length; console.log(` ${code} ${text.padEnd(40)} ${w ? `${w}w` : m ? `${m}m` : "-"}`); } // The completion times are the model's input (DR11). const MIN = { weight: 15, criterion: 20, source: 5 }; const count = { weight: ANALYSES.filter((a) => !a.weights).length, criterion: ANALYSES.filter((a) => a.missing).length, source: ANALYSES.reduce((t, a) => t + a.unsourced, 0) }; const total = Object.entries(count).reduce((t, [k, n]) => t + n * MIN[k], 0); console.log("\ncompletion cost:"); for (const [k, n] of Object.entries(count)) console.log(` ${k.padEnd(9)} ${n} items x ${MIN[k]} min = ${n * MIN[k]} min`); const before = ANALYSES.filter((a) => reproducible(a, {})).length; const after = ANALYSES.filter((a) => reproducible(a, { writeWeight: true, addCriterion: true })).length; console.log(` total ${total} min; reproducible ${before}/${ANALYSES.length} -> ` + `${after}/${ANALYSES.length}, per gained analysis ${(total / (after - before)).toFixed(1)} min`); ``` ``` as recorded : correct 14, missing 6, wrong 5 (5 questions x 5 analyses) after completion : correct 25, missing 0, wrong 0 (5 questions x 5 analyses) not-correct answers per question as recorded (w = wrong, m = missing): O1 which criteria were used 2w O2 what were the criteria's weights 2m O3 where did this score come from 4m O4 does the same table give the same result 3w O5 which alternatives were evaluated - completion cost: weight 2 items x 15 min = 30 min criterion 2 items x 20 min = 40 min source 8 items x 5 min = 40 min total 110 min; reproducible 2/5 -> 5/5, per gained analysis 36.7 min ``` Fourteen of the twenty-five answers are correct, six missing, five wrong. The distribution of wrong answers shows the flaws' nature: missing criterion and non-reproducibility produce wrong, unwritten weight and unsourced score produce missing. That difference determines the reader's behavior. Someone who sees that a weight is not written knows something is missing and can ask; someone looking at a missing criterion sees nothing, because the table looks complete. Completion costs 110 minutes: two weight vectors, two criterion columns, and eight score sources. At the end of that time the number of reproducible analyses rises from 2 to 5, that is, 36.7 minutes per gained analysis. The number itself depends on the model's inputs; what it carries is in the ratios. Writing the score sources takes 36 percent of the total time and does not change reproducibility at all — it only raises verifiability from zero to five. Writing the weights, by contrast, touches three analyses in 30 minutes. The order that follows from this is: missing criterion first, then weight, then score source last. That order comes not from the flaws' cost but from how detectable they are. The most insidious flaw is not the cheapest one — it is the one that does not show itself. ## Summary - A trade-off analysis's recorded state is measured by whether a third person can arrive at the same winner; two of the five records were reproducible as recorded. - The three flaws break it differently: a missing criterion makes the table look complete, an unwritten weight pushes the reader into assuming equal weight, an unsourced score blocks verification without changing the winner. - Writing the weights raised the reproducible count from 2 to 3, and adding the missing criterion also from 2 to 3; the analysis carrying both flaws became reproducible only once both were fixed, and the count rose to 5. - Reproducibility and verifiability are separate measures: even once all five analyses were reproducible, only one was verifiable without the score sources written. - In the question set, missing criterion and non-reproducibility produced wrong answers, unwritten weight and unsourced score produced missing answers; completion cost 110 minutes, 36.7 minutes per gained analysis. ## Next Step Everything measured up to this point concerned a decision already made: its record, the process that produced it, and the table it rested on. In all three, the decision settles at some point and the record closes. But something stays open next to every decision: the chance that the assumption it rests on does not hold. In the search-index analysis, the data-freshness criterion was given a weight of 0.50 because freshness was assumed critical; if that assumption is wrong, the decision is wrong too, and only time reveals it. This is not a decision to record — it is an uncertainty to track. The next lesson models that uncertainty with likelihood and impact and runs it across a period, counting the realization rate of recorded risk and the risk that occurs despite never being recorded at all.