--- title: 'The Proposal and Evaluation Process' source: 'https://academia.sh/en/courses/architectural-documentation/proposal-and-evaluation-process' course: 'Architectural Decisions and Documentation' language: en updated: '2026-08-23T07:01:04+00:00' license: 'CC BY-SA 4.0' --- # The Proposal and Evaluation Process Measuring how a decision gets made: the same decision set is run through verbal approval and through a written proposal plus an evaluation round, the rate of decisions that change direction under evaluation, the number of rounds, the number of roles involved, and the time elapsed are counted, a question set is applied to the record the process leaves behind to separate a correct, missing, or wrong answer, and the decisions the written process only delays are measured. The previous lesson measured how decisions get recorded, and left a team name in the "issuer" field of all four records. That field says who a decision came from, not how it came about. A decision that passes on verbal approval and one evaluated through a written proposal sent to every relevant role produce the same record; the difference between the two does not show up in the record. This lesson's question is: does the process itself change the decision, and at what cost if it does. "A written process is good" carries no decision. The same decision set is run through two schemes, and the rate of decisions that change direction under evaluation, the number of rounds, the number of roles involved, and the time elapsed are counted. ## The Same Decision, Three Processes The model has eight decisions, and behind each one stands one or two pieces of information: a fact known to someone in a particular part of the system but not to the person writing the proposal. Each piece of information has three properties: which role holds it, which evaluation round it surfaces in, and whether it can flip the decision. These three are the model's inputs (**DR7**); the scenario is fictional. The processes are separated by access to pieces of information. In verbal approval there are two roles — the proposer and the approver — so only what those two know surfaces. In the written-proposal scheme, the proposal goes to six roles. The second distinction is in the number of rounds: in one scheme a round repeats only when the decision changes, in the other at least two rounds are run regardless. The durations and round rules are also the model's inputs (**DR8**). ```js // proposal/process.mjs — the same decision set through three processes: verbal approval and two written-proposal schemes // The decisions and pieces of information are the model's input (DR7); the scenario is fictional. export const DECISIONS = [ { name: "member-data-retention", info: [ { role: "business", round: 1, flips: true, note: "the regulation requires deletion after two years" }] }, { name: "branch-reporting", info: [ { role: "business", round: 1, flips: false, note: "the branch wants its month-end report same-day" }, { role: "budget", round: 2, flips: true, note: "central reporting requires a second server" }] }, { name: "catalog-upgrade", info: [ { role: "integration", round: 1, flips: true, note: "the new version breaks four fields in the mapping layer" }, { role: "budget", round: 1, flips: false, note: "the upgrade requires a license difference" }] }, { name: "loan-limit", info: [ { role: "loan", round: 1, flips: false, note: "the limit varies by branch type" }] }, { name: "search-index", info: [ { role: "integration", round: 2, flips: false, note: "the index holds the same data as the catalog" }] }, { name: "backup-window", info: [ { role: "proposer", round: 1, flips: false, note: "load drops outside the window" }] }, { name: "card-printing", info: [ { role: "budget", round: 1, flips: false, note: "unit cost is lower with outside printing" }, { role: "business", round: 2, flips: true, note: "the card must be printed the same day as member registration" }] }, { name: "notification-channel", info: [ { role: "loan", round: 1, flips: true, note: "a third of members have already picked a channel" }] }, ]; // In verbal approval, only what the two roles in the meeting know surfaces; in the written // scheme, the proposal goes to every role. The durations and round rules are the model's input (DR8). export const PROCESS = { "verbal-approval": { role: ["proposer", "manager"], writing: 0, personMin: 30, days: 0, minRounds: 1, maxRounds: 1 }, "written-one-round": { role: ["proposer", "manager", "business", "loan", "integration", "budget"], writing: 60, personMin: 20, days: 2, minRounds: 1, maxRounds: 3 }, "written-two-round": { role: ["proposer", "manager", "business", "loan", "integration", "budget"], writing: 60, personMin: 20, days: 2, minRounds: 2, maxRounds: 3 }, }; export function run(decision, processName) { const p = PROCESS[processName]; const seen = []; let round = 0, flips = 0; while (round < p.maxRounds) { round += 1; const fresh = decision.info.filter((b) => b.round === round && p.role.includes(b.role)); seen.push(...fresh); if (fresh.some((b) => b.flips)) { flips += 1; continue; } if (round >= p.minRounds) break; } return { round, flips, personMin: p.writing + round * p.role.length * p.personMin, days: round * p.days, seen, missed: decision.info.filter((b) => !seen.includes(b)) }; } if (import.meta.url.endsWith(process.argv[1].split("/").pop())) { console.log(`${DECISIONS.length} decisions, ${DECISIONS.flatMap((d) => d.info).length} pieces of information, ` + `${DECISIONS.flatMap((d) => d.info).filter((b) => b.flips).length} of them flip the decision`); for (const name of Object.keys(PROCESS)) { const r = DECISIONS.map((d) => run(d, name)); const changed = r.map((x, i) => x.flips && DECISIONS[i].name).filter(Boolean); const missed = r.flatMap((x) => x.missed); console.log(`\n[${name}] ${PROCESS[name].role.length} roles, rounds ${PROCESS[name].minRounds}-${PROCESS[name].maxRounds}`); console.log(` changed decisions ${changed.length}/${DECISIONS.length}: ${changed.join(", ") || "-"}`); console.log(` average rounds ${(r.reduce((t, x) => t + x.round, 0) / r.length).toFixed(2)}, ` + `total ${r.reduce((t, x) => t + x.personMin, 0)} person-min, ${r.reduce((t, x) => t + x.days, 0)} days`); console.log(` information that never surfaced ${missed.length}, of which would have flipped the decision ` + `${missed.filter((b) => b.flips).length}`); } } ``` ``` 8 decisions, 11 pieces of information, 5 of them flip the decision [verbal-approval] 2 roles, rounds 1-1 changed decisions 0/8: - average rounds 1.00, total 480 person-min, 0 days information that never surfaced 10, of which would have flipped the decision 5 [written-one-round] 6 roles, rounds 1-3 changed decisions 3/8: member-data-retention, catalog-upgrade, notification-channel average rounds 1.38, total 1800 person-min, 22 days information that never surfaced 3, of which would have flipped the decision 2 [written-two-round] 6 roles, rounds 2-3 changed decisions 5/8: member-data-retention, branch-reporting, catalog-upgrade, card-printing, notification-channel average rounds 2.25, total 2640 person-min, 36 days information that never surfaced 0, of which would have flipped the decision 0 ``` In verbal approval, none of the eight decisions changes direction. The reason is not that the decisions are correct — it is that none of the five pieces of information that could flip a decision are in that room. The verbal process never sees ten pieces of information, and five of them are the ones that would flip a decision. The result a process produces is bounded by what the participating roles know; where that boundary sits is invisible once the process is over. The written-proposal scheme flips three of the same eight decisions. These three were not chosen by chance: in each of them, the information that flipped the decision sat in a role the proposal writer had no access to — in the regulation, in the mapping layer, in the member records. What the written process does is not think a decision through better; it is bring into the discussion the role that holds the information capable of flipping it. The two pieces of information the one-round scheme misses are a separate finding. In the branch-reporting and card-printing decisions, the information that came up in the first round did not flip the decision, so a second round never opened, and the two pieces of information due to surface in the second round went unseen. The scheme that runs at least two rounds flips five of the same decisions, and the missed information drops to zero. Round count is a separate variable determining whether information surfaces: reading a proposal once is not the same as reading it twice. ## Missing Answers and Wrong Answers Every process leaves a record behind, and someone arriving six months later asks that record questions. The measure here goes one step past the previous lesson's: a question can be left unanswered, but it can also be answered wrong. If a process assumes it collected every objection, it says "there was no objection" — when in fact the objection never arose because it was never asked. What counts as which state is the model's input (**DR9**): if there is information that never surfaced, the answer to "was there an objection" is wrong, and the answer to "what information was evaluated" is missing. ```js // proposal/questions.mjs — the question set applied to the three processes; an answer can be correct, missing, or wrong import { DECISIONS, PROCESS, run } from "./process.mjs"; // The rule that determines the status of an answer is the model's input (DR9). const QUESTIONS = [ ["Y1", "who saw it", () => "correct"], ["Y2", "was there an objection", (r) => (r.missed.length ? "wrong" : "correct")], ["Y3", "why did the decision go this way", (r) => (r.missed.some((b) => b.flips) ? "wrong" : "correct")], ["Y4", "what information was evaluated", (r) => (r.missed.length ? "missing" : "correct")], ["Y5", "how many rounds did it take to finalize", (r, p) => (p.writing ? "correct" : "missing")], ]; const report = {}; for (const name of Object.keys(PROCESS)) { const count = { correct: 0, missing: 0, wrong: 0 }; const questionStatus = {}; for (const d of DECISIONS) { const r = run(d, name); for (const [code, , rule] of QUESTIONS) { const status = rule(r, PROCESS[name]); count[status] += 1; questionStatus[code] = (questionStatus[code] ?? []).concat(status); } } const runs = DECISIONS.map((d) => run(d, name)); report[name] = { count, questionStatus, personMin: runs.reduce((t, x) => t + x.personMin, 0), days: runs.reduce((t, x) => t + x.days, 0), changed: runs.filter((x) => x.flips).length, idlePersonMin: runs.filter((x) => !x.flips).reduce((t, x) => t + x.personMin, 0), idleDays: runs.filter((x) => !x.flips).reduce((t, x) => t + x.days, 0) }; } const N = DECISIONS.length * QUESTIONS.length; console.log(`question set ${QUESTIONS.length} questions x ${DECISIONS.length} decisions = ${N} answers`); for (const [name, r] of Object.entries(report)) console.log(` ${name.padEnd(19)} correct ${String(r.count.correct).padStart(2)} missing ` + `${String(r.count.missing).padStart(2)} wrong ${String(r.count.wrong).padStart(2)}`); console.log("\nnot-correct answers per question (w = wrong, m = missing):"); console.log(" ".repeat(43) + Object.keys(PROCESS).map((a) => a.padStart(19)).join("")); for (const [code, text] of QUESTIONS) console.log(` ${code} ${text.padEnd(38)}` + Object.values(report).map((r) => { const w = r.questionStatus[code].filter((d) => d === "wrong").length; const m = r.questionStatus[code].filter((d) => d === "missing").length; return (w ? `${w}w` : m ? `${m}m` : "-").padStart(19); }).join("")); console.log("\nwhat the extra cost buys:"); const names = Object.keys(PROCESS); for (let i = 1; i < names.length; i++) { const a = report[names[i - 1]], b = report[names[i]]; const extraChanged = b.changed - a.changed; console.log(` ${names[i - 1]} -> ${names[i]}: +${b.personMin - a.personMin} person-min, +${b.days - a.days} days, ` + `+${extraChanged} decisions -> per flipped decision ${((b.personMin - a.personMin) / extraChanged).toFixed(0)} person-min`); } console.log("\nspent on decisions that did not change (pure delay):"); for (const [name, r] of Object.entries(report)) console.log(` ${name.padEnd(19)} ${DECISIONS.length - r.changed} decisions, ${r.idlePersonMin} person-min, ${r.idleDays} days`); ``` ``` question set 5 questions x 8 decisions = 40 answers verbal-approval correct 13 missing 15 wrong 12 written-one-round correct 32 missing 3 wrong 5 written-two-round correct 40 missing 0 wrong 0 not-correct answers per question (w = wrong, m = missing): verbal-approval written-one-round written-two-round Y1 who saw it - - - Y2 was there an objection 7w 3w - Y3 why did the decision go this way 5w 2w - Y4 what information was evaluated 7m 3m - Y5 how many rounds did it take to finalize 8m - - what the extra cost buys: verbal-approval -> written-one-round: +1320 person-min, +22 days, +3 decisions -> per flipped decision 440 person-min written-one-round -> written-two-round: +840 person-min, +14 days, +2 decisions -> per flipped decision 420 person-min spent on decisions that did not change (pure delay): verbal-approval 8 decisions, 480 person-min, 0 days written-one-round 5 decisions, 900 person-min, 10 days written-two-round 3 decisions, 900 person-min, 12 days ``` The distribution of the forty answers gives 13, 32, and 40 correct across the three processes. But the real number is not in the correct column — it is in the wrong one. Verbal approval answers twelve of the forty questions wrong; fifteen questions get no answer at all. The difference matters: an unanswered question keeps getting asked, a wrongly answered question closes. The sentence "no one objected to this decision" looks true because the objection was never asked for, and once that sentence enters the record, no one asks again. The per-question table shows where the wrong answers pile up. Y1 is correct in all three processes: who took part is known under every scheme, because participation is the process itself. Y2 and Y3 break down together with the information that never surfaced — seven and five decisions in verbal approval, three and two in the one-round written scheme. Y4 and Y5 produce missing, not wrong: which information was evaluated and how many rounds it took to finalize stay unanswered when no record was kept, but they are not invented. ## The Cost Paid and Pure Delay The cost of the written process is in the output's last two sections. Moving from verbal approval to the one-round written scheme adds 1320 person-minutes and 22 calendar days, and in return flips three decisions: 440 person-minutes per flipped decision. Making a second round mandatory adds another 840 person-minutes and 14 days and flips two more decisions: 420 person-minutes. The unit cost of the two transitions comes out nearly the same, meaning the second round does not have diminishing returns relative to the first. In this model, the return on adding rounds is at the same level as the return on adding roles. The last table is the other side of the coin. In the one-round scheme, five decisions never changed direction, and those five decisions ate 900 person-minutes and 10 calendar days. In the two-round scheme, three decisions did not change, again 900 person-minutes and 12 days. This is the cost of the decisions the process delays but does not change, and it is the measured form of the objection raised against written process. The objection is right: a substantial share of the time spent flips no decision at all. The answer to that objection is this: which decision will flip is known only after the process runs. The notification-channel decision looked like a three-line matter and flipped; the search-index decision looked contested and came out the same after two rounds. Pure delay is the price of not being able to know in advance which decision will flip. What is measurable is not eliminating the delay — it is knowing the person-minutes per flipped decision. ## Summary - Verbal approval flipped none of the eight decisions; the reason is not that the decisions were correct — it is that the five pieces of information capable of flipping a decision were not in a two-person room. Ten pieces of information were never seen. - The written-proposal scheme flipped three of the same decisions; the scheme running at least two rounds flipped five and dropped the missed information to zero — round count is a variable independent of role count. - The question set gave 13, 32, and 40 correct answers; the real distinction is in the wrong answers: verbal approval answered twelve questions wrong and fifteen missing, and a wrong answer closes the question. - Wrong answers piled up on "was there an objection" and "why did the decision go this way"; fields with no record produced missing, not wrong, answers. - The extra cost's return sits at the same level across both transitions: 440 and 420 person-minutes per flipped decision; against that, in the one-round scheme, five decisions spent 900 person-minutes and 10 days without ever changing direction. ## Next Step When an evaluation round flips a decision, what remains is the information that it "changed" — not what was traded for what. The catalog-upgrade decision flipped because of four fields in the mapping layer; in the record, that is a one-line rationale. But behind that decision stood multiple alternatives, multiple qualities, and every alternative's standing on every quality. The previous course measured how that table gets built and how the weights determine the winner. What is left open is whether the table itself makes it into the record: can a third person, looking at the table in the record, arrive at the same result. The next lesson measures that.