Quality Attribute Scenarios

The previous lessons split views and diagram types by question: which picture answers which is now clear. But part of what someone who comes later asks cannot be answered by a picture. Questions like “is this system fast enough,” “what happens when this branch goes offline,” “is this slowdown acceptable” are not closed by a box-and-arrow picture; their answer is a number. This lesson measures where that number gets written, and what happens when it is not.

What a quality attribute is and which fields a quality attribute scenario has were defined in the Quality and Testing Fundamentals course; that definition is not repeated here. The question here is narrow: given a requirement list, how many of its items are testable. Testability is not an opinion but a criterion — a requirement is testable if a test procedure can be written from it, and a procedure needs a stimulus to trigger, an environment to set up, a response to observe, and a threshold to compare against. Which quality attributes matter for a given system is separate, taken up in the course that follows this one; no attribute list is derived here.

The Requirement Set to Measure

The block below models twelve quality requirements in two forms: the plain sentence and the fielded scenario. The fielded form carries only what the plain sentence says — no information that is not in the sentence gets written into a field. The model is also written to disk; the next two blocks read the same set.

// set.mjs — models the requirement set in two forms and measures testability
import { mkdirSync, writeFileSync } from "node:fs";

// VM16: twelve quality requirements collected from the regional library network. The
// fielded form carries exactly what the plain sentence says; no field is invented.
const REQUIREMENT = [
  ["Catalog search must respond quickly.",
    "a member searches the catalog", "", "a result list is returned", "", ""],
  ["Lending must not stop when the branch connection drops.",
    "the connection between the branch and the center drops", "", "lending continues", "", ""],
  ["The system must stay up during peak hours.",
    "", "peak hour", "the system keeps accepting requests", "", ""],
  ["A loan record must not be lost.",
    "", "", "the record stays readable", "", ""],
  ["The overdue notification must go out on time.",
    "the loan period expires", "", "a notification is sent", "", ""],
  ["The catalog bulk upload must fit inside the nightly window.",
    "a bulk upload starts", "the nightly maintenance window", "the upload finishes", "", ""],
  ["A member must be able to see their own loan history without waiting.",
    "the member opens the history page", "", "the history is listed", "", ""],
  ["Under 500 concurrent members, the loan operation must complete within 400 ms for 95% of requests.",
    "a member sends a loan request", "500 concurrent members", "a loan record is created",
    "95% of requests under 400 ms", ""],
  ["When the central database node is lost, the loan service must fail over to the replica within 60 seconds.",
    "the primary database node stops responding", "business-hours load", "the service runs on the replica node",
    "failover within 60 seconds", ""],
  ["A new branch definition must reach production in a single deployment within 1 hour.",
    "a branch definition is added", "a single deployment window", "the branch appears in production",
    "1 hour from addition to release", "branch opening calendar: two branches a month"],
  ["The system must support backups.",
    "", "", "", "", ""],
  ["The report screen must open within a reasonable time.",
    "a manager opens the report screen", "", "the report is displayed", "", ""],
];

const FIELD = ["stimulus", "environment", "response", "measure", "source"];
const SET = REQUIREMENT.map(([plain, ...value], i) => ({
  no: i + 1, plain, field: Object.fromEntries(FIELD.map((f, j) => [f, value[j]])),
}));

mkdirSync("set", { recursive: true });
writeFileSync("set/set.mjs",
  `export const FIELD = ${JSON.stringify(FIELD)};\n` +
  `export const SET = ${JSON.stringify(SET, null, 0)};\n`);

// Testability criterion: writing a test procedure requires a stimulus to trigger, an
// environment to set up, a response to observe, and a threshold to compare against. The
// threshold is a number carrying a unit; both forms are searched with the same pattern.
const THRESHOLD = /(\d+\s*(ms|seconds?|minutes?|hours?|days?)|\d+\s*%|\d+%)/;
const CONDITION = /(is lost|concurrent|deployment|window|drops|peak|load|under)/;
const FOUR = ["stimulus", "environment", "response", "measure"];

const plainTestable = (r) => THRESHOLD.test(r.plain) && CONDITION.test(r.plain);
const fieldedTestable = (r) =>
  FOUR.every((f) => r.field[f] !== "") && THRESHOLD.test(r.field.measure);

console.log(`${"no".padStart(2)}  ${"plain form".padEnd(12)}${"fielded form (filled field)".padEnd(40)}empty field`);
for (const r of SET) {
  const filled = FOUR.filter((f) => r.field[f] !== "");
  console.log(`${String(r.no).padStart(2)}  ` +
    `${(plainTestable(r) ? "testable" : "-").padEnd(12)}` +
    `${(filled.join(",") || "-").padEnd(40)}` +
    `${4 - filled.length}`);
}

const emptyFour = SET.reduce((s, r) => s + FOUR.filter((f) => r.field[f] === "").length, 0);
const emptyFive = SET.reduce((s, r) => s + FIELD.filter((f) => r.field[f] === "").length, 0);
const wordsPlain = SET.reduce((s, r) => s + r.plain.length, 0);
const wordsFielded = SET.reduce((s, r) =>
  s + FIELD.reduce((t, f) => t + f.length + 2 + r.field[f].length, 0), 0);

console.log(`\ntestable: plain form ${SET.filter(plainTestable).length}/${SET.length}, ` +
  `fielded form ${SET.filter(fieldedTestable).length}/${SET.length}`);
console.log(`empty field: in the four fields ${emptyFour}, including the fifth field (source) ${emptyFive}`);
console.log(`writing cost (characters): plain ${wordsPlain}, fielded ${wordsFielded} ` +
  `(${(wordsFielded / wordsPlain).toFixed(2)}x)`);

no  plain form  fielded form (filled field)             empty field
 1  -           stimulus,response                       2
 2  -           stimulus,response                       2
 3  -           environment,response                    2
 4  -           response                                3
 5  -           stimulus,response                       2
 6  -           stimulus,environment,response           1
 7  -           stimulus,response                       2
 8  testable    stimulus,environment,response,measure   0
 9  testable    stimulus,environment,response,measure   0
10  testable    stimulus,environment,response,measure   0
11  -           -                                       4
12  -           stimulus,response                       2

testable: plain form 3/12, fielded form 3/12
empty field: in the four fields 20, including the fifth field (source) 31
writing cost (characters): plain 706, fielded 1406 (1.99x)

What the Fielded Form Changes

In both forms, the number of testable requirements is three. This breaks the most common expectation loaded onto the fielded scenario: opening fields does not manufacture information. If the sentence carries no threshold, neither does the field; changing the form does not make a requirement testable.

What does change is this: the plain form gives one piece of information for the nine untestable requirements — no number. The fielded form gives, for each one, which field is empty. The fourth requirement is missing three fields, the eleventh is empty in all four, the sixth is missing only the measure. This separates the size and the kind of the gap; neither shows up in the plain sentence.

The empty-field count is twenty, and that number is directly a list of questions. The fielded form’s writing cost is 1.99 times the plain form’s: 706 characters against 1,406. That is the price; the payoff is knowing exactly where to ask each of the twenty questions.

The fifth field stands apart. source holds where the threshold came from, and it is filled in only one of the twelve requirements. The testability criterion looks at the first four fields only, because a test procedure can be written without knowing where the threshold came from. Why the fifth field still matters is measured in this lesson’s last section.

The Cost of an Unwritten Threshold

An untestable requirement does not charge a cost immediately; the cost surfaces at delivery. When a threshold is not written down, it does not disappear — it sits in each side’s head as two separate thresholds. The requester thinks of one number when they say “fast,” the developer another. The block below models this: each requirement gets a scale where smaller is better, a threshold per side, and a measured value for the delivered system.

// disagreement.mjs — reads set/set.mjs written by set.mjs; measures the disagreement an
// untestable requirement causes at delivery
import { SET } from "./set/set.mjs";

// VM17: every requirement has a scale where smaller is better. An untestable requirement
// carries two unwritten thresholds, one for the requester and one for the developer. The
// three testable requirements have a written threshold; both sides use the same number.
const THRESHOLD = [                // [scale, requester's threshold, developer's threshold, delivery range]
  ["search response time (ms)", 300, 1200, [200, 1500]],
  ["loan-halt duration (min)", 5, 120, [0, 180]],
  ["rejected requests (per mille)", 1, 20, [0, 40]],
  ["lost records per year", 0, 3, [0, 6]],
  ["notification delay (hr)", 1, 24, [0, 36]],
  ["bulk upload duration (min)", 60, 240, [30, 300]],
  ["history page (ms)", 500, 2500, [300, 3000]],
  ["loan operation p95 (ms)", 400, 400, [250, 900]],
  ["failover (sec)", 60, 60, [20, 180]],
  ["release rollout (min)", 60, 60, [20, 240]],
  ["restore from backup (hr)", 2, 24, [1, 48]],
  ["report screen (sec)", 3, 20, [1, 30]],
];

// VM18: the delivered measurement is drawn from each requirement's own range with a
// uniform distribution. The generator is hand-written; the seed is visible.
const generator = (seed) => () => {
  seed = (seed + 0x6d2b79f5) | 0;
  let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
const SEED = 20802;
const DELIVERY = 200;
const random = generator(SEED);

const result = THRESHOLD.map(() => ({ accept: 0, dispute: 0, reject: 0 }));
let disputesPerDelivery = 0;
for (let t = 0; t < DELIVERY; t += 1) {
  let n = 0;
  THRESHOLD.forEach(([, requester, developer, [low, high]], i) => {
    const measured = low + random() * (high - low);
    if (measured <= Math.min(requester, developer)) result[i].accept += 1;
    else if (measured > Math.max(requester, developer)) result[i].reject += 1;
    else { result[i].dispute += 1; n += 1; }
  });
  disputesPerDelivery += n;
}

console.log(`seed ${SEED}, ${DELIVERY} deliveries; each row is one requirement (delivery count)`);
console.log(`${"no".padStart(2)}  ${"scale".padEnd(30)}${"threshold".padStart(12)}` +
  `${"accept".padStart(8)}${"dispute".padStart(13)}${"reject".padStart(7)}`);
THRESHOLD.forEach(([scale, requester, developer], i) => {
  console.log(`${String(i + 1).padStart(2)}  ${scale.padEnd(30)}` +
    `${(requester === developer ? `${requester}` : `${requester}/${developer}`).padStart(12)}` +
    `${String(result[i].accept).padStart(8)}${String(result[i].dispute).padStart(13)}` +
    `${String(result[i].reject).padStart(7)}`);
});

const written = THRESHOLD.map(([, a, b], i) => (a === b ? i : -1)).filter((i) => i >= 0);
const unwritten = THRESHOLD.map((_, i) => i).filter((i) => !written.includes(i));
const sum = (list) => list.reduce((s, i) => s + result[i].dispute, 0);
console.log(`\n${written.length} requirements with a written threshold: ${sum(written)} disputes`);
console.log(`${unwritten.length} requirements with no written threshold: ${sum(unwritten)} disputes`);
console.log(`average disputes per delivery: ` +
  `${(disputesPerDelivery / DELIVERY).toFixed(2)} (across ${SET.length} requirements)`);

seed 20802, 200 deliveries; each row is one requirement (delivery count)
no  scale                            threshold  accept      dispute reject
 1  search response time (ms)         300/1200      12          142     46
 2  loan-halt duration (min)             5/120      10          112     78
 3  rejected requests (per mille)         1/20       3           99     98
 4  lost records per year                  0/3       0           90    110
 5  notification delay (hr)               1/24       5          117     78
 6  bulk upload duration (min)          60/240      13          148     39
 7  history page (ms)                 500/2500       8          158     34
 8  loan operation p95 (ms)                400      45            0    155
 9  failover (sec)                          60      56            0    144
10  release rollout (min)                   60      35            0    165
11  restore from backup (hr)              2/24       5           92    103
12  report screen (sec)                   3/20       9          104     87

3 requirements with a written threshold: 0 disputes
9 requirements with no written threshold: 1062 disputes
average disputes per delivery: 5.31 (across 12 requirements)

In the three requirements with a written threshold, the disagreement count is zero — a structural result of the model: with a single threshold, both sides look at the same measurement and reach the same conclusion. The eighth requirement is rejected in 155 of 200 deliveries; a failure, but an undisputed one. Rejection is a decision, not a disagreement.

The nine requirements with no written threshold produce 1,062 disagreements in 200 deliveries — 5.31 per delivery. That means every delivery, more than five of the twelve requirements get two different answers to “was it met.” The highest rate is on the seventh requirement: 158 of 200 deliveries end in disagreement. The reason shows in the scale — the gap between the two thresholds (500 to 2,500 ms) covers most of the delivery range. The probability of disagreement grows with the gap between the two sides’ implicit thresholds. The more ambiguous the sentence, the wider the gap.

That gives the lesson’s cost comparison. Writing the set in fielded form costs 700 extra characters and twenty questions asked, paid once. Leaving a requirement untestable costs 5.31 disagreements per delivery, paid again at every delivery.

The Question Set and the Fifth Field

This course’s measure is how many of the questions someone later asks a document, the document answers. Five questions get asked of quality requirements, and each lands on one field.

// question.mjs — reads set/set.mjs written by set.mjs; applies the question set to the set
import { SET } from "./set/set.mjs";

// The five questions someone who comes later asks about a quality requirement; each question lands on one field.
const QUESTION = [
  ["When is this requirement considered satisfied?", "measure"],
  ["Under what condition will it be measured?", "environment"],
  ["What will trigger the test?", "stimulus"],
  ["What will be observed from the system?", "response"],
  ["Where did this number come from?", "source"],
];

console.log(`${"question".padEnd(48)}${"answered".padStart(11)}${"unanswered".padStart(13)}`);
let answered = 0;
for (const [question, field] of QUESTION) {
  const d = SET.filter((r) => r.field[field] !== "").length;
  answered += d;
  console.log(`${question.padEnd(48)}${`${d}/${SET.length}`.padStart(11)}` +
    `${String(SET.length - d).padStart(13)}`);
}
const total = QUESTION.length * SET.length;
console.log(`total ${total} questions: ${answered} answered, ${total - answered} unanswered`);

// Number of questions to ask to make a requirement testable (the fifth field is excluded:
// testability only looks at the four fields).
const FOUR = ["measure", "environment", "stimulus", "response"];
const missing = SET.map((r) => [r.no, FOUR.filter((f) => r.field[f] === "").length,
  r.plain.slice(0, 34)]).sort((a, b) => a[1] - b[1] || a[0] - b[0]);
console.log(`\n${"no".padStart(2)}  ${"to ask".padStart(9)}  requirement`);
for (const [no, n, c] of missing) console.log(`${String(no).padStart(2)}  ${String(n).padStart(9)}  ${c}`);
console.log(`to make the set testable: ${missing.reduce((s, [, n]) => s + n, 0)} questions; ` +
  `closed by one question ${missing.filter(([, n]) => n === 1).length}, ` +
  `needing three or more ${missing.filter(([, n]) => n >= 3).length}`);

question                                           answered   unanswered
When is this requirement considered satisfied?         3/12            9
Under what condition will it be measured?              5/12            7
What will trigger the test?                            9/12            3
What will be observed from the system?                11/12            1
Where did this number come from?                       1/12           11
total 60 questions: 29 answered, 31 unanswered

no     to ask  requirement
 8          0  Under 500 concurrent members, the 
 9          0  When the central database node is 
10          0  A new branch definition must reach
 6          1  The catalog bulk upload must fit i
 1          2  Catalog search must respond quickl
 2          2  Lending must not stop when the bra
 3          2  The system must stay up during pea
 5          2  The overdue notification must go o
 7          2  A member must be able to see their
12          2  The report screen must open within
 4          3  A loan record must not be lost.
11          4  The system must support backups.
to make the set testable: 20 questions; closed by one question 1, needing three or more 2

29 of the sixty questions get answered, 31 do not. The answer rate falls steadily: what to observe from the system is known for eleven of the twelve requirements, what to trigger for nine, under what condition to measure for five, when it counts as met for only three. In other words, the requirement’s author can describe what they want, not when it is enough.

The lowest row is the fifth field: the threshold’s source is written in only one of twelve requirements. Even two of the three requirements with a written threshold do not record where the number came from. The distinction looks small but its consequence is heavy: the 400 ms threshold is testable, but six months later, when someone asks “can this threshold be raised,” it cannot be answered. In the one requirement whose source is written (the branch opening calendar, two branches a month), that question is answered: if the calendar slows, the threshold can be loosened. The measure makes the requirement testable; the threshold’s source makes it changeable.

The last table shows how the twenty questions are distributed: three are already testable, one closes with a single question, six need two each, two need three or four. The eleventh requirement is not really a requirement but a heading; with all four fields empty, it is not filled by one question but by a separate piece of work.

Summary

• Testability is a criterion: a requirement is testable if a test procedure can be written from it; the procedure needs a stimulus, an environment, a response, and a unit-bearing threshold.
• The same twelve requirements come out 3/12 testable in both forms — the fielded form does not manufacture information, it makes the missing information visible: 20 empty fields, i.e. 20 askable questions.
• The fielded form’s writing cost is 1.99x (706 versus 1,406 characters), paid once; the nine requirements with no written threshold produce 1,062 disagreements in 200 deliveries, 5.31 per delivery.
• The probability of disagreement grows with the gap between the two sides’ implicit thresholds; the three requirements with a written threshold have zero disagreements, and even its rejections are undisputed.
• 31 of 60 applications of the five questions go unanswered; the least-answered is “where did this number come from” (11/12 unanswered) — the measure enables testing, the source enables changing it.

Next Step

When the requirement set is made measurable, a document results: fielded scenarios, thresholds, sources. That document is in step with the code on the day it is written. The next lesson measures what comes after: the code keeps changing, the document stays put, and the gap grows. This gap is found by counting, not guessing: reality extracted from a module graph is compared against the document’s claim, how drift grows across a sequence of changes is tracked, and two keeping-current regimes — manual update and a generated audit — are compared by the drift caught, the maintenance cost imposed, and the false alarms produced.