---
title: 'Quality and Testing Fundamentals'
source: 'https://academia.sh/en/courses/testing-fundamentals'
language: en
updated: '2026-08-23T09:28:37+00:00'
license: 'CC BY-SA 4.0'
lessons: 12
outcomes:
    - 'Separates the concepts of verification and validation'
    - 'Classifies test levels and test types'
    - 'Writes a good test case and defect report'
---

# Quality and Testing Fundamentals

Separating process quality from product quality, the difference between verification and validation, turning non-functional expectations into measurable criteria, test levels and types, the equivalence class and boundary value techniques, writing test cases, where the expected result comes from, and the error life cycle.

## The Quality Concept

Quality assurance's process and product faces, the distinction between verification and validation, turning non-functional expectations into measurable criteria, the evidence-seeking testing mindset, and the delivery model's effect on feedback time.

### What Is Quality Assurance

Source: https://academia.sh/en/courses/testing-fundamentals/what-is-quality-assurance

The definition of quality as conformance to stated and implied expectations, measuring product quality and process quality separately, and the defect that an unchecked rule hides.

A program can be running and still be wrong. It can pass its tests and still do something
nobody wanted. It can work correctly and still be unusable because it takes ten seconds to
respond. What these three sentences share is that the word "working" carries no judgment by
itself. This curriculum builds that judgment: how do you decide that a piece of software is
the thing that was wanted, and what evidence does that decision rest on?

The Testing and Monitoring topic in the Frontend Quality course introduced unit, component,
integration, and end-to-end tests in the browser context. There, testing was a quality tool
for one particular domain — the interface running in the browser. Here the topic reverses:
the discipline of testing itself is treated independently of any domain. Examples are written
with tools set up in the JavaScript and TypeScript course and the Node.js Runtime course, but
the questions do not depend on language or platform.

#### Quality Is a Judgment of Conformance

Quality is the degree to which a product satisfies its **stated** and **implied**
expectations. Both halves of this definition are required.

A stated expectation is written down: "student members borrow a book for 28 days." An
implied expectation is not written down, but everyone notices when it is violated: a
member's debt should not be visible to another member, a return should not be processed
twice, a page should not take minutes to load. However carefully the specification is
written, it never covers every implied expectation.

Quality is therefore not an absolute property but a **judgment of conformance**: it cannot
be measured until the question "conformance to what" has an answer. The domain used
throughout the course is a small lending library. This domain is the same one used in the
Backend Development curriculum's library system; no code from that system is assumed here —
the rules are built from scratch, and the same library is developed across the lessons.

```js
// library.mjs — loan rules
export const RULES = {
  student: { loanDays: 28, maxLoans: 10 },
  member: { loanDays: 14, maxLoans: 5 },
};

export function loanPeriod(memberType) {
  const rule = RULES[memberType];
  if (rule === undefined) throw new Error(`unknown member type: ${memberType}`);
  return rule.loanDays;
}

export function maxLoans(memberType) {
  return RULES[memberType]?.maxLoans ?? 5;
}

export function dueDate(checkoutDay, memberType) {
  return checkoutDay + loanPeriod(memberType);
}
```

Dates are represented as day numbers: `dueDate(1000, 'member')` says that a book checked out
on day one thousand is due back on day one thousand and fourteen. Calendar arithmetic and
time zone issues are not the subject of this quality discussion; a fixed day counter keeps
the examples deterministic.

#### Product Quality and Process Quality

**Product quality** is a property of the thing in hand: does the library apply its rules
correctly, is it fast, is it secure? **Process quality** is a property of the way of working
that produced that thing: is every rule tied to a check, are changes reviewed, are defects
recorded along with where they came from?

The link between them is one-directional and probabilistic. A good process does not
guarantee a good product, but a bad process makes a good product **unrepeatable**: one
release turns out well, the next does not, and there is no way to know where the difference
came from. This is precisely what quality assurance is concerned with.

Measuring the two properties separately first requires a machine-readable list of the rules.

```js
// specification.mjs — machine-readable list of the written rules
export const SPECIFICATION = [
  { id: 'K1', text: 'Student loan period is 28 days' },
  { id: 'K2', text: 'Member loan period is 14 days' },
  { id: 'K3', text: 'A student may borrow at most 10 books' },
  { id: 'K4', text: 'A member may borrow at most 5 books' },
  { id: 'K5', text: 'An unknown member type is rejected' },
  { id: 'K6', text: 'The due date is the checkout day plus the loan period' },
];
```

#### Separating the Two Measurements

Product quality measurement answers the question "how many of the written checks pass?"
Process quality measurement asks an entirely different question: "how many of the rules in
the specification have a check?" The second question is called **traceability**, and its
answer looks not at the product itself but at the link between the product and the
specification.

```js
// check.mjs — two separate measurements: product quality and process quality
import assert from 'node:assert/strict';
import { SPECIFICATION } from './specification.mjs';
import { loanPeriod, maxLoans, dueDate } from './library.mjs';

const CHECKS = {
  K1: () => assert.equal(loanPeriod('student'), 28),
  K2: () => assert.equal(loanPeriod('member'), 14),
  K3: () => assert.equal(maxLoans('student'), 10),
  K6: () => assert.equal(dueDate(1000, 'member'), 1014),
};

let passed = 0;
const failed = [];
for (const [id, check] of Object.entries(CHECKS)) {
  try {
    check();
    passed += 1;
  } catch {
    failed.push(id);
  }
}

const unchecked = SPECIFICATION.filter((k) => CHECKS[k.id] === undefined).map((k) => k.id);

console.log(`product quality: ${passed}/${passed + failed.length} checks passed`);
console.log(`process quality: ${SPECIFICATION.length - unchecked.length} of ${SPECIFICATION.length} rules checked`);
console.log(`unchecked rules: ${unchecked.join(', ')}`);
```

```
product quality: 4/4 checks passed
process quality: 4 of 6 rules checked
unchecked rules: K4, K5
```

The first line looks flawless. The second line says what the first line does not cover.
Product quality measurement only reports on **questions that were asked**; questions that
were never asked do not appear inside that measurement. Process quality measurement names
exactly that gap.

#### The Cost of a Rule Left Unchecked

What happens once the two missing checks are written becomes visible.

```js
// check-full.mjs — both previously unchecked rules are now written
import assert from 'node:assert/strict';
import { loanPeriod, maxLoans, dueDate } from './library.mjs';

const CHECKS = {
  K1: () => assert.equal(loanPeriod('student'), 28),
  K2: () => assert.equal(loanPeriod('member'), 14),
  K3: () => assert.equal(maxLoans('student'), 10),
  K4: () => assert.equal(maxLoans('member'), 5),
  K5: () => {
    assert.throws(() => loanPeriod('guest'));
    assert.throws(() => maxLoans('guest'));
  },
  K6: () => assert.equal(dueDate(1000, 'member'), 1014),
};

for (const [id, check] of Object.entries(CHECKS)) {
  try {
    check();
    console.log(`${id} passed`);
  } catch (error) {
    console.log(`${id} failed: ${error.message.split('\n')[0]}`);
  }
}
```

```
K1 passed
K2 passed
K3 passed
K4 passed
K5 failed: Missing expected exception.
K6 passed
```

The defect is in the `maxLoans` function: instead of raising an error for an unknown member
type, it silently returns five. An unregistered person can borrow a book, and the system
mistakes it for member behavior. The defect was always there; product quality measurement
could not see it, because no one had asked that question.

The lesson here is more general than the fact that the fourth line failed. Product measurement
only speaks within its own scope; what tells you where that scope ends is process
measurement. The sentence "all tests pass" is not a quality claim unless you know which
questions were never asked.

#### Assurance, Control, and Testing

The three terms are often used interchangeably; their scopes differ.

**Quality assurance** is process-oriented and preventive. Having rules in writing, tying
every rule to a check, reviewing changes, and recording defects all fall under this heading.
Its question is: what prevents a defect from arising?

**Quality control** is product-oriented and detective. It looks at whether the release in
hand meets expectations. Its question is: does this release have a defect?

**Testing** is the primary method of quality control, but not the only one. Review, formal
analysis, and static inspection also find defects. Testing is distinguished by producing
evidence by **running** the program: it compares observed behavior against expected
behavior.

The relationship among the three is hierarchical: testing is a tool of quality control;
quality control is a part of quality assurance. This course is primarily about testing, but
the question of what testing can and cannot prove is answered at the assurance level.

What testing cannot prove also has a name: a test can only show the **presence** of a
defect, not its absence. A finite set of inputs cannot exhaust an infinite input space.
Testing's job is therefore not "proving flawlessness" but "choosing the inputs most likely
to find a defect." Every technique in test design is an answer to this selection problem.

#### Summary

- Quality is the degree of conformance to stated and implied expectations; it cannot be
  measured until the question of conformance to what is answered.
- Product quality is a property of the thing in hand, process quality of the way of working
  that produced it; a good process does not guarantee a good product, but a bad process
  makes a good product unrepeatable.
- Traceability measurement asks "how many of the rules have a check"; product measurement
  only reports on questions that were asked.
- In the example, all four written checks passed; once one of the two unchecked rules was
  written, a defect surfaced in which an unknown member type was silently accepted.
- Quality assurance is preventive and process-oriented, quality control is detective and
  product-oriented; testing is quality control's method of producing evidence by running the
  program.
- Testing can show the presence of a defect, not its absence.

#### Next Step

In this lesson, the checks were derived from the specification: whatever the written rule
said, the check asked exactly that. This is the question "does the library conform to the
specification?" But what if the specification itself is wrong? The rules can be fully
implemented, every check can be green, and the result can still be wrong — because the
written rule is not the rule the library actually needs to enforce. The next lesson separates
these two questions: building it right and building the right thing.

### Verification and Validation

Source: https://academia.sh/en/courses/testing-fundamentals/verification-and-validation

Separating verification, which tests the product's conformance to the specification, from validation, which tests the specification's conformance to the real need, shown through an example that passes every check yet computes the wrong fee.

The previous lesson derived checks from the specification: whatever the written rule said,
the check asked exactly that. When every check passes, the conclusion is: the library
conforms to the written rule. That conclusion is narrower than it sounds. The written rule
itself might not be the rule the library actually needs to enforce. In that case the program
does the wrong thing flawlessly.

This lesson separates those two questions. The first asks whether the product conforms to
the specification; the second asks whether the specification conforms to the need. Adding a
new rule to the same library will show that the two are independent: the late fee
calculation.

#### Two Separate Questions

**Verification** asks: are we building the product right? Its criterion sits one step behind
the product — the specification, a design decision, an acceptance criterion. Producing the
answer means comparing the product against that written criterion.

**Validation** asks: are we building the right thing? Its criterion sits one step behind the
specification itself — the user's work, the organization's rule, the service's purpose.
Producing the answer means comparing the specification against real usage.

The difference between the two questions is where the criterion comes from. Verification's
criterion is a document; validation's criterion is the world outside that document. This is
why work that only verifies is self-contained: it confirms whatever the document says and
never raises the possibility that the document itself is wrong.

#### A Wrong Program That Conforms to the Specification

A late fee rule is added to the library. The item written during analysis is this:

- **K7** — The late fee is 2 units per day.
- **K8** — Negative days late is rejected.

The implementation follows the item exactly.

```js
// fee.mjs — literal implementation of the K7 rule from the specification
export const DAILY_RATE = 2;

export function lateFee(daysLate) {
  if (daysLate < 0) throw new Error('days late cannot be negative');
  return daysLate * DAILY_RATE;
}
```

The checks are also derived from the same item. The same arrangement as the previous lesson
is used: every check is tied to a rule number.

```js
// verification.mjs — checks derived from the written specification
import assert from 'node:assert/strict';
import { lateFee } from './fee.mjs';

const CHECKS = {
  'K7 2 units per day': () => assert.equal(lateFee(5), 10),
  'K7 no days late, no fee': () => assert.equal(lateFee(0), 0),
  'K7 fee is directly proportional to days late': () => {
    assert.equal(lateFee(10), 2 * lateFee(5));
  },
  'K8 negative days late is rejected': () => assert.throws(() => lateFee(-1)),
};

let passed = 0;
for (const [name, check] of Object.entries(CHECKS)) {
  try {
    check();
    passed += 1;
  } catch (error) {
    console.log(`failed: ${name} — ${error.message.split('\n')[0]}`);
  }
}
console.log(`verification: ${passed}/${Object.keys(CHECKS).length} checks passed`);
```

```
verification: 4/4 checks passed
```

Verification is complete. The rule consisted of two items, both items were checked, and both
passed. The traceability measurement also comes out flawless: every written rule has a
check. At this point what's in hand is a program proven to fully conform to its
specification.

#### Testing Against Reality

The library's front desk had been collecting fees for years. This is validation's
criterion: not the document, but transactions that actually happened. The program recomputes
those transactions and compares the result against the amount that was collected.

```js
// validation.mjs — comparison against fees actually collected at the desk
import { lateFee } from './fee.mjs';

const DESK_RECORDS = [
  { transaction: 'A-101', daysLate: 0, collected: 0 },
  { transaction: 'A-102', daysLate: 2, collected: 0 },
  { transaction: 'A-103', daysLate: 3, collected: 0 },
  { transaction: 'A-104', daysLate: 4, collected: 2 },
  { transaction: 'A-105', daysLate: 10, collected: 14 },
  { transaction: 'A-106', daysLate: 40, collected: 20 },
];

let matched = 0;
for (const record of DESK_RECORDS) {
  const computed = lateFee(record.daysLate);
  if (computed === record.collected) matched += 1;
  else console.log(`${record.transaction}: ${record.daysLate} days — program ${computed}, desk ${record.collected}`);
}
console.log(`validation: ${matched}/${DESK_RECORDS.length} records matched`);
```

```
A-102: 2 days — program 4, desk 0
A-103: 3 days — program 6, desk 0
A-104: 4 days — program 8, desk 2
A-105: 10 days — program 20, desk 14
A-106: 40 days — program 80, desk 20
validation: 1/6 records matched
```

One of the six records matches. The matching record is the transaction with no days late —
the case where the fee calculation never runs at all. The pattern in the five mismatched
records gives away two missing rules: the first three days are free, and the fee stops at an
upper limit. Both rules were being applied at the desk, but neither had made it into the
specification.

The defect is not in the code. The code correctly does what was asked of it. The defect is
in the analysis stage, and no amount of verification can find it — because verification
uses, as its criterion, exactly the flawed document.

#### The Corrected Specification

Validation's output is not a code fix but a specification fix. K7 is split into three items:
the **grace period** is three days, every day after that costs 2 units, and the fee never
exceeds 20 units.

```js
// fee2.mjs — specification corrected after validation
export const GRACE_DAYS = 3;
export const DAILY_RATE = 2;
export const FEE_CAP = 20;

export function lateFee(daysLate) {
  if (daysLate < 0) throw new Error('days late cannot be negative');
  const billable = Math.max(daysLate - GRACE_DAYS, 0);
  return Math.min(billable * DAILY_RATE, FEE_CAP);
}
```

Both checks can now be applied at the same time. Verification is derived from the new items;
validation uses the same desk records.

```js
// two-checks.mjs — the corrected specification is both verified and validated
import assert from 'node:assert/strict';
import { lateFee } from './fee2.mjs';

const CHECKS = {
  'K7a first 3 days are free': () => assert.equal(lateFee(3), 0),
  'K7b 2 units for each day after': () => assert.equal(lateFee(6), 6),
  'K7c fee never exceeds 20 units': () => assert.equal(lateFee(100), 20),
  'K8 negative days late is rejected': () => assert.throws(() => lateFee(-1)),
};

let passed = 0;
for (const [name, check] of Object.entries(CHECKS)) {
  try { check(); passed += 1; } catch { console.log(`failed: ${name}`); }
}

const DESK_RECORDS = [
  { daysLate: 0, collected: 0 }, { daysLate: 2, collected: 0 },
  { daysLate: 3, collected: 0 }, { daysLate: 4, collected: 2 },
  { daysLate: 10, collected: 14 }, { daysLate: 40, collected: 20 },
];
const matched = DESK_RECORDS.filter((r) => lateFee(r.daysLate) === r.collected).length;

console.log(`verification: ${passed}/${Object.keys(CHECKS).length} checks passed`);
console.log(`validation  : ${matched}/${DESK_RECORDS.length} records matched`);
```

```
verification: 4/4 checks passed
validation  : 6/6 records matched
```

In the first version, the first line was also full. What distinguishes the two versions is
the second line. When a quality claim is made, you should ask which line it shows.

#### Every Stage Has Its Own Test

Verification is not a single activity; every intermediate artifact has its own
verification. The design is verified against the specification, the code against the
design, the configuration against the code. This pairing of development stages with testing
stages is known as the **V-model**: every decomposition step on the left has a corresponding
integration and testing step on the right. The model is debatable as a process proposal, but
the pairing it establishes is durable — every decision level gets its own criterion and its
own test.

Verification's methods do not require running the program. **Review**, **static analysis**,
and type checking also look for deviation from the specification in the product. Testing is
the kind of verification that runs the program.

Validation's methods, on the other hand, require the person who will use the product, or
that person's work: acceptance testing, comparison against real transaction records, user
trials, **specification by example** — writing agreed-upon concrete examples in place of the
rule's abstract sentence. The desk records above are the raw form of this last method.

The cost of validation arriving late is bigger than verification's. Wrong code gets fixed; a
design, data schema, interface, and training material all built on a wrong specification get
redone in their entirety. In the fee calculation example, only a single function changed, but
if the amounts already collected had been wrong, a retroactive correction would also have
been needed. This is why the validation question is asked while the specification is being
written — agreeing on a few concrete examples is a small fraction of the correction that
would otherwise come later.

#### Summary

- Verification asks whether the product conforms to the specification, validation whether
  the specification conforms to the real need; the two criteria come from different places.
- Work that only verifies is self-contained: because it takes the document as its criterion,
  it never raises the possibility that the document is wrong.
- In the example, all four checks passed, but only one of six desk records matched; the
  defect was in the analysis stage, not the code.
- Validation's output is most often a specification fix, not a code fix.
- Verification can also be done through review and static analysis; validation requires the
  work of the person who will use the product.
- A wrong specification costs more than wrong code, because it takes everything built on top
  of it down with it.

#### Next Step

This lesson discussed the fee calculation producing the right result. The right result is
only one part of the expectations. Even if the library computes the correct fee, it is
unusable if answering takes a minute, if it does not fit in memory at ten thousand records,
or if it shows another member's debt. These expectations are most often not written into the
specification, and when they are, they appear as unmeasurable sentences like "make it fast."
The next lesson names these expectations and turns them into measurable criteria.

### Quality Attributes

Source: https://academia.sh/en/courses/testing-fundamentals/quality-attributes

Separating functional expectations from quality attributes, classifying attribute families, and turning an unmeasurable expectation into a measurable criterion in stimulus-environment-response-measure form.

The previous lesson was about the fee calculation producing the right result. The right
result is only one part of the expectations. The library is unusable if it computes the
correct fee but takes minutes to produce the report, if it shows one member's debt to
another member, or if a single corrupted record makes it stop working entirely. These
expectations are most often not written into the specification, and when they are, they
appear as untestable sentences like "make it fast," "make it robust."

This lesson names those sentences and makes them testable. The criterion for testability is
singular: an expectation is testable if it can be turned into a criterion whose pass or fail
can be checked.

#### Functional Expectation and Quality Attribute

A **functional requirement** states **what** the system does: a book four days late incurs a
2-unit fee, an unknown member type is rejected. It is answered as true or false; there is no
value in between.

A **quality attribute** states **how** the system does that work: how fast, how robust, how
secure, how modifiable. Its answer comes in degrees and only gains meaning together with a
threshold. This is why, for attributes, what's discussed is not correctness but a **budget**.

The distinction is not merely a classification exercise; the two kinds of expectations are
tested differently. A functional expectation can be tested with a single input–output pair.
An attribute can only be tested with a measurement setup, a load, and a threshold.

#### Attribute Families

Attributes cannot be discussed until they are named. The commonly used families and what
they mean in the library's context are these:

| Attribute family | Library question |
|---|---|
| functional suitability | Does the fee calculation conform to the rule? |
| performance efficiency | How long does the report take to produce? |
| reliability | What happens when a corrupted record arrives? |
| usability | In how many steps does staff see the debt? |
| security | Is one member's debt visible to another? |
| maintainability | From how many places does the fee rule change? |
| portability | Does it install on a different runtime? |
| compatibility | Can the old record format be read? |

The list is not closed; every domain has its own additions. What matters is asking which
family an expectation belongs to — because the family determines how the measure gets built.

#### Turning an Unmeasurable Expectation into a Criterion

The sentence "make the report fast" cannot be tested, because the questions of how fast,
under what load, in what environment have no answer. The form that fills this gap has four
parts and is known as a **quality attribute scenario**:

- **Stimulus**: staff requests the overdue-member report.
- **Environment**: the library has 8,000 members and 80,000 loan records, and the system is
  running under ordinary load.
- **Response**: the report produces each member's total late-fee debt.
- **Response measure**: the median of five runs does not exceed 100 milliseconds.

The fourth part is what makes the expectation testable. The budget concept set up in the
Performance Budget lesson of the Frontend Quality course is used here in its
domain-independent form: a criterion is written together with a number, a load, and a
measurement method.

#### Measuring the Criterion

Once a criterion is written, it can be measured. The report is written in two forms: a
version that scans all loan records for every member, and a version that walks the records
once and totals them by member number.

```js
// report.mjs — two implementations of the overdue-member report
export function generateData(memberCount, loanCount) {
  const members = Array.from({ length: memberCount }, (_, i) => ({ no: i, type: i % 3 === 0 ? 'student' : 'member' }));
  const loans = Array.from({ length: loanCount }, (_, i) => ({
    memberNo: (i * 7919) % memberCount,
    daysLate: i % 47,
  }));
  return { members, loans };
}

function fee(daysLate) {
  return Math.min(Math.max(daysLate - 3, 0) * 2, 20);
}

export function scanningReport(members, loans) {
  return members.map((m) => {
    let debt = 0;
    for (const l of loans) if (l.memberNo === m.no) debt += fee(l.daysLate);
    return { no: m.no, debt };
  });
}

export function indexedReport(members, loans) {
  const debts = new Map();
  for (const l of loans) debts.set(l.memberNo, (debts.get(l.memberNo) ?? 0) + fee(l.daysLate));
  return members.map((m) => ({ no: m.no, debt: debts.get(m.no) ?? 0 }));
}
```

Both versions satisfy the same functional requirement; the reports they produce are
identical. Where they differ is the attribute criterion.

```js
// measurement.mjs — measuring the response-time criterion
import { generateData, scanningReport, indexedReport } from './report.mjs';

const BUDGET_MS = 100;
const { members, loans } = generateData(8000, 80000);

function medianTime(fn, runs = 5) {
  const times = [];
  for (let i = 0; i < runs; i += 1) {
    const start = performance.now();
    fn(members, loans);
    times.push(performance.now() - start);
  }
  times.sort((a, b) => a - b);
  return times[Math.floor(runs / 2)];
}

for (const [name, fn] of [['scanning', scanningReport], ['indexed', indexedReport]]) {
  const median = medianTime(fn);
  const verdict = median < BUDGET_MS ? 'passed' : 'missed';
  console.log(`${name.padEnd(9)}: ${BUDGET_MS} ms budget — ${verdict}`);
}
```

```
scanning : 100 ms budget — missed
indexed  : 100 ms budget — passed
```

The output writes the verdict, not the raw time; raw time is machine-dependent and does not
carry over from one lesson to another. On the machine used while preparing this text, the
scanning version's median measured approximately 560 ms and the indexed version's
approximately 3 ms — both numbers vary with the hardware and runtime version they run on.
What does not change is that the criterion separates the two versions: once the budget is
set, the answer to "is it fast" stops being a matter of debate.

Basing the measurement on the median of five runs rather than a single run is deliberate.
The first run carries the runtime's warm-up effect; the median leaves that spike out.

#### A Second Attribute: Reliability

Performance is not the only attribute, and every attribute demands its own measure. A
reliability scenario is written like this: when a portion of the records arrive corrupted,
the report does not stop — it skips the corrupted record and reports the skipped ratio; if
the ratio exceeds five percent, the report is considered invalid.

```js
// reliability.mjs — measuring the reliability criterion
import { generateData } from './report.mjs';

const THRESHOLD = 0.05;
const { loans } = generateData(8000, 80000);
for (let i = 0; i < loans.length; i += 33) loans[i].daysLate = -1;

let processed = 0;
let skipped = 0;
for (const l of loans) {
  if (!Number.isInteger(l.daysLate) || l.daysLate < 0) skipped += 1;
  else processed += 1;
}

const ratio = skipped / loans.length;
console.log(`processed loans: ${processed}`);
console.log(`skipped loans  : ${skipped}`);
console.log(`skipped ratio  : ${(ratio * 100).toFixed(2)} (threshold ${THRESHOLD * 100})`);
console.log(`report status  : ${ratio <= THRESHOLD ? 'valid' : 'invalid'}`);
```

```
processed loans: 77575
skipped loans  : 2425
skipped ratio  : 3.03 (threshold 5)
report status  : valid
```

Because this measurement does not involve time, it gives the same result on every machine.
Not every attribute criterion is time-dependent: ratio, count, size, and step count are also
criteria, and because they are deterministic, testing them is cheaper.

#### Attributes Compete with Each Other

The indexed version cut the time down, but it did so by building a map holding as many
entries as there are members. Memory consumption grew. The version that skips corrupted
records kept the report running, but it silently produced an incomplete total; continuity
was bought at the cost of accuracy.

This is a general condition: attributes cannot be improved independently of one another.
Performance and memory, security and usability, flexibility and simplicity most often pull
in opposite directions. This is why attribute decisions are **trade-offs** and must be
written down along with their rationale: which attribute was given up, and how much, for the
sake of which other attribute? This is where writing a budget shows its real function. The
budget settles the trade-off through a single discussion; every measurement after that is
made against that decision.

#### Summary

- A functional expectation states what the system does, a quality attribute states how it
  does it; the first is answered true-false, the second in degrees.
- An attribute expectation becomes testable once it is written as stimulus, environment,
  response, and response measure; an expectation without a criterion cannot be tested.
- Two implementations satisfying the same function can give different results against the
  same budget; in the example the scanning version missed the 100 ms budget and the indexed
  version passed it.
- If a measurement involves time, the output should write the verdict, not the raw time; raw
  time is machine-dependent.
- Ratio, count, and size criteria are deterministic and cost less to test than time
  criteria.
- Attributes compete with each other; the budget settles the trade-off once and gives
  subsequent measurements a shared criterion.

#### Next Step

In both lessons, the defect surfaced only because someone went looking for it: because the
desk records were compared, because the budget was measured. If no one had made that
comparison, neither defect would have appeared. This raises a question — what does the
person looking for the defect do differently? The next lesson treats testing as a mindset
before it is a set of techniques: a view trying to show that the program works and a view
searching for where it breaks look at the same code and find different things.

### The Testing Mindset

Source: https://academia.sh/en/courses/testing-fundamentals/testing-mindset

The view trying to show that a program works and the view searching for where it breaks giving different results on the same code, confirmation bias, and the diminishing returns of repeated questions.

In the previous two lessons, the defect surfaced only because someone went looking for it:
because the desk records were compared, because the budget was measured. If no one had made
that comparison, neither defect would have appeared. Testing's techniques come after this;
first comes the question of what the person looking is looking for.

This lesson compares two views on the same function. The first wants to show that the
program works, the second searches for where it breaks. Both write tests, both are honest,
and what they find is different.

#### Two Views Look at the Same Code

A loan decision is added to the library. The written rule has three items: no loan is given
once active loans reach the member type's limit, no loan is given to a member whose debt has
reached 20 units, and an unknown member type is rejected.

```js
// loan.mjs — first implementation of the loan decision
const RULES = { student: { maxLoans: 10 }, member: { maxLoans: 5 } };
export const DEBT_LIMIT = 20;

export function canBorrow(memberType, activeLoans, debt) {
  const limit = RULES[memberType]?.maxLoans ?? 5;
  if (activeLoans > limit) return false;
  if (debt > DEBT_LIMIT) return false;
  return true;
}
```

The loop that was repeated across the previous lessons for running checks is moved into its
own file.

```js
// runner.mjs — shared driver that runs a list of checks
export function run(label, checks) {
  let passed = 0;
  const failed = [];
  for (const [name, check] of Object.entries(checks)) {
    try { check(); passed += 1; } catch { failed.push(name); }
  }
  console.log(`${label}: ${passed}/${Object.keys(checks).length} passed`);
  for (const name of failed) console.log(`  failed: ${name}`);
}
```

Two check suites are written. The first shows that the rule works as described: a student
with three books borrows, a member with ten books cannot. The second probes the edges of the
same rule.

```js
// two-views.mjs — comparing happy-path checks with questioning checks
import assert from 'node:assert/strict';
import { canBorrow } from './loan.mjs';
import { run } from './runner.mjs';

const HAPPY_PATH = {
  'student with three books borrows a new book': () => assert.equal(canBorrow('student', 3, 0), true),
  'member with one book borrows a new book': () => assert.equal(canBorrow('member', 1, 0), true),
  'student with twenty books cannot borrow': () => assert.equal(canBorrow('student', 20, 0), false),
  'member with ten books cannot borrow': () => assert.equal(canBorrow('member', 10, 0), false),
  'member with a debt of fifty cannot borrow': () => assert.equal(canBorrow('member', 0, 50), false),
};

const QUESTIONING = {
  'member exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 5, 0), false),
  'student exactly at the limit cannot borrow': () => assert.equal(canBorrow('student', 10, 0), false),
  'debt exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 0, 20), false),
  'unknown member type is rejected': () => assert.throws(() => canBorrow('guest', 0, 0)),
  'negative loan count is rejected': () => assert.throws(() => canBorrow('member', -1, 0)),
};

run('happy path ', HAPPY_PATH);
run('questioning', QUESTIONING);
```

```
happy path : 5/5 passed
questioning: 0/5 passed
  failed: member exactly at the limit cannot borrow
  failed: student exactly at the limit cannot borrow
  failed: debt exactly at the limit cannot borrow
  failed: unknown member type is rejected
  failed: negative loan count is rejected
```

The first suite paints a flawless picture. The second suite shows five separate defects in
the same ten-line function: two comparison operators have shifted the limit by one unit, the
debt limit carries the same shift, an unknown member type is silently counted as a member,
and an invalid number is never checked at all.

All of the defects fell outside the first suite's scope. The reason is not that the first
suite was poorly written; every question it asked was chosen from points far from the
boundary.

#### Confirmation Bias

The person who wrote the first suite read the rule and chose examples that show the rule
working. This is what **confirmation bias** looks like in test writing: searching for
evidence consistent with a claim instead of testing the claim.

The source of the bias is how the question is framed. The question "can a student borrow a
book?" looks for an example that can. The question "under what condition can a student not
borrow, and where is the boundary of that condition?" requires probing both sides of the
boundary. The second question takes more work and finds more defects.

This is where the mindset is defined: the person writing tests does not assume the program
is correct; they look for evidence that would test the claim that it is. If they do not find
any, the claim stands for the time being.

#### Where Questioning Questions Come From

The questioning checks were not generated at random. Each came from a question class:

- **Boundary**: if a threshold is crossed, the threshold itself, one below it, and one above
  it are tried. Three of the five defects came from this class.
- **Gap**: what happens if a value is missing, not found, or could not be computed? The
  unknown member type belongs to this class.
- **Type and domain**: what happens if a negative number, a fractional number, or a
  non-numeric value arrives where a number is expected?
- **Order**: what if operations arrive in an unexpected sequence? Lending out a book that has
  not been returned yet belongs to this class.
- **Repetition**: if the same operation arrives twice, does the total double?
- **Scale**: if the input grows a thousandfold, does the behavior stay the same?
- **Authorization**: does the person performing the operation have the right to do so?

This list is used as a **checklist**. A checklist's function is not to find the defect but
to remind you of the question you forgot to ask.

#### The Same Questions' Returns Diminish

The five defects are fixed. The comparisons are corrected to include equality at the
boundary, an unknown type raises an error, and an invalid number is rejected.

```js
// loan2.mjs — after the fixes found by the questioning checks
const RULES = { student: { maxLoans: 10 }, member: { maxLoans: 5 } };
export const DEBT_LIMIT = 20;

export function canBorrow(memberType, activeLoans, debt) {
  const rule = RULES[memberType];
  if (rule === undefined) throw new Error(`unknown member type: ${memberType}`);
  if (!Number.isInteger(activeLoans) || activeLoans < 0) throw new Error('invalid active loan count');
  if (activeLoans >= rule.maxLoans) return false;
  if (debt >= DEBT_LIMIT) return false;
  return true;
}
```

The same questioning suite is run again, and a new question class is added alongside it:
what happens when the input is not a number?

```js
// new-class.mjs — the same questions repeated, and a new question class compared
import assert from 'node:assert/strict';
import { canBorrow } from './loan2.mjs';
import { run } from './runner.mjs';

const QUESTIONING = {
  'member exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 5, 0), false),
  'student exactly at the limit cannot borrow': () => assert.equal(canBorrow('student', 10, 0), false),
  'debt exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 0, 20), false),
  'unknown member type is rejected': () => assert.throws(() => canBorrow('guest', 0, 0)),
  'negative loan count is rejected': () => assert.throws(() => canBorrow('member', -1, 0)),
};

const NEW_QUESTION_CLASS = {
  'fractional loan count is rejected': () => assert.throws(() => canBorrow('member', 2.5, 0)),
  'no loan is given if debt could not be computed': () => assert.throws(() => canBorrow('member', 0, Number.NaN)),
};

run('old questions', QUESTIONING);
run('new questions', NEW_QUESTION_CLASS);
```

```
old questions: 5/5 passed
new questions: 1/2 passed
  failed: no loan is given if debt could not be computed
```

The old questions no longer find anything. This is an expected outcome: once a defect is
fixed, the test that found it turns into a sentry waiting for that defect to come back —
valuable, but no longer producing new information. A test suite's defect-finding return
diminishes as it is repeated; this is called the **pesticide paradox**. New defects are only
found with new question classes.

The sixth defect found is instructive too: when the debt cannot be computed, the value is
not a number, so every comparison comes out false and the member borrows anyway. Defects
clustering in the same region is not unusual either; if a defect is concentrating somewhere,
deepening the search there is productive. This is called **defect clustering**.

#### Limits of the Mindset

The questioning stance does not mean that every failing test is a defect. A warning that
comes up because the test itself was written wrong, or because the expected value is stale,
is a **false positive** and erodes trust. In the opposite direction, a passing test might
have **missed** a defect. Both kinds of error lower the test suite's information value.

All tests passing is not, by itself, grounds for shipping either. If the product runs
flawlessly and does the wrong thing, the test result says nothing about that; this is where
the validation question set apart in the second lesson belongs. This situation is called the
**absence-of-errors fallacy**.

The final limit is personal. A person testing their own code looks through the assumptions
they made while writing it; an input that never occurred to them while writing does not
occur to them while testing either. The remedy for this is not abandoning the mindset but
multiplying the viewpoint: review, having a second person write scenarios, systematic
application of technique. The next topic is devoted to these techniques.

#### Summary

- On the same function, the happy-path checks passed five for five while the checks asking
  boundary and gap questions revealed five separate defects.
- Confirmation bias is searching for evidence consistent with a claim instead of testing the
  claim; how the question is framed determines which evidence gets sought.
- Questioning questions are generated systematically from the boundary, gap, type, order,
  repetition, scale, and authorization classes.
- Once a defect is fixed, the test that found it turns into a sentry; a repeated suite's
  defect-finding return diminishes, and a new defect calls for a new question class.
- False positives and missed defects lower a test suite's information value in both
  directions; all tests passing does not show that the wrong thing was not done.

#### Next Step

Up to this point, testing has been treated as an activity where one person is alone with one
function. In reality, when a test is written, who writes it, and how quickly its result
reaches someone depends on the way of working. The same check suite carries different value
when it is run once at the end of a release versus when it is run on every change. The next
lesson examines how the delivery model determines testing practice and what the feedback
point changes.

### Delivery Models and Testing

Source: https://academia.sh/en/courses/testing-fundamentals/delivery-models-and-testing

The question of whether testing is a separate stage or an activity that runs throughout the process, the feedback point's effect on delay and candidate change count, and how a suite's duration determines its placement.

Up to this point, testing has been treated as an activity where one person is alone with one
function. In reality, when a test is written, who writes it, and how quickly its result
reaches someone depends on the way of working. The same check suite carries different value
when it is run once at the end of a release versus when it is run on every change.

This lesson makes that difference measurable. What gets measured is not testing itself, but
the distance between a defect's occurrence and its detection.

#### The Delivery Model Determines Where Testing Sits

In the **waterfall** model, work is divided into stages, and each stage hands a document to
the next: analysis, design, development, testing, delivery. Testing here is a **stage** —
performed at the end, on a product already finished beforehand. The model's strength is that
every stage leaves its own output in writing; its weakness is that the validation question
gets deferred to the very end.

In the **incremental and iterative** model, the product grows in working pieces. Each
iteration contains its own analysis, development, and testing. Testing turns from a stage
into an activity distributed inside every iteration.

In **continuous delivery**, every change is considered a deliverable candidate; the only
obstacle in front of delivery is the result of the automated check suite. Here testing stops
being merely an activity and turns into a **gate**: a change whose suite does not pass does
not move forward.

The V-model pairing mentioned in the second lesson holds in all three models — every
decision level has a corresponding testing level. What changes is not the pairing but the
time between its two ends.

#### The Feedback Point

A defect is born the moment the code is written. It is noticed at the first point where a
suite runs against it. The distance between the two has two measures: **delay**, the time
between the defect being born and word of it arriving; and **candidate change count**, the
number of changes accumulated in that time that could be the defect's source.

```js
// feedback.mjs — the same check suite's feedback at different points
const POINTS = [
  { name: 'at commit', delayMin: 0.5, candidates: 1 },
  { name: 'before push', delayMin: 3, candidates: 4 },
  { name: 'integration', delayMin: 20, candidates: 12 },
  { name: 'nightly run', delayMin: 720, candidates: 40 },
  { name: 'before release', delayMin: 20160, candidates: 380 },
];

for (const p of POINTS) {
  console.log(`${p.name.padEnd(16)}: ${String(p.delayMin).padStart(7)} min delay, ${String(p.candidates).padStart(3)} candidate changes`);
}

const first = POINTS[0];
const last = POINTS[POINTS.length - 1];
console.log(`ratio: delay ${last.delayMin / first.delayMin}x, candidates ${last.candidates / first.candidates}x`);
```

```
at commit       :     0.5 min delay,   1 candidate changes
before push     :       3 min delay,   4 candidate changes
integration     :      20 min delay,  12 candidate changes
nightly run     :     720 min delay,  40 candidate changes
before release  :   20160 min delay, 380 candidate changes
ratio: delay 40320x, candidates 380x
```

The numbers in the table are assumptions representing a library team's working tempo; they
are not measured values but model inputs. The conclusion that comes out of the model,
though, does not depend much on the numbers: as the feedback point shifts later, delay and
candidate count grow together.

The effect of the two kinds of growth is different. Delay causes the person who wrote the
defect to lose context; the rationale for a decision made two weeks ago is not remembered.
Candidate count, on the other hand, grows the isolation work: with a single change, the
problem is in it; with three hundred and eighty changes, the problem is somewhere among
them, and finding it is a separate search problem.

This is where the waterfall model's practical cost lies. When the testing stage is
concentrated at a single point at the end of a release, every defect surfaces at the most
expensive feedback point.

#### A Suite's Duration Determines Its Placement

The feedback point cannot be chosen freely. If a check suite is to run on every commit, its
duration must be under a second; as the suite's duration grows, its point shifts later.
Duration here is a placement criterion.

```js
// wait.mjs — a blocking wait that simulates access to an external resource
export function wait(ms) {
  Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
}
```

```js
// suite-duration.mjs — the duration of two check suites and which feedback point they belong to
import assert from 'node:assert/strict';
import { wait } from './wait.mjs';

const fee = (day) => Math.min(Math.max(day - 3, 0) * 2, 20);

const FAST = [
  () => assert.equal(fee(0), 0),
  () => assert.equal(fee(3), 0),
  () => assert.equal(fee(4), 2),
  () => assert.equal(fee(13), 20),
  () => assert.equal(fee(99), 20),
];

const SLOW = [
  () => { wait(400); assert.equal(fee(10), 14); },
  () => { wait(400); assert.equal(fee(20), 20); },
  () => { wait(400); assert.equal(fee(1), 0); },
];

const THRESHOLD_MS = 1000;
for (const [name, suite] of [['fast', FAST], ['slow', SLOW]]) {
  const start = performance.now();
  for (const check of suite) check();
  const duration = performance.now() - start;
  const location = duration < THRESHOLD_MS ? 'can run on every commit' : 'moves to before push';
  console.log(`${name} suite (${suite.length} checks): ${THRESHOLD_MS} ms threshold — ${location}`);
}
```

```
fast suite (5 checks): 1000 ms threshold — can run on every commit
slow suite (3 checks): 1000 ms threshold — moves to before push
```

The rule the two suites test is the same; where they differ is that every check in the slow
suite contains a wait that simulates access to an external resource. The five-check suite is
faster than the three-check suite, because it is not the check count but the external
dependency that decides. Because this measurement uses a blocking wait, it gives the same
verdict on every machine; only the raw duration is environment-dependent.

The rule that emerges is practical: a suite's feedback point is determined not by its check
count but by its slowest dependency. If you want to keep part of a suite fast, the external
dependency has to be removed from it. This is exactly the subject of the next course.

#### The Practices the Model Brings

The delivery model determines not only timing but also which testing practices are
necessary.

In a model where testing happens at a single point at the end of a release, a **code
freeze** period is unavoidable: an interval is needed during which changes are stopped so
defects can be found and fixed. When a change can be delivered at any moment, there is no
such interval; instead, a **quality gate** is defined that every change must pass.

When the feedback point is late, defects are found in batches and prioritization becomes a
separate work item; when it is early, defects are found one at a time and most are fixed
instantly. This is why the discipline of keeping defect reports also shifts weight depending
on the model.

The common point is this: no model makes testing cheaper. What the model determines is when
the same testing happens and at what cost.

#### Summary

- In waterfall, testing is a stage; in the incremental model, an activity distributed inside
  every iteration; in continuous delivery, a gate in front of progress.
- The feedback point is measured by two quantities: delay and candidate change count; as the
  point shifts later, both grow together.
- Delay leads to a loss of context; a growing candidate count turns isolation into a separate
  search problem.
- A suite's feedback point is determined by its slowest dependency, not its check count; in
  the example, the five-check suite came out faster than the three-check suite.
- The model does not make testing cheaper; it determines when the same testing happens and
  at what cost.

#### Next Step

This topic established what quality is, which questions it is tested against, and how the
process positions that testing. What's missing is testing's own internal order: what does a
check cover, how far does it reach, what information is it written with? The next topic
classifies tests. The first question concerns scope — the same defect can be caught by a test
that exercises a single function or by a test that brings up the entire system; the cost and
the message of the two are not the same.

## Test Classification

Test levels and their costs, the black-white-grey box distinction, functional test types, the equivalence class, boundary value, and decision table techniques, writing test cases, the test oracle as the source of the expected result, and the error life cycle.

### Test Levels

Source: https://academia.sh/en/courses/testing-fundamentals/test-levels

Separating unit, integration, system, and acceptance levels by the boundary they cover, catching the same defect at three levels, and diagnostic distance growing along with the level.

The previous topic established what quality is, what questions it is tested against, and
how the process positions that testing. What is missing is the internal order of testing
itself. The same defect can be caught by a test that exercises a single function or by one
that brings up the whole system; the two do not cost the same and do not say the same
thing.

This lesson separates tests by the **boundary they cover**. The class names — unit,
integration, system, acceptance — are names for this single criterion.

#### What Determines the Level Is Coverage

**Unit test** exercises a single function or module, touching no other part. Its input is
directly the function's input, its output is directly the return value.

**Integration test** exercises two or more parts working together. The defect is no longer
inside the parts but can be in the contract between them: a misordered argument, a
mismatched unit, an unexpected null value.

**System test** exercises all parts together, through the real entry point. Its question is
whether the result the user sees is correct.

**Acceptance test** overlaps with the system test in coverage; where it differs is the
source of the criterion. The acceptance test's expected result is written in the language
of the party who will use the product and forms the condition of delivery. The validation
question separated in the second lesson is asked at this level.

The Testing and Monitoring topic in the Frontend Quality course built the same distinction
in a browser context; here the levels are treated independently of the domain.

#### The Same Defect at Three Levels

The library is split into three files. The rule layer computes the fee for a single delay,
the aggregation layer derives a member's debt, and the transaction layer generates the
return receipt.

```js
// fee.mjs — rule layer: the fee for a single delay
export const GRACE_DAYS = 3;
export const DAILY_FEE = 2;
export const UPPER_LIMIT = 20;

export function lateFee(lateDays) {
  return Math.max(lateDays - GRACE_DAYS, 0) * DAILY_FEE;
}
```

```js
// debt.mjs — aggregation layer: the sum of a member's delays
import { lateFee } from './fee.mjs';

export function memberDebt(loans) {
  return loans.reduce((total, o) => total + lateFee(o.lateDays), 0);
}
```

```js
// receipt.mjs — transaction layer: generating the return receipt
import { lateFee } from './fee.mjs';
import { memberDebt } from './debt.mjs';

export function returnReceipt(member, loans) {
  return {
    memberNo: member.no,
    items: loans.map((o) => ({ book: o.book, fee: lateFee(o.lateDays) })),
    total: memberDebt(loans),
  };
}
```

There is a defect in the rule layer: `UPPER_LIMIT` is defined but never used in the
computation. The same defect is searched for at all three levels. The block below
generates three test files and runs each one separately with Node's built-in test runner.
Because the duration fields in the output depend on the environment, only the result count
and the assertion message are filtered through.

```bash
cat > unit.test.mjs <<'EOF'
import test from 'node:test';
import assert from 'node:assert/strict';
import { lateFee } from './fee.mjs';

test('a single delay fee does not exceed the upper limit', () => {
  const fee = lateFee(100);
  assert.ok(fee <= 20, `fee ${fee} units`);
});
EOF

cat > integration.test.mjs <<'EOF'
import test from 'node:test';
import assert from 'node:assert/strict';
import { memberDebt } from './debt.mjs';

test('the total debt of a member does not exceed the upper limit', () => {
  const debt = memberDebt([{ lateDays: 100 }, { lateDays: 30 }]);
  assert.ok(debt <= 20, `debt ${debt} units`);
});
EOF

cat > system.test.mjs <<'EOF'
import test from 'node:test';
import assert from 'node:assert/strict';
import { returnReceipt } from './receipt.mjs';

test('the return receipt total does not exceed the upper limit', () => {
  const receipt = returnReceipt({ no: 7 }, [{ book: 'K-1', lateDays: 100 }]);
  assert.ok(receipt.total <= 20, `receipt total ${receipt.total} units`);
});
EOF

for group in unit integration system; do
  node --test "$group.test.mjs" > "$group.log" 2>&1
  printf '%-12s %s | %s\n' "$group" "$(grep -E '^ℹ fail ' "$group.log")" \
    "$(grep -m1 AssertionError "$group.log" | sed 's/.*: //')"
done
```

```
unit         ℹ fail 1 | fee 194 units
integration  ℹ fail 1 | debt 248 units
system       ℹ fail 1 | receipt total 194 units
```

All three levels caught the defect. This does not mean the levels are redundant; what
differs is the cost of catching it.

#### Diagnostic Distance

A test failing does not directly say where the defect is; it only says the defect is
somewhere within its coverage. The wider the coverage, the larger the area in which the
defect must be searched for.

```js
// levels.mjs — the same defect caught at three levels, and the file count each covers
import assert from 'node:assert/strict';
import { lateFee } from './fee.mjs';
import { memberDebt } from './debt.mjs';
import { returnReceipt } from './receipt.mjs';

const LEVELS = [
  {
    name: 'unit', files: ['fee.mjs'],
    check: () => assert.equal(lateFee(100), 20),
  },
  {
    name: 'integration', files: ['fee.mjs', 'debt.mjs'],
    check: () => assert.equal(memberDebt([{ lateDays: 100 }, { lateDays: 2 }]), 20),
  },
  {
    name: 'system', files: ['fee.mjs', 'debt.mjs', 'receipt.mjs'],
    check: () => {
      const receipt = returnReceipt({ no: 7 }, [{ book: 'K-1', lateDays: 100 }]);
      assert.equal(receipt.total, 20);
    },
  },
];

for (const s of LEVELS) {
  let result = 'passed';
  try { s.check(); } catch { result = 'failed'; }
  console.log(`${s.name.padEnd(12)}: ${result} — spans ${s.files.length} file${s.files.length === 1 ? '' : 's'}`);
}
```

```
unit        : failed — spans 1 file
integration : failed — spans 2 files
system      : failed — spans 3 files
```

When a unit test fails, there is exactly one file to look at. When a system test fails, the
defect could be in any of the three files, and finding out which one is a separate effort.
This can be called **diagnostic distance**: the work between a test failing and the defect
being found.

Three more costs grow along with coverage. **Setup cost**: a system test needs data,
configuration, and an entry point. **Duration**: as coverage widens, the run time grows and
the feedback point from the previous lesson shifts back. **Fragility**: any part within the
coverage can fail the test, so tests with wide coverage fail more often for unrelated
reasons — this is the main source of the false positive named in the fourth lesson.

Narrow coverage, in turn, has its own cost. A unit test cannot see the defect **between**
parts: each part can work correctly on its own and still produce a wrong result once
combined. The defect above was visible at every level because it lived in a single layer;
contract mismatches do not behave that way.

#### The Distribution of Levels

The relative number of levels is a choice, and it is named after one of two extremes. In
the **test pyramid**, the base is wide: many fast unit tests, fewer integration tests, and
few system tests at the top. The rationale is cost distribution — most defects are caught
at the cheap level, and the expensive level only tests the combination of parts.

The reverse arrangement is called the **ice-cream cone**: few unit tests, many system
tests, and manual testing at the top. In this arrangement, every piece of feedback is slow
and every failing test is expensive.

The pyramid is not a rule but a cost observation. The ratios change as the domain changes:
a heavy computation layer widens the base, a thin interface layer shifts weight toward the
upper level. The question to ask when deciding stays fixed — at which level can this defect
be caught most cheaply?

#### Summary

- Test levels are separated not by what they test but by the boundary they cover: unit
  tests a single part, integration tests the combination of parts, system tests the whole.
- The acceptance test overlaps with the system test in coverage; it is separated by its
  criterion coming from the language of the party who will use the product.
- In the example, a defect in a single layer was caught at all three levels; the difference
  was not in catching it but in the cost of catching it.
- As coverage widens, diagnostic distance, setup cost, duration, and fragility all grow
  together; narrow coverage, in turn, cannot see the defect between parts.
- The test pyramid is not a rule but a cost observation; the deciding criterion is at which
  level a defect can be caught most cheaply.

#### Next Step

The level determines how far a test reaches. There is an independent second question: what
did the person who wrote the test look at? All three tests above were derived from the
specification — they read the upper-limit rule and asked about it. Had they been written by
looking into the code, different questions would have come out, because the code can have
paths that are never in the specification. The next lesson takes up this information-level
distinction and shows that the two approaches find different kinds of defects.

### Black, White and Grey Box

Source: https://academia.sh/en/courses/testing-fundamentals/black-white-and-gray-box

The distinction of what information a test is written from; specification-based testing finding a missing function, structure-based testing finding an undocumented branch, and comparing the two approaches by branch coverage.

The previous lesson asked how far a test reaches. There is an independent second question:
what did the person who wrote the test look at? The three tests there were also derived
from the specification — the upper-limit rule was read and asked about. Had they been
written by looking into the code, different questions would have come out, because the
code can have paths that are never in the specification.

This lesson establishes that information-level distinction and shows, on the same
function, that the two approaches find **different kinds** of defects.

#### The Information-Level Distinction

**Black-box testing** uses only what is visible from the outside: the specification, the
interface, input and output. How the code is written is unknown. This is why it is also
called **specification-based testing**.

**White-box testing** is written by looking at the source code: which conditions exist,
which branches exist, which loop bounds exist? The name **structure-based testing**
describes this view better — its criterion is the structure of the program.

**Grey-box testing** lies between the two. The source code is not read, but part of the
internal structure is known: the data schema, the record format, log entries, error codes.
This partial knowledge guides input selection.

The distinction is independent of level. A unit test can be written as black-box or
white-box; the same holds for a system test.

#### The Traced Fee Computation

For the comparison to be measurable, the fee computation is written in a form that records
which branches run. The `TRACE` set holds the numbers of the branches that ran.

```js
// fee-traced.mjs — the fee computation that records which branches ran
export const BRANCHES = ['D1', 'D2', 'D3', 'D4'];
export const TRACE = new Set();

export function lateFee(lateDays, memberNo, offBranch = false) {
  if (memberNo === 0) { TRACE.add('D1'); return 0; }
  TRACE.add('D2');
  let fee = Math.max(lateDays - 3, 0) * 2;
  if (fee > 20) { TRACE.add('D3'); fee = 20; } else { TRACE.add('D4'); }
  return fee;
}
```

The specification has four items: the first three days are free, every following day costs
2 units, the fee does not exceed 20 units, and an off-branch return incurs a 5-unit
surcharge. The last item is the library's new decision and carries the number K12.

#### What Black-Box Finds

Black-box checks are derived only from these four items. Because the `memberNo` condition
inside the code is unknown, no check targets it.

```js
// black-box.mjs — checks derived only from the specification
import assert from 'node:assert/strict';
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const CHECKS = {
  'K7a first 3 days are free': () => assert.equal(lateFee(3, 41), 0),
  'K7b 2 units per day': () => assert.equal(lateFee(6, 41), 6),
  'K7c fee does not exceed 20 units': () => assert.equal(lateFee(100, 41), 20),
  'K12 off-branch return incurs a 5-unit surcharge': () => assert.equal(lateFee(4, 41, true), 7),
};

let passed = 0;
const failed = [];
for (const [name, check] of Object.entries(CHECKS)) {
  try { check(); passed += 1; } catch { failed.push(name); }
}

console.log(`black-box      : ${passed}/${Object.keys(CHECKS).length} checks passed`);
for (const name of failed) console.log(`  failed: ${name}`);
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);
console.log(`  unreached: ${BRANCHES.filter((d) => !TRACE.has(d)).join(', ')}`);
```

```
black-box      : 3/4 checks passed
  failed: K12 off-branch return incurs a 5-unit surcharge
branch coverage: 3/4 branches ran
  unreached: D1
```

Black-box found a function that **does not exist at all** in the code: the off-branch
return rule exists in the specification but not in the implementation. White-box cannot
find this defect by its very structure — a test cannot be written for a branch that does
not exist. A missing function is visible only by comparing against the specification.

The last line of the output, however, shows black-box's blind spot: one of the four
branches never ran. A branch with no counterpart in the specification is not the target of
any check derived from the specification. The role of the **branch coverage** measurement
here is not to find a defect but to name the region nobody is looking at.

#### What White-Box Finds

Someone looking at the source code sees the `D1` branch and writes a check that targets it
directly.

```js
// white-box.mjs — a check targeting the unreached branch seen from the source code
import assert from 'node:assert/strict';
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const fee = lateFee(100, 0);
try {
  assert.ok(fee === 20, `fee for member number 0 is ${fee}, expected 20`);
  console.log('D1 check: passed');
} catch (error) {
  console.log(`D1 check: failed — ${error.message}`);
}
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);
```

```
D1 check: failed — fee for member number 0 is 0, expected 20
branch coverage: 1/4 branches ran
```

What it finds is a processing shortcut: member number zero is never charged a fee at all. A
branch like this is usually code that was added during some experiment and never removed;
because it has no counterpart in the specification, no black-box check ever asks about it,
and it settles into the product unnoticed.

The second line of the same output shows white-box's own blind spot. A set that targets a
single branch runs only that branch; tests written by looking at structure cover the
code's **existing** paths, not its **missing** ones. Work that takes structure as its
criterion never looks for a rule that was never written.

#### Grey-Box Uses the Knowledge In Between

In the grey-box approach the source code is not read, but the schema of member records is
known: the member number is an integer starting from zero. This knowledge is enough to
guide input selection.

```js
// grey-box.mjs — probes derived from knowledge of the data schema
import { lateFee, BRANCHES, TRACE } from './fee-traced.mjs';

const MEMBER_NO_RANGE = { min: 0, max: 9999 };
const DELAY_SAMPLES = [0, 4, 100];

const deviations = [];
for (const memberNo of [MEMBER_NO_RANGE.min, MEMBER_NO_RANGE.max]) {
  for (const days of DELAY_SAMPLES) {
    const expected = Math.min(Math.max(days - 3, 0) * 2, 20);
    const observed = lateFee(days, memberNo);
    if (observed !== expected) deviations.push(`memberNo=${memberNo} days=${days}: ${observed} (expected ${expected})`);
  }
}

console.log(`grey-box       : ${deviations.length} deviations found`);
for (const s of deviations) console.log(`  ${s}`);
console.log(`branch coverage: ${TRACE.size}/${BRANCHES.length} branches ran`);
```

```
grey-box       : 2 deviations found
  memberNo=0 days=4: 0 (expected 2)
  memberNo=0 days=100: 0 (expected 20)
branch coverage: 4/4 branches ran
```

Schema knowledge gives no access to the source code; it only says which inputs are
boundaries. Knowing that zero is the low end of a member-number range is enough to try that
value, and the undocumented branch surfaces without ever looking at the source code. All
four of the four branches ran.

#### Which Approach, When

The three approaches do not replace one another; they close different gaps.

Black-box finds the gap between the specification and the product, and it is unaffected by
changes to the code; these tests survive when the code is refactored. It is the only valid
approach at the acceptance level and at external interfaces, because the criterion is what
the user sees.

White-box finds paths with no counterpart in the specification and names coverage gaps. Its
cost is being tied to the structure of the code: when the structure changes, the test
itself changes.

Grey-box works from the external interface while using internal knowledge to select input.
This is the most common case in practice; the person writing the test usually knows the
schema, the error codes, and the log entries.

The practical criterion is this: the body of the checks is written from the specification,
and the coverage measurement is read from the structure. When a coverage gap is seen, the
question to ask should not be "should I write a test here" but "does this branch have a
counterpart in the specification." If it does not, what should be written is not a test but
the removal of that branch.

#### Summary

- Black-box relies on the specification, white-box on the structure of the source code,
  grey-box on partial internal knowledge such as schema and record format; the distinction
  is independent of test level.
- Black-box finds a function that does not exist in the code at all; white-box cannot see
  this defect because it cannot write a test for a branch that does not exist.
- White-box finds branches with no counterpart in the specification; in the example, the
  undocumented shortcut that leaves member number zero charged no fee at all surfaced this
  way.
- The branch coverage measurement does not find defects; it names the region nobody is
  looking at.
- The right question for a coverage gap is not "should I write a test" but "does this
  branch have a counterpart in the specification."

#### Next Step

The level and the information level determine where a test sits, not its purpose. The same
black-box unit test can be run to see whether the release is standing, or to check whether
a fixed defect has come back. The test is the same in both cases; what changes is why it is
run and what is done with its result. The next lesson names tests by this purpose: smoke,
sanity, regression, and acceptance.

### Functional Test Types

Source: https://academia.sh/en/courses/testing-fundamentals/functional-test-types

Separating smoke, sanity, regression, confirmation, and acceptance tests by purpose, catching different defects in the same release, and the maintenance cost of the regression set.

The level and the information level determine where a test sits, not its purpose. The same
black-box unit test can be run to see whether the release is standing, or to check whether
a fixed defect has come back. The test is the same in both cases; what changes is why it is
run and what is done with its result.

This lesson names tests by their purpose. The function of the names is to build a shared
understanding of when a set is run and what happens when it fails.

#### Purpose Names the Test

**Smoke test** is wide and shallow: it tries every main path once, testing none of them
deeply. Its question is one sentence — is this release worth working on? The decision when
it fails is also single: the release is rejected, and no detailed testing follows.

**Sanity test** is narrow and deep: it probes the region a specific change touches, in a
few minutes. Its question is — does this fix look reasonable? It is a pre-filter done
before running the full set.

**Regression test** checks whether behavior that used to work has broken. Its set
accumulates over time: every fixed defect leaves behind a sentinel that will warn if it
comes back.

**Confirmation test** checks that a single defect reported as fixed is actually fixed. It
is a step-by-step repeat of the defect report, and its coverage is exactly that report.

**Acceptance test** is the condition of delivery, and its criterion comes from the party
who will use the product; this is the place defined for it in the previous two lessons.

#### Smoke and Regression in the Same Release

Three versions of the library are set up: a sound version, a version with a core operation
broken, and a version where a previously fixed defect has come back.

```js
// version.mjs — the library's sound version and two broken versions
export function baseLibrary() {
  return {
    findMember: (no) => (no >= 0 ? { no, type: no % 3 === 0 ? 'student' : 'member' } : null),
    lateFee: (days) => Math.min(Math.max(days - 3, 0) * 2, 20),
    lend: (member, book) => ({ memberNo: member.no, book, status: 'lent' }),
  };
}

export const VERSIONS = {
  'S0 sound': baseLibrary(),
  'S1 core broken': { ...baseLibrary(), lend: () => { throw new Error('record could not be opened'); } },
  'S2 old defect returned': { ...baseLibrary(), lateFee: (days) => Math.max(days - 3, 0) * 2 },
};
```

The smoke set tries the three main paths once each. The regression set is the sentinel of
three defect reports; each check's name carries the number of the report it comes from.

```js
// types.mjs — comparing the smoke and regression sets across three versions
import assert from 'node:assert/strict';
import { VERSIONS } from './version.mjs';

const SMOKE = {
  'a member record opens': (k) => assert.ok(k.findMember(41)),
  'a loan is issued': (k) => assert.equal(k.lend(k.findMember(41), 'K-1').status, 'lent'),
  'a fee is computed': (k) => assert.equal(typeof k.lateFee(5), 'number'),
};

const REGRESSION = {
  'H-14 fee does not exceed the upper limit': (k) => assert.equal(k.lateFee(100), 20),
  'H-22 grace period is three days': (k) => assert.equal(k.lateFee(3), 0),
  'H-31 an invalid member number returns null': (k) => assert.equal(k.findMember(-1), null),
};

function run(set, library) {
  const failed = [];
  for (const [name, check] of Object.entries(set)) {
    try { check(library); } catch { failed.push(name); }
  }
  return failed;
}

for (const [versionName, library] of Object.entries(VERSIONS)) {
  const line = [];
  for (const [setName, set] of [['smoke', SMOKE], ['regression', REGRESSION]]) {
    const failed = run(set, library);
    line.push(`${setName} ${Object.keys(set).length - failed.length}/${Object.keys(set).length}${failed.length ? ` (${failed[0]})` : ''}`);
  }
  console.log(`${versionName.padEnd(24)}: ${line.join(' | ')}`);
}
```

```
S0 sound                : smoke 3/3 | regression 3/3
S1 core broken          : smoke 2/3 (a loan is issued) | regression 3/3
S2 old defect returned  : smoke 3/3 | regression 2/3 (H-14 fee does not exceed the upper limit)
```

Two defects were caught by two separate sets. When the core operation breaks, the
regression set stays quiet, because its sentinels sit in the region of past defects; lending
a book never produced a defect there, so there is no sentinel for it. When the old defect
comes back, the smoke set stays quiet, because it only asked whether the fee computation
returned a number.

Both sets are correct in purpose, and both are incomplete. This incompleteness is not a
defect; it follows from the definition: the shallow set does not know depth, the narrow set
does not know breadth.

#### Confirmation Testing Alone Is Not Enough

A fix is made for the fee defect. The person writing the fix restores the upper limit, but
drops the grace period.

```js
// sanity.mjs — the outcome of confirmation, sanity, and regression sets on the same fix
import assert from 'node:assert/strict';
import { baseLibrary } from './version.mjs';

const fixed = { ...baseLibrary(), lateFee: (days) => Math.min(days * 2, 20) };

const CONFIRMATION_TEST = {
  'H-14 the state in the report': () => assert.equal(fixed.lateFee(100), 20),
};

const SANITY = {
  'fee at 0 days is 0': () => assert.equal(fixed.lateFee(0), 0),
  'fee at 3 days is 0': () => assert.equal(fixed.lateFee(3), 0),
  'fee at 4 days is 2': () => assert.equal(fixed.lateFee(4), 2),
  'fee at 13 days is 20': () => assert.equal(fixed.lateFee(13), 20),
  'fee at 14 days is 20': () => assert.equal(fixed.lateFee(14), 20),
};

const REGRESSION = {
  'H-14 fee does not exceed the upper limit': () => assert.equal(fixed.lateFee(100), 20),
  'H-22 grace period is three days': () => assert.equal(fixed.lateFee(3), 0),
  'H-31 an invalid member number returns null': () => assert.equal(fixed.findMember(-1), null),
};

for (const [name, set] of [['confirmation', CONFIRMATION_TEST], ['sanity test ', SANITY], ['regression  ', REGRESSION]]) {
  const failed = Object.entries(set).filter(([, d]) => { try { d(); return false; } catch { return true; } });
  console.log(`${name}: ${Object.keys(set).length - failed.length}/${Object.keys(set).length} passed${failed.length ? ` — first failed: ${failed[0][0]}` : ''}`);
}
```

```
confirmation: 1/1 passed
sanity test : 3/5 passed — first failed: fee at 3 days is 0
regression  : 2/3 passed — first failed: H-22 grace period is three days
```

Confirmation testing passed: the state in the defect report now gives the correct result.
The report could be closed. The sanity test, however, showed a second breakage in the same
region within a few seconds — the fix damaged its own surroundings.

Two rules follow from this. When a defect is fixed, not only the state in the report but
also the boundary values of that rule are tested. And confirmation testing passing is not
enough to close the report; the region of the fixed defect is also probed.

#### The Maintenance Cost of the Sets

The regression set grows in one direction only: every new defect adds a sentinel, and none
of them leaves on its own. Over time, the set becomes slow enough to push the feedback
point back.

There are three levers. **Regression test selection** runs only the checks that touch the
changed region; the full set is left for a point further back. **Pruning** removes the
sentinel for a rule that no longer has a counterpart — no rule, no sentinel. Pruning
**flaky tests** protects trust as well: a check that fails for unrelated reasons turns,
when it fails, into noise nobody looks at, and it lowers the value of the whole set.

The maintenance of the smoke set runs the opposite way: it must stay small. As a smoke set
grows, it stops being able to do its own job, because giving a fast answer is its only job.

#### Summary

- Test types are separated by purpose: smoke is wide and shallow, sanity is narrow and
  deep, regression is the sentinel of past defects, confirmation testing is the repeat of a
  single report.
- In the example, only the smoke set caught the core breakage, and only the regression set
  caught the returning defect; the shortfall of both sets follows from their definitions.
- Confirmation testing passing shows the defect was fixed, not that the fix left its
  surroundings undamaged; in the example, the sanity test surfaced the second breakage.
- The regression set grows in one direction; selective runs, pruning checks with no
  remaining counterpart, and removing flaky tests are all parts of its maintenance.
- The maintenance of the smoke set is staying small; once it grows, it loses its function
  of giving a fast answer.

#### Next Step

So far, the inputs to the checks were chosen as examples: 3 days, 4 days, 100 days. The
reason for the choice was intuition. There are systematic ways to select a finite subset
from an infinite input set, and these ways justify which inputs are tried. The next lesson
establishes these techniques: grouping inputs that behave the same into classes, probing
the edges of the classes, and counting condition combinations exhaustively.

### Test Design Techniques

Source: https://academia.sh/en/courses/testing-fundamentals/test-design-techniques

Systematic ways to select a finite check set from an infinite input space: equivalence classes, boundary value analysis, and the decision table that counts condition combinations exhaustively.

So far, the inputs to the checks were chosen as examples: 3 days, 4 days, 100 days. The
reason for the choice was intuition, and intuition is not repeatable — when two people
test the same function, two different sets come out, and what either one covers stays
unclear.

This lesson turns input selection into a justifiable process. Three techniques are
covered: grouping inputs that behave the same into classes, probing the edges of the
classes, and counting condition combinations exhaustively.

#### The Input Space Is Infinite, the Check Count Is Finite

The boundary established in the first lesson becomes operative here: a test can show the
presence of a defect, not its absence, because an infinite input set cannot be tried with
a finite number of inputs. So the question to ask should not be "was everything tried" but
"what do the ones that were tried represent."

Test design techniques establish this representation relationship. Each technique divides
the input space by a criterion and justifies selecting from the partitions. Because the
criterion is written down, **why** particular inputs were chosen, and what was left out,
is written down too.

The loan decision is used as the example. The decision depends on three rules: the book
is not lent if it is reserved for someone else, it is not lent if the member's debt has
reached the limit, and it is not lent if the member's active loan count has reached their
type's limit.

```js
// decision.mjs — the three-rule implementation of the loan decision
export const LOAN_LIMIT = { student: 10, member: 5 };
export const DEBT_LIMIT = 20;

export function loanDecision({ memberType, activeLoans, debt, reservedForAnother }) {
  if (!(memberType in LOAN_LIMIT)) throw new Error(`unknown member type: ${memberType}`);
  if (!Number.isInteger(activeLoans) || activeLoans < 0) throw new Error('invalid active loan count');
  if (reservedForAnother) return 'reserved';
  if (debt > DEBT_LIMIT) return 'debt';
  if (activeLoans >= LOAN_LIMIT[memberType]) return 'limit';
  return 'granted';
}
```

#### Equivalence Classes

An **equivalence class** is a set of inputs the program handles the same way. The
assumption is: if one member of a class exposes a defect, the other members of the same
class do too. If this assumption holds, one representative per class is enough.

The debt value is split into two classes: those below the limit and those that reach it.
The active loan count is split into two the same way. Reservation status is already
two-valued.

```js
// classes.mjs — one representative from each equivalence class
import { loanDecision } from './decision.mjs';

const base = { memberType: 'member', activeLoans: 2, debt: 8, reservedForAnother: false };
const REPRESENTATIVES = [
  ['debt class: below limit', { debt: 8 }, 'granted'],
  ['debt class: above limit', { debt: 35 }, 'debt'],
  ['loan class: below limit', { activeLoans: 2 }, 'granted'],
  ['loan class: above limit', { activeLoans: 7 }, 'limit'],
  ['reserved: no', { reservedForAnother: false }, 'granted'],
  ['reserved: yes', { reservedForAnother: true }, 'reserved'],
];

let passed = 0;
for (const [name, change, expected] of REPRESENTATIVES) {
  const observed = loanDecision({ ...base, ...change });
  if (observed === expected) passed += 1;
  else console.log(`  deviation: ${name} — expected ${expected}, observed ${observed}`);
}
console.log(`equivalence classes: ${passed}/${REPRESENTATIVES.length} representatives matched`);
```

```
equivalence classes: 6/6 representatives matched
```

All six representatives matched. The payoff of the technique shows here: instead of tens
of thousands of possible debt values, two values were tried, and it became written down
that every class was covered.

Two kinds of division are made when splitting classes. A **valid partition** collects the
inputs the program must accept, an **invalid partition** collects the ones it must reject
— a negative loan count and an unknown member type are invalid partitions. Invalid
partitions are tried one at a time; if two invalid values are combined in the same check,
it is common for the first rejection to shadow the second and leave the defect invisible.

#### Boundary Value Analysis

The equivalence class assumption weakens in one place: at the **edge** of the classes. A
large share of defects cluster in comparison operators, and an operator error shows up
only at the edge — a representative chosen from the middle of the class cannot see it.

**Boundary value analysis** produces three values for each class edge: one below the
threshold, the threshold itself, and one above it. The values are not derived by hand but
from the declared thresholds; when a threshold changes, the checks change with it
automatically.

```js
// boundary.mjs — values derived from the edges of the classes
import { loanDecision, LOAN_LIMIT, DEBT_LIMIT } from './decision.mjs';

const neighbors = (threshold) => [threshold - 1, threshold, threshold + 1];
const base = { memberType: 'member', activeLoans: 2, debt: 8, reservedForAnother: false };

const CHECKS = [
  ...neighbors(DEBT_LIMIT).map((debt) => ({
    name: `debt=${debt}`, input: { debt }, expected: debt >= DEBT_LIMIT ? 'debt' : 'granted',
  })),
  ...neighbors(LOAN_LIMIT.member).map((activeLoans) => ({
    name: `activeLoans=${activeLoans}`, input: { activeLoans },
    expected: activeLoans >= LOAN_LIMIT.member ? 'limit' : 'granted',
  })),
];

let deviations = 0;
for (const d of CHECKS) {
  const observed = loanDecision({ ...base, ...d.input });
  if (observed !== d.expected) {
    deviations += 1;
    console.log(`  deviation: ${d.name} — expected ${d.expected}, observed ${observed}`);
  }
}
console.log(`boundary values: ${CHECKS.length} checks, ${deviations} deviation${deviations === 1 ? '' : 's'}`);
```

```
  deviation: debt=20 — expected debt, observed granted
boundary values: 6 checks, 1 deviation
```

Six checks, one deviation. A member with debt at exactly 20 units is still lent the book;
the rule says "reaching the limit" while the implementation says "exceeding the limit."
This defect would not have been visible with any representative chosen from the middle of
the classes, and indeed it was not visible in the previous set.

How many values should be tried? A three-value boundary (one below, the threshold, one
above), as above, catches every operator error. A two-value boundary (the threshold and
one above) is cheaper and finds most of the same defects. The choice is made according to
the risk the threshold carries — three values are preferred for money, authority, and
duration thresholds.

#### Decision Table

Equivalence classes and boundary values handle inputs **one at a time**. When rules affect
one another, this is not enough: which condition combinations were tried and which were
skipped stays invisible.

A **decision table** writes conditions as rows and combinations as columns, and states
which action each combination produces. Three conditions mean eight combinations; when the
count is done by a program, none of them is skipped.

```js
// decision-table.mjs — counting all combinations of three conditions and merging rules
const CONDITIONS = ['reserved', 'debt', 'loan'];

function action([reserved, debt, limit]) {
  if (reserved) return 'reserved';
  if (debt) return 'debt';
  if (limit) return 'limit';
  return 'granted';
}

const combinations = [];
for (let i = 0; i < 2 ** CONDITIONS.length; i += 1) {
  const row = CONDITIONS.map((_, j) => Boolean((i >> (CONDITIONS.length - 1 - j)) & 1));
  combinations.push({ row, action: action(row) });
}

console.log(`full count: ${combinations.length} combinations (${CONDITIONS.join(' / ')})`);
for (const b of combinations) {
  console.log(`  ${b.row.map((v) => (v ? 'Y' : 'N')).join(' ')} -> ${b.action}`);
}

const groups = new Map();
for (const b of combinations) {
  if (!groups.has(b.action)) groups.set(b.action, []);
  groups.get(b.action).push(b.row);
}

console.log(`merged rule count: ${groups.size}`);
for (const [act, rows] of groups) {
  const rule = CONDITIONS.map((_, j) => {
    const values = new Set(rows.map((s) => s[j]));
    return values.size === 1 ? ([...values][0] ? 'Y' : 'N') : '-';
  });
  const coverage = rule.reduce((n, k) => n * (k === '-' ? 2 : 1), 1);
  console.log(`  ${rule.join(' ')} -> ${act.padEnd(8)} ${rows.length} combination${rows.length === 1 ? '' : 's'}, rule covers ${coverage} combination${coverage === 1 ? '' : 's'}`);
}
```

```
full count: 8 combinations (reserved / debt / loan)
  N N N -> granted
  N N Y -> limit
  N Y N -> debt
  N Y Y -> debt
  Y N N -> reserved
  Y N Y -> reserved
  Y Y N -> reserved
  Y Y Y -> reserved
merged rule count: 4
  N N N -> granted  1 combination, rule covers 1 combination
  N N Y -> limit    1 combination, rule covers 1 combination
  N Y - -> debt     2 combinations, rule covers 2 combinations
  Y - - -> reserved 4 combinations, rule covers 4 combinations
```

Eight combinations collapsed into four rules. The `-` mark shows a **don't care**
condition: once the book is reserved for someone else, debt and loan count do not change
the result, because the reservation check comes before the others. The correctness of the
merge is confirmed in the last column — the number of combinations a rule covers must
equal the number of combinations in its group.

The real payoff of the decision table is making the rules' **priority** visible. What
response should a member with debt at the limit and a reserved book get? The table forces
this question to be asked; if the specification never wrote it down, the gap surfaces
while drawing the table. The check count is also determined in a stable way: four checks
are enough for four rules, eight checks are unnecessary.

As the condition count grows, the combination count doubles; ten conditions mean one
thousand and twenty-four combinations. At this point exhaustive counting is dropped, and
combinations where each condition alone determines the result are selected instead. The
selection criterion changes; the table itself does not.

#### The Division of Labor Among Techniques

The three techniques answer the same question from different places, and none replaces
the others.

Equivalence classes shrink the input space and write down what is being represented.
Boundary values catch defects at the edges of the classes — in the example, only this
technique found the one defect. The decision table ensures condition combinations are
counted exhaustively and makes priority gaps in the specification visible.

Sequential behavior has its own technique: when a record moves from state to state, what
needs to be tried is not values but transitions. This technique is taken up in the final
lesson.

#### Summary

- Test design techniques take input selection out of intuition and turn it into a
  justifiable process; because the criterion is written down, what was left out is written
  down too.
- An equivalence class is the set of inputs the program handles the same way; the
  one-representative-per-class assumption holds in the middle of the class and weakens at
  its edge.
- Invalid partitions are tried one at a time; when combined, the first rejection shadows
  the second.
- Boundary value analysis tries one below the threshold, the threshold, and one above it;
  in the example, only this technique found the defect that a member with debt at exactly
  the limit was still lent the book.
- The decision table counts condition combinations exhaustively, lowers the check count by
  merging don't-care conditions, and makes priority gaps in the specification visible.

#### Next Step

The techniques determined which inputs to try. Being able to explain what was tried to
someone else is a separate matter: what state to start in, which steps to follow, what to
observe? An automated check carries this information inside the code and only tells it to
whoever reads the code. The next lesson gives this information a written form and shows
the move from that form to a runnable test.

### Writing Test Cases

Source: https://academia.sh/en/courses/testing-fundamentals/writing-test-scenarios

A test case's identifier, precondition, steps, and expected-result parts, the criteria for a well-written test case, and the move from a written test case to an executable test.

The techniques settled which inputs to try. Being able to describe what was tried to
someone else is a separate task: what state does it start from, which steps are
followed, what is observed? An automated check carries this information inside code and
speaks only to whoever reads the code. A form is needed that a librarian, a reviewer, or
the analyst who wrote the rule can read.

This lesson builds that form and shows the move from the form to an executable test.

#### Parts of a Test Case

A **test case** consists of six parts.

**Identifier** gives the test case a fixed number; a defect report and a regression set
refer to this number. **Title** states what the test case checks in a single sentence
and includes the expected outcome — whoever reads the title understands the test's
intent.

**Traceability link** shows which rule the test case comes from. The process-quality
measurement built in the first lesson depends exactly on this link: which rule has a
test case, and which does not?

**Precondition** states what state the system must be in for the test case to start.
Without a precondition a test case is not repeatable; running it twice gives two
different results.

**Steps** give the work to be done in order. Because a test case has a single purpose,
the step count is small; a long list of steps is usually several test cases stuck
together.

**Expected result** states what will be observed. It must be observable: something on
screen, a returned value, a saved row. "The system behaves correctly" is not an
observable result.

A **postcondition** can be added: the state the system is left in once the test case
finishes. When it is written, test cases do not depend on the state another one leaves
behind.

#### Criteria for a Good Test Case

Two ways of writing the same rule as a test case can be compared. The weak version reads:
"A member in debt tries to borrow a book, the system gives the appropriate response."
This sentence has three flaws. How much debt is not stated, so the precondition is
ambiguous. Whether it is under or over the limit is unknown, so which equivalence class
it represents is ambiguous. "Appropriate response" is not an observable result, so
whether the test case passes or fails is left to the reader.

The strong version asks the same rule this way: precondition "a member whose debt is
exactly 20 units, with one active loan," step "the member wants to borrow a book sitting
on the shelf," expected result "the request is rejected and the debt limit is given as
the reason."

The criteria fall under four headings. **Single purpose**: one test case checks one
rule; two rules make two test cases, because it must be clear which one broke when it
fails. **Determinism**: it gives the same result with the same precondition; it does not
depend on today's date, a random value, or ordering. **Independence**: it does not rely
on the state left behind by another test case. **Observability**: it is something a
decision can be made about by looking at the result.

#### Test Cases as Data

If a test case's structure is regular, it can be represented as data. This representation
delivers two gains: the test cases stay readable, and the same list produces both a
report and an executable test.

```js
// decision.mjs — loan decision with the boundary defect fixed
export const LOAN_LIMIT = { student: 10, member: 5 };
export const DEBT_LIMIT = 20;

export function loanDecision({ memberType, activeLoans, debt, reservedForAnother }) {
  if (!(memberType in LOAN_LIMIT)) throw new Error(`unknown member type: ${memberType}`);
  if (reservedForAnother) return 'reserved';
  if (debt >= DEBT_LIMIT) return 'debt';
  if (activeLoans >= LOAN_LIMIT[memberType]) return 'limit';
  return 'granted';
}
```

```js
// test-cases.mjs — written test cases represented as data
export const RULES = ['K3', 'K4', 'K13', 'K14', 'K15'];

export const TEST_CASES = [
  {
    id: 'TS-01', rule: 'K4',
    title: 'A member whose loan limit is full is not given a new book',
    precondition: { memberType: 'member', activeLoans: 5, debt: 0, reservedForAnother: false },
    step: 'The member wants to borrow a book sitting on the shelf',
    expected: 'limit',
  },
  {
    id: 'TS-02', rule: 'K13',
    title: 'A member whose debt has reached the limit is not given a book',
    precondition: { memberType: 'member', activeLoans: 1, debt: 20, reservedForAnother: false },
    step: 'The member wants to borrow a book sitting on the shelf',
    expected: 'debt',
  },
  {
    id: 'TS-03', rule: 'K14',
    title: 'A book reserved for someone else is not given out',
    precondition: { memberType: 'student', activeLoans: 0, debt: 0, reservedForAnother: true },
    step: 'The student wants to borrow the book on the reservation list',
    expected: 'reserved',
  },
  {
    id: 'TS-04', rule: 'K3',
    title: 'A student whose limit is not full is given the book',
    precondition: { memberType: 'student', activeLoans: 9, debt: 0, reservedForAnother: false },
    step: 'The student wants to borrow a book sitting on the shelf',
    expected: 'granted',
  },
  {
    id: 'TS-05', rule: 'K3',
    title: 'A student whose loan limit is full is not given a new book',
    precondition: { memberType: 'student', activeLoans: 10, debt: 0, reservedForAnother: false },
    step: 'The student wants to borrow a book sitting on the shelf',
    expected: 'limit',
  },
];
```

The driver that runs the list also produces the traceability report.

```js
// test-case-runner.mjs — running the test cases and the traceability report
import { loanDecision } from './decision.mjs';
import { TEST_CASES, RULES } from './test-cases.mjs';

let matched = 0;
for (const c of TEST_CASES) {
  const observed = loanDecision(c.precondition);
  const result = observed === c.expected ? 'matched' : `DEVIATION (observed ${observed})`;
  console.log(`${c.id} [${c.rule}] ${result}: ${c.title}`);
  if (observed === c.expected) matched += 1;
}

const covered = new Set(TEST_CASES.map((c) => c.rule));
const uncovered = RULES.filter((r) => !covered.has(r));
console.log(`result: ${matched}/${TEST_CASES.length} test cases matched`);
console.log(`traceability: ${covered.size} of ${RULES.length} rules have a test case`);
console.log(`rules without a test case: ${uncovered.join(', ')}`);
```

```
TS-01 [K4] matched: A member whose loan limit is full is not given a new book
TS-02 [K13] matched: A member whose debt has reached the limit is not given a book
TS-03 [K14] matched: A book reserved for someone else is not given out
TS-04 [K3] matched: A student whose limit is not full is given the book
TS-05 [K3] matched: A student whose loan limit is full is not given a new book
result: 5/5 test cases matched
traceability: 4 of 5 rules have a test case
rules without a test case: K15
```

All five of the five test cases matched, but the last line repeats the warning from the black-box
lesson: the off-branch return rule has no test case at all. Because the test case list is written
down, this gap has turned into a countable size.

#### From a Test Case to an Executable Test

The same list can also produce the tests to hand to a runner. The block below walks the
test case list and defines one test per test case, then runs it with Node's built-in
runner. Because the duration fields in the output depend on the machine, only the count
lines are filtered through.

```bash
cat > test-cases.test.mjs <<'EOF'
import test from 'node:test';
import assert from 'node:assert/strict';
import { loanDecision } from './decision.mjs';
import { TEST_CASES } from './test-cases.mjs';

for (const c of TEST_CASES) {
  test(`${c.id} ${c.title}`, () => {
    assert.equal(loanDecision(c.precondition), c.expected);
  });
}
EOF
node --test test-cases.test.mjs 2>&1 | grep -E '^ℹ (tests|pass|fail) '
```

```
ℹ tests 5
ℹ pass 5
ℹ fail 0
```

The move is nothing more than three matchings: precondition to setup data, step to the
call, expected result to the assertion. If a test case does not carry these three parts,
it cannot be turned into an executable test — and if it cannot be turned into one, it
carries the same ambiguity when run by hand.

That the test name starts with the test case identifier is deliberate. The first thing
read when a test fails is which test case broke; the identifier keeps the path from
defect report to test case, and from test case to rule, open.

#### Where a Manually Run Test Case Belongs

Not every test case turns into an automated test. Whether the librarian finds the screen
layout understandable, whether a warning message is clear enough — these are checked
with a written test case, but they require human judgment.

There is one more kind: **exploratory testing**. Here no test case is written in
advance; the person testing chooses the next try based on the result of the previous
one. Exploratory testing does not replace a written test case, it fills its gap — the
question the fourth lesson mentioned as often forgotten is usually found this way. Once
found, it is turned into a written test case and joins the regression set.

#### Summary

- A test case consists of identifier, title, traceability link, precondition, steps, and
  expected result; a postcondition keeps test cases independent of one another.
- The criteria for a good test case are single purpose, determinism, independence, and
  an observable result; "the system gives the appropriate response" is not an observable
  result.
- When test cases are represented as data, the same list produces both a traceability
  report and an executable test.
- The move from test case to test is three matchings: precondition to setup, step to
  call, expected result to assertion.
- In the example all five test cases matched, but it emerged in a countable way that one
  of five rules had no test case at all.
- Exploratory testing does not replace a written test case; what it finds is turned into
  a written test case and joins the regression set.

#### Next Step

A test case's most critical field was passed over quietly: where does the expected
result come from? In the five test cases above the values were read from the
specification, because the rule was written down. If the rule is not written down, if
the computation is too complex to check by hand, or if no one knows the correct result,
how is the expected result determined? The next lesson takes up this question and
builds the source of the expected result with three separate methods, then compares
them.

### The Test Oracle

Source: https://academia.sh/en/courses/testing-fundamentals/test-oracle

The problem of where the expected result comes from and a comparison of three answers — an independent second computation, a metamorphic relation, and a golden file.

In the previous lesson's test cases, the expected result was read from the
specification: the rule was written down, and the test case repeated it. This is the
easiest case. If the rule is not written down, if the computation is too complex to
verify by hand, or if no one fully knows the correct result, where does the expected
result come from?

This question is called the **oracle problem**. A **test oracle** is the source that
decides whether the observed result is correct. Without a test oracle there is no test;
only a program that has been run.

#### The Oracle Problem

An oracle has a cost, and that cost is often higher than the test itself. Calling a
function is cheap; knowing **independently** that the returned value is correct is
expensive.

The most common oracle is the **specified oracle**: the expected value is read from the
specification and written into the test case. Its limit was seen in the second lesson —
if the specification is wrong, the oracle is wrong too, and every test confirms a wrong
truth as correct.

Its second limit is scale. For twenty inputs the expected value can be written by hand;
for eight hundred it cannot. Once the input space grows, the expected value has to be
**generated**.

For comparison, two more rules are added to the fee computation: a book returned to a
different branch adds 5 units, and student members pay half the total, rounded down
since there is no fractional currency.

```js
// fee.mjs — three-rule total fee computation
export function totalFee({ daysLate, memberType, offBranch = false }) {
  const base = Math.min(Math.max(daysLate - 3, 0) * 2, 20);
  const total = base + (offBranch ? 5 : 0);
  return memberType === 'student' ? Math.round(total / 2) : total;
}

export function inputs() {
  const list = [];
  for (const memberType of ['member', 'student']) {
    for (const offBranch of [false, true]) {
      for (let daysLate = 0; daysLate <= 20; daysLate += 1) {
        list.push({ daysLate, memberType, offBranch });
      }
    }
  }
  return list;
}
```

Writing the expected value by hand for eighty-four inputs is not reasonable. Three
separate oracles are built.

#### Independent Second Computation

The first approach computes the same rule **a different way**. If the actual
implementation uses a closed arithmetic form, the second computation sums the days one
by one. Because the two paths differ, the odds that they share the same defect are low.

```js
// oracle-second-computation.mjs — a second computation derived independently of the rule
import { totalFee, inputs } from './fee.mjs';

function reference({ daysLate, memberType, offBranch }) {
  let base = 0;
  for (let day = 1; day <= daysLate; day += 1) {
    if (day <= 3) continue;
    if (base >= 20) break;
    base += 2;
  }
  const total = base + (offBranch ? 5 : 0);
  return memberType === 'student' ? Math.floor(total / 2) : total;
}

const deviations = inputs().filter((g) => totalFee(g) !== reference(g));
console.log(`second computation: ${inputs().length} inputs, ${deviations.length} deviations`);
for (const g of deviations.slice(0, 3)) {
  console.log(`  day=${g.daysLate} ${g.memberType} offBranch=${g.offBranch}: computed ${totalFee(g)}, reference ${reference(g)}`);
}
```

```
second computation: 84 inputs, 21 deviations
  day=0 student offBranch=true: computed 3, reference 2
  day=1 student offBranch=true: computed 3, reference 2
  day=2 student offBranch=true: computed 3, reference 2
```

Twenty-one deviations were found. The defect is in the rounding: the rule says round
down, the implementation rounds to the nearest. When a student and an off-branch return
combine, the total becomes an odd number and the two roundings diverge.

This approach is also called a **pseudo-oracle**. Its cost is plain: doing the same work
twice. For this reason it is applied only in computation-heavy and high-risk areas. It
also has a trap — if the second computation is written by looking at the first, it
inherits the same misunderstanding and the two sides stay silent together.

#### Metamorphic Relation

The second approach never knows the expected value at all. Instead it builds a
**metamorphic relation** that states how the output must change when the input changes
in a specific way. If the relation does not hold, there is a defect; if it holds, only
that relation has been learned about.

```js
// oracle-metamorphic.mjs — relations built without knowing the expected value
import { totalFee, inputs } from './fee.mjs';

const RELATIONS = {
  'MI1 fee does not decrease as delay increases': () => {
    for (const memberType of ['member', 'student']) {
      for (let day = 1; day <= 20; day += 1) {
        const before = totalFee({ daysLate: day - 1, memberType });
        const after = totalFee({ daysLate: day, memberType });
        if (after < before) return `day=${day} ${memberType}: ${before} -> ${after}`;
      }
    }
    return null;
  },
  'MI2 fee does not exceed 25 units': () => {
    for (const g of inputs()) {
      if (totalFee(g) > 25) return `day=${g.daysLate} ${g.memberType}: ${totalFee(g)}`;
    }
    return null;
  },
  'MI3 twice the student fee does not exceed the member fee': () => {
    for (const g of inputs().filter((x) => x.memberType === 'member')) {
      const member = totalFee(g);
      const student = totalFee({ ...g, memberType: 'student' });
      if (student * 2 > member) return `day=${g.daysLate} offBranch=${g.offBranch}: student ${student}, member ${member}`;
    }
    return null;
  },
};

for (const [name, relation] of Object.entries(RELATIONS)) {
  const violation = relation();
  console.log(`${name}: ${violation === null ? 'holds' : `violated — ${violation}`}`);
}
```

```
MI1 fee does not decrease as delay increases: holds
MI2 fee does not exceed 25 units: holds
MI3 twice the student fee does not exceed the member fee: violated — day=0 offBranch=true: student 3, member 5
```

Two of the three relations held, one was violated, and it showed the same defect —
without the expected value being written anywhere. For this reason metamorphic relations
are counted as a **partial oracle**: they can show that a defect exists, but they cannot
confirm correctness.

That two relations held is also information. One relation holding says the class of
defect that relation defines is absent; all of them holding does not say the program is
correct. Because relations hold across the entire input space, they suit use together
with generated inputs.

#### Golden File

The third approach **records** the expected value. The program's output today is
written to a file, and later runs are compared against that file. This file is called a
**golden file**.

```js
// oracle-golden.mjs — using a recorded output as an oracle
import { writeFileSync, readFileSync } from 'node:fs';
import { totalFee, inputs } from './fee.mjs';

const golden = inputs().map((g) => ({ ...g, fee: totalFee(g) }));
writeFileSync('golden.json', JSON.stringify(golden));
console.log(`golden file: ${golden.length} records written`);

const recorded = JSON.parse(readFileSync('golden.json', 'utf8'));
const unchanged = recorded.filter((k) => totalFee(k) !== k.fee);
console.log(`unchanged version: ${unchanged.length} differences`);

function raisesLimitTo25({ daysLate, memberType, offBranch }) {
  const base = Math.min(Math.max(daysLate - 3, 0) * 2, 25);
  const total = base + (offBranch ? 5 : 0);
  return memberType === 'student' ? Math.round(total / 2) : total;
}
const changed = recorded.filter((k) => raisesLimitTo25(k) !== k.fee);
console.log(`version that raises the limit: ${changed.length} differences`);

const defectiveRecord = recorded.find((k) => k.memberType === 'student' && k.offBranch && k.daysLate === 0);
console.log(`golden file's day=0 student offBranch record: ${defectiveRecord.fee} (correct is 2)`);
```

```
golden file: 84 records written
unchanged version: 0 differences
version that raises the limit: 28 differences
golden file's day=0 student offBranch record: 3 (correct is 2)
```

The golden file caught the version that silently raised the cap in twenty-eight records.
This is where it is strong: it detects an **unwanted change** cheaply and removes the
work of writing expected values entirely.

The last line shows its weakness. Because the file was produced from output that already
carried the rounding defect, it recorded the defect too. That defect now counts as
"expected behavior," and fixing it lowers the golden file — whoever fixes it silences the
test by refreshing the file. A golden file does not find defects; it freezes existing
behavior.

For this reason a golden file must be read when it is accepted. A golden file too large
to read is an oracle that has never been checked.

#### Choosing an Oracle

The three approaches do not do the same job, and they are used together.

The specified oracle is used for a small number of high-value test cases: the examples
that define the rule itself. The independent second computation comes into play in
computation-heavy areas and when the input space is large. Metamorphic relations work
where the expected value cannot be known and are most productive together with generated
inputs. The golden file catches unwanted change wherever the output is broad and
structural.

There is also a **human oracle**: the person who knows the correct answer. It is the most
expensive and slowest oracle, but for some questions it is the only one — a program
cannot decide whether a warning message is understandable.

There is one criterion for selection: the oracle must be **independent** of the program
it tests. An oracle derived from the program's own output is the program confirming
itself.

#### Summary

- A test oracle is the source that decides whether the observed result is correct;
  without an oracle there is no test, only a run.
- The specified oracle confirms a wrong truth if the specification is wrong, and it
  cannot be written by hand once the input space grows.
- The independent second computation computes the same rule a different way; in the
  example it found twenty-one deviations in eighty-four inputs, but if written by
  looking at the first, it inherits the same misunderstanding.
- A metamorphic relation shows a defect without ever knowing the expected value; it is a
  partial oracle, and its holding does not confirm correctness.
- A golden file catches unwanted change cheaply, but it freezes the defect present at the
  moment it was produced as expected behavior.
- The oracle's one criterion is independence from the program it tests.

#### Next Step

When an oracle shows a deviation, the work does not end, it begins. The deviation must be
recorded, described in a repeatable way, prioritized, fixed, and the fix confirmed. Each
of these steps is a state, and only specific transitions between them are valid. The
course's final lesson builds this cycle.

### The Error Life Cycle

Source: https://academia.sh/en/courses/testing-fundamentals/defect-life-cycle

The distinction between error, defect, and failure, the fields of a defect report, separating severity from priority, and modeling the report's life cycle as a state machine to measure transition coverage.

When an oracle shows a deviation, the work does not end, it begins. The deviation must
be recorded, described in a repeatable way, prioritized, fixed, and the fix confirmed.
Each of these steps is a state, and only specific transitions between them are valid.

This lesson builds that cycle. The cycle itself is also a system and can be tested with
the techniques from earlier lessons; the final section does exactly that.

#### Error, Defect, and Failure

These three terms are often used as synonyms in everyday language, but they name
separate things.

**Error** is the mistake a person makes: misunderstanding a rule, misreading a boundary,
mistyping an operator. An error occurs in a person, not in a product.

**Defect** is the trace the error leaves in the product: a wrongly written condition, a
missing check, a wrong specification item. A defect sits in the product and is there
even if no one runs it.

**Failure** is the defect showing itself outwardly: the program's observed behavior
deviating from what is expected. Not every defect turns into a failure — if the branch
containing the defect never runs, no failure appears.

The distinction is useful in practice. A defect report describes the failure, because
that is what is observed. The fix targets the defect. The work done so the defect does
not recur looks at the error: does the same misunderstanding exist elsewhere too?

#### Fields of a Defect Report

A good defect report has a single criterion: another person reading the report must be
able to reproduce the failure without asking its author any questions.

The report carries the following fields. **Identifier** and **title**: the title states
the failure in a single sentence, containing an observation, not a rationale.
**Environment**: version, configuration, data set — the context needed to reproduce the
failure. **Precondition and steps**: the same form as in the Writing Test Cases lesson.
**Observed result** and **expected result**: the two are written separately, because the
reader must see the difference. **Evidence**: log entry, output, screenshot.
**Traceability link**: which rule, which test case, which version.

The most often skipped field is the expected result. The sentence "The fee is computed
incorrectly" is not a report, it is a complaint; a report says "for day=0, student,
off-branch return, 3 units were computed; per the rule it should have been 2."

#### Severity and Priority

Two fields are often confused, and confusing them produces the wrong order of work.

**Severity** is the magnitude of the failure's impact: is it data loss, a wrong amount,
a layout defect? It is determined by looking at the product itself and is largely
objective.

**Priority** is when the fix will be made. In addition to impact, it looks at how many
users are affected, whether a workaround exists, the release schedule, and the risk of
the fix.

The two are independent. Data loss in a report that runs once a year can be high
severity, low priority. A typo on the splash screen can be low severity, high priority.
A report format that collapses the two fields into one forces these two decisions to be
confused with each other.

#### The Life Cycle Is a State Machine

The path a report follows is not free. An unverified report cannot be closed, a fix
cannot be attached to a closed report. These constraints form a state machine.

```js
// cycle.mjs — the defect report's state machine
export const TRANSITIONS = {
  new: { assign: 'assigned', reject: 'rejected' },
  assigned: { fix: 'fixed', reject: 'rejected' },
  fixed: { verify: 'verified', reopen: 'reopened' },
  verified: { close: 'closed' },
  closed: { reopen: 'reopened' },
  rejected: { reopen: 'reopened' },
  reopened: { assign: 'assigned' },
};

export const STATES = Object.keys(TRANSITIONS);
export const EVENTS = ['assign', 'fix', 'verify', 'close', 'reject', 'reopen'];

export function apply(state, event) {
  const next = TRANSITIONS[state]?.[event];
  if (next === undefined) throw new Error(`invalid transition: ${state} + ${event}`);
  return next;
}
```

Once the machine is written down, valid and invalid transitions can be counted.

```js
// count.mjs — counting every state-event pair
import { STATES, EVENTS, TRANSITIONS, apply } from './cycle.mjs';

let valid = 0;
let rejectedCount = 0;
for (const state of STATES) {
  for (const event of EVENTS) {
    if (TRANSITIONS[state][event] !== undefined) { valid += 1; continue; }
    try { apply(state, event); } catch { rejectedCount += 1; }
  }
}

const total = STATES.length * EVENTS.length;
console.log(`state count      : ${STATES.length}`);
console.log(`event count      : ${EVENTS.length}`);
console.log(`all pairs        : ${total}`);
console.log(`valid transition : ${valid}`);
console.log(`invalid pair     : ${total - valid}, rejected: ${rejectedCount}`);
```

```
state count      : 7
event count      : 6
all pairs        : 42
valid transition : 10
invalid pair     : 32, rejected: 32
```

Only ten of the forty-two pairs are valid; the remaining thirty-two must be rejected,
and are. This second number is often overlooked. That a report can be closed without
ever reaching the "verified" state is a defect, but no happy-path test case shows it.

That **verified** is a separate step is also deliberate. The person who makes the fix
cannot close the report; the authority to close belongs to the party that reported or
verified the failure. The confirmation test runs at this step.

#### State Transition Testing

The technique deferred in the fourth lesson is applied here. **State transition
testing** is used when what needs to be tried is not values but transitions, and its
measure is **transition coverage**: how many of the valid transitions have been
exercised at least once?

```js
// transition-coverage.mjs — transition coverage of two test-case sets
import { TRANSITIONS, apply } from './cycle.mjs';

const ALL_TRANSITIONS = Object.entries(TRANSITIONS)
  .flatMap(([state, events]) => Object.keys(events).map((event) => `${state}+${event}`));

function coverage(paths) {
  const seen = new Set();
  for (const path of paths) {
    let state = 'new';
    for (const event of path) {
      seen.add(`${state}+${event}`);
      state = apply(state, event);
    }
  }
  return seen;
}

const HAPPY_PATH = [['assign', 'fix', 'verify', 'close']];
const WIDE_SET = [
  ['assign', 'fix', 'verify', 'close', 'reopen', 'assign'],
  ['assign', 'fix', 'reopen', 'assign'],
  ['reject', 'reopen', 'assign', 'reject'],
];

for (const [label, paths] of [['happy path', HAPPY_PATH], ['wide set', WIDE_SET]]) {
  const seen = coverage(paths);
  const missing = ALL_TRANSITIONS.filter((g) => !seen.has(g));
  console.log(`${label.padEnd(10)}: ${seen.size}/${ALL_TRANSITIONS.length} transitions exercised`);
  if (missing.length) console.log(`  not exercised: ${missing.join(', ')}`);
}
```

```
happy path: 4/10 transitions exercised
  not exercised: new+reject, assigned+reject, fixed+reopen, closed+reopen, rejected+reopen, reopened+assign
wide set  : 10/10 transitions exercised
```

The happy path exercises forty percent of the transitions. All six untried transitions
are unusual flows: rejection, reopening, verification failing. These are the flows that
draw the most disagreement in real use, and for exactly that reason are the least
tested.

Adding three paths completed the coverage. The measurement also said which paths needed
to be added — the list of missing transitions is directly the list of test cases still
to be written.

#### Summary

- Error is the mistake a person makes, defect is the trace it leaves in the product,
  failure is the observed outward expression of a defect; not every defect turns into a
  failure.
- The criterion for a defect report is that another reader can reproduce the failure
  without asking questions; observed and expected results are written separately.
- Severity states the failure's impact, priority states when the fix will happen; the
  two are independent, and collapsing them into one field confuses the decisions.
- The life cycle is a state machine; in the example, ten of forty-two state-event pairs
  are valid and thirty-two must be rejected.
- State transition testing's measure is transition coverage; the happy path exercised
  four of ten transitions, and the list of untried ones gave the list of test cases to
  write.

#### Course Wrap-Up

The course answered two questions. The first was **what** testing is. Quality was
defined as a judgment of fitness; product quality was separated from process quality;
verification was pulled apart from validation, and it was shown that a program fully
conforming to its specification can still be wrong. Non-functional expectations were
named and turned into measurable criteria. It was shown that testing is an attitude
before it is a set of techniques, and that two mindsets looking at the same code find
different things. It was shown that the delivery model does not make testing cheaper, it
only determines when the same check is performed.

The second was **which** test to write. Tests were classified by the boundary they
cover, the knowledge they rely on, and their purpose. Input selection was taken out of
intuition and grounded with equivalence classes, boundary values, and the decision
table. What was to be tested was given a test-case form, and the move from that form to
an executable test was made. The question of where the expected result comes from was
taken up as a separate problem, and three oracles were compared. Finally, a found
deviation's path from report to closure was built as a state machine.

The unanswered question is **how** to write a test. What is a check's internal
structure; how are arrange, act, and assert separated? What should an assertion say, and
what should it not say? How is a unit tested without touching the outside world — how
are dependencies replaced with test doubles, and which kind is used where? What does
coverage measurement say, and which question does it not answer? How does writing the
test before the code change the design?

The next course, **Unit Testing and Test-Driven Development**, takes up these questions.
Writing fast and independent tests, building meaningful assertions, supplying
dependencies from outside, using the right kind of test double in the right place,
reading statement and branch coverage, and the red–green–refactor cycle are built there. The
tests selected in this course are the tests that will be written there.