The Error Life Cycle

When an oracle shows a deviation, the work does not end, it begins. The deviation must be recorded, described in a repeatable way, prioritized, fixed, and the fix confirmed. Each of these steps is a state, and only specific transitions between them are valid.

This lesson builds that cycle. The cycle itself is also a system and can be tested with the techniques from earlier lessons; the final section does exactly that.

Error, Defect, and Failure

These three terms are often used as synonyms in everyday language, but they name separate things.

Error is the mistake a person makes: misunderstanding a rule, misreading a boundary, mistyping an operator. An error occurs in a person, not in a product.

Defect is the trace the error leaves in the product: a wrongly written condition, a missing check, a wrong specification item. A defect sits in the product and is there even if no one runs it.

Failure is the defect showing itself outwardly: the program’s observed behavior deviating from what is expected. Not every defect turns into a failure — if the branch containing the defect never runs, no failure appears.

The distinction is useful in practice. A defect report describes the failure, because that is what is observed. The fix targets the defect. The work done so the defect does not recur looks at the error: does the same misunderstanding exist elsewhere too?

Fields of a Defect Report

A good defect report has a single criterion: another person reading the report must be able to reproduce the failure without asking its author any questions.

The report carries the following fields. Identifier and title: the title states the failure in a single sentence, containing an observation, not a rationale. Environment: version, configuration, data set — the context needed to reproduce the failure. Precondition and steps: the same form as in the Writing Test Cases lesson. Observed result and expected result: the two are written separately, because the reader must see the difference. Evidence: log entry, output, screenshot. Traceability link: which rule, which test case, which version.

The most often skipped field is the expected result. The sentence “The fee is computed incorrectly” is not a report, it is a complaint; a report says “for day=0, student, off-branch return, 3 units were computed; per the rule it should have been 2.”

Severity and Priority

Two fields are often confused, and confusing them produces the wrong order of work.

Severity is the magnitude of the failure’s impact: is it data loss, a wrong amount, a layout defect? It is determined by looking at the product itself and is largely objective.

Priority is when the fix will be made. In addition to impact, it looks at how many users are affected, whether a workaround exists, the release schedule, and the risk of the fix.

The two are independent. Data loss in a report that runs once a year can be high severity, low priority. A typo on the splash screen can be low severity, high priority. A report format that collapses the two fields into one forces these two decisions to be confused with each other.

The Life Cycle Is a State Machine

The path a report follows is not free. An unverified report cannot be closed, a fix cannot be attached to a closed report. These constraints form a state machine.

// cycle.mjs — the defect report's state machine
export const TRANSITIONS = {
  new: { assign: 'assigned', reject: 'rejected' },
  assigned: { fix: 'fixed', reject: 'rejected' },
  fixed: { verify: 'verified', reopen: 'reopened' },
  verified: { close: 'closed' },
  closed: { reopen: 'reopened' },
  rejected: { reopen: 'reopened' },
  reopened: { assign: 'assigned' },
};

export const STATES = Object.keys(TRANSITIONS);
export const EVENTS = ['assign', 'fix', 'verify', 'close', 'reject', 'reopen'];

export function apply(state, event) {
  const next = TRANSITIONS[state]?.[event];
  if (next === undefined) throw new Error(`invalid transition: ${state} + ${event}`);
  return next;
}

Once the machine is written down, valid and invalid transitions can be counted.

// count.mjs — counting every state-event pair
import { STATES, EVENTS, TRANSITIONS, apply } from './cycle.mjs';

let valid = 0;
let rejectedCount = 0;
for (const state of STATES) {
  for (const event of EVENTS) {
    if (TRANSITIONS[state][event] !== undefined) { valid += 1; continue; }
    try { apply(state, event); } catch { rejectedCount += 1; }
  }
}

const total = STATES.length * EVENTS.length;
console.log(`state count      : ${STATES.length}`);
console.log(`event count      : ${EVENTS.length}`);
console.log(`all pairs        : ${total}`);
console.log(`valid transition : ${valid}`);
console.log(`invalid pair     : ${total - valid}, rejected: ${rejectedCount}`);

state count      : 7
event count      : 6
all pairs        : 42
valid transition : 10
invalid pair     : 32, rejected: 32

Only ten of the forty-two pairs are valid; the remaining thirty-two must be rejected, and are. This second number is often overlooked. That a report can be closed without ever reaching the “verified” state is a defect, but no happy-path test case shows it.

That verified is a separate step is also deliberate. The person who makes the fix cannot close the report; the authority to close belongs to the party that reported or verified the failure. The confirmation test runs at this step.

State Transition Testing

The technique deferred in the fourth lesson is applied here. State transition testing is used when what needs to be tried is not values but transitions, and its measure is transition coverage: how many of the valid transitions have been exercised at least once?

// transition-coverage.mjs — transition coverage of two test-case sets
import { TRANSITIONS, apply } from './cycle.mjs';

const ALL_TRANSITIONS = Object.entries(TRANSITIONS)
  .flatMap(([state, events]) => Object.keys(events).map((event) => `${state}+${event}`));

function coverage(paths) {
  const seen = new Set();
  for (const path of paths) {
    let state = 'new';
    for (const event of path) {
      seen.add(`${state}+${event}`);
      state = apply(state, event);
    }
  }
  return seen;
}

const HAPPY_PATH = [['assign', 'fix', 'verify', 'close']];
const WIDE_SET = [
  ['assign', 'fix', 'verify', 'close', 'reopen', 'assign'],
  ['assign', 'fix', 'reopen', 'assign'],
  ['reject', 'reopen', 'assign', 'reject'],
];

for (const [label, paths] of [['happy path', HAPPY_PATH], ['wide set', WIDE_SET]]) {
  const seen = coverage(paths);
  const missing = ALL_TRANSITIONS.filter((g) => !seen.has(g));
  console.log(`${label.padEnd(10)}: ${seen.size}/${ALL_TRANSITIONS.length} transitions exercised`);
  if (missing.length) console.log(`  not exercised: ${missing.join(', ')}`);
}

happy path: 4/10 transitions exercised
  not exercised: new+reject, assigned+reject, fixed+reopen, closed+reopen, rejected+reopen, reopened+assign
wide set  : 10/10 transitions exercised

The happy path exercises forty percent of the transitions. All six untried transitions are unusual flows: rejection, reopening, verification failing. These are the flows that draw the most disagreement in real use, and for exactly that reason are the least tested.

Adding three paths completed the coverage. The measurement also said which paths needed to be added — the list of missing transitions is directly the list of test cases still to be written.

Summary

• Error is the mistake a person makes, defect is the trace it leaves in the product, failure is the observed outward expression of a defect; not every defect turns into a failure.
• The criterion for a defect report is that another reader can reproduce the failure without asking questions; observed and expected results are written separately.
• Severity states the failure’s impact, priority states when the fix will happen; the two are independent, and collapsing them into one field confuses the decisions.
• The life cycle is a state machine; in the example, ten of forty-two state-event pairs are valid and thirty-two must be rejected.
• State transition testing’s measure is transition coverage; the happy path exercised four of ten transitions, and the list of untried ones gave the list of test cases to write.

Course Wrap-Up

The course answered two questions. The first was what testing is. Quality was defined as a judgment of fitness; product quality was separated from process quality; verification was pulled apart from validation, and it was shown that a program fully conforming to its specification can still be wrong. Non-functional expectations were named and turned into measurable criteria. It was shown that testing is an attitude before it is a set of techniques, and that two mindsets looking at the same code find different things. It was shown that the delivery model does not make testing cheaper, it only determines when the same check is performed.

The second was which test to write. Tests were classified by the boundary they cover, the knowledge they rely on, and their purpose. Input selection was taken out of intuition and grounded with equivalence classes, boundary values, and the decision table. What was to be tested was given a test-case form, and the move from that form to an executable test was made. The question of where the expected result comes from was taken up as a separate problem, and three oracles were compared. Finally, a found deviation’s path from report to closure was built as a state machine.

The unanswered question is how to write a test. What is a check’s internal structure; how are arrange, act, and assert separated? What should an assertion say, and what should it not say? How is a unit tested without touching the outside world — how are dependencies replaced with test doubles, and which kind is used where? What does coverage measurement say, and which question does it not answer? How does writing the test before the code change the design?

The next course, Unit Testing and Test-Driven Development, takes up these questions. Writing fast and independent tests, building meaningful assertions, supplying dependencies from outside, using the right kind of test double in the right place, reading statement and branch coverage, and the red–green–refactor cycle are built there. The tests selected in this course are the tests that will be written there.