The Testing Mindset

In the previous two lessons, the defect surfaced only because someone went looking for it: because the desk records were compared, because the budget was measured. If no one had made that comparison, neither defect would have appeared. Testing’s techniques come after this; first comes the question of what the person looking is looking for.

This lesson compares two views on the same function. The first wants to show that the program works, the second searches for where it breaks. Both write tests, both are honest, and what they find is different.

Two Views Look at the Same Code

A loan decision is added to the library. The written rule has three items: no loan is given once active loans reach the member type’s limit, no loan is given to a member whose debt has reached 20 units, and an unknown member type is rejected.

// loan.mjs — first implementation of the loan decision
const RULES = { student: { maxLoans: 10 }, member: { maxLoans: 5 } };
export const DEBT_LIMIT = 20;

export function canBorrow(memberType, activeLoans, debt) {
  const limit = RULES[memberType]?.maxLoans ?? 5;
  if (activeLoans > limit) return false;
  if (debt > DEBT_LIMIT) return false;
  return true;
}

The loop that was repeated across the previous lessons for running checks is moved into its own file.

// runner.mjs — shared driver that runs a list of checks
export function run(label, checks) {
  let passed = 0;
  const failed = [];
  for (const [name, check] of Object.entries(checks)) {
    try { check(); passed += 1; } catch { failed.push(name); }
  }
  console.log(`${label}: ${passed}/${Object.keys(checks).length} passed`);
  for (const name of failed) console.log(`  failed: ${name}`);
}

Two check suites are written. The first shows that the rule works as described: a student with three books borrows, a member with ten books cannot. The second probes the edges of the same rule.

// two-views.mjs — comparing happy-path checks with questioning checks
import assert from 'node:assert/strict';
import { canBorrow } from './loan.mjs';
import { run } from './runner.mjs';

const HAPPY_PATH = {
  'student with three books borrows a new book': () => assert.equal(canBorrow('student', 3, 0), true),
  'member with one book borrows a new book': () => assert.equal(canBorrow('member', 1, 0), true),
  'student with twenty books cannot borrow': () => assert.equal(canBorrow('student', 20, 0), false),
  'member with ten books cannot borrow': () => assert.equal(canBorrow('member', 10, 0), false),
  'member with a debt of fifty cannot borrow': () => assert.equal(canBorrow('member', 0, 50), false),
};

const QUESTIONING = {
  'member exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 5, 0), false),
  'student exactly at the limit cannot borrow': () => assert.equal(canBorrow('student', 10, 0), false),
  'debt exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 0, 20), false),
  'unknown member type is rejected': () => assert.throws(() => canBorrow('guest', 0, 0)),
  'negative loan count is rejected': () => assert.throws(() => canBorrow('member', -1, 0)),
};

run('happy path ', HAPPY_PATH);
run('questioning', QUESTIONING);

happy path : 5/5 passed
questioning: 0/5 passed
  failed: member exactly at the limit cannot borrow
  failed: student exactly at the limit cannot borrow
  failed: debt exactly at the limit cannot borrow
  failed: unknown member type is rejected
  failed: negative loan count is rejected

The first suite paints a flawless picture. The second suite shows five separate defects in the same ten-line function: two comparison operators have shifted the limit by one unit, the debt limit carries the same shift, an unknown member type is silently counted as a member, and an invalid number is never checked at all.

All of the defects fell outside the first suite’s scope. The reason is not that the first suite was poorly written; every question it asked was chosen from points far from the boundary.

Confirmation Bias

The person who wrote the first suite read the rule and chose examples that show the rule working. This is what confirmation bias looks like in test writing: searching for evidence consistent with a claim instead of testing the claim.

The source of the bias is how the question is framed. The question “can a student borrow a book?” looks for an example that can. The question “under what condition can a student not borrow, and where is the boundary of that condition?” requires probing both sides of the boundary. The second question takes more work and finds more defects.

This is where the mindset is defined: the person writing tests does not assume the program is correct; they look for evidence that would test the claim that it is. If they do not find any, the claim stands for the time being.

Where Questioning Questions Come From

The questioning checks were not generated at random. Each came from a question class:

• Boundary: if a threshold is crossed, the threshold itself, one below it, and one above it are tried. Three of the five defects came from this class.
• Gap: what happens if a value is missing, not found, or could not be computed? The unknown member type belongs to this class.
• Type and domain: what happens if a negative number, a fractional number, or a non-numeric value arrives where a number is expected?
• Order: what if operations arrive in an unexpected sequence? Lending out a book that has not been returned yet belongs to this class.
• Repetition: if the same operation arrives twice, does the total double?
• Scale: if the input grows a thousandfold, does the behavior stay the same?
• Authorization: does the person performing the operation have the right to do so?

This list is used as a checklist. A checklist’s function is not to find the defect but to remind you of the question you forgot to ask.

The Same Questions’ Returns Diminish

The five defects are fixed. The comparisons are corrected to include equality at the boundary, an unknown type raises an error, and an invalid number is rejected.

// loan2.mjs — after the fixes found by the questioning checks
const RULES = { student: { maxLoans: 10 }, member: { maxLoans: 5 } };
export const DEBT_LIMIT = 20;

export function canBorrow(memberType, activeLoans, debt) {
  const rule = RULES[memberType];
  if (rule === undefined) throw new Error(`unknown member type: ${memberType}`);
  if (!Number.isInteger(activeLoans) || activeLoans < 0) throw new Error('invalid active loan count');
  if (activeLoans >= rule.maxLoans) return false;
  if (debt >= DEBT_LIMIT) return false;
  return true;
}

The same questioning suite is run again, and a new question class is added alongside it: what happens when the input is not a number?

// new-class.mjs — the same questions repeated, and a new question class compared
import assert from 'node:assert/strict';
import { canBorrow } from './loan2.mjs';
import { run } from './runner.mjs';

const QUESTIONING = {
  'member exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 5, 0), false),
  'student exactly at the limit cannot borrow': () => assert.equal(canBorrow('student', 10, 0), false),
  'debt exactly at the limit cannot borrow': () => assert.equal(canBorrow('member', 0, 20), false),
  'unknown member type is rejected': () => assert.throws(() => canBorrow('guest', 0, 0)),
  'negative loan count is rejected': () => assert.throws(() => canBorrow('member', -1, 0)),
};

const NEW_QUESTION_CLASS = {
  'fractional loan count is rejected': () => assert.throws(() => canBorrow('member', 2.5, 0)),
  'no loan is given if debt could not be computed': () => assert.throws(() => canBorrow('member', 0, Number.NaN)),
};

run('old questions', QUESTIONING);
run('new questions', NEW_QUESTION_CLASS);

old questions: 5/5 passed
new questions: 1/2 passed
  failed: no loan is given if debt could not be computed

The old questions no longer find anything. This is an expected outcome: once a defect is fixed, the test that found it turns into a sentry waiting for that defect to come back — valuable, but no longer producing new information. A test suite’s defect-finding return diminishes as it is repeated; this is called the pesticide paradox. New defects are only found with new question classes.

The sixth defect found is instructive too: when the debt cannot be computed, the value is not a number, so every comparison comes out false and the member borrows anyway. Defects clustering in the same region is not unusual either; if a defect is concentrating somewhere, deepening the search there is productive. This is called defect clustering.

Limits of the Mindset

The questioning stance does not mean that every failing test is a defect. A warning that comes up because the test itself was written wrong, or because the expected value is stale, is a false positive and erodes trust. In the opposite direction, a passing test might have missed a defect. Both kinds of error lower the test suite’s information value.

All tests passing is not, by itself, grounds for shipping either. If the product runs flawlessly and does the wrong thing, the test result says nothing about that; this is where the validation question set apart in the second lesson belongs. This situation is called the absence-of-errors fallacy.

The final limit is personal. A person testing their own code looks through the assumptions they made while writing it; an input that never occurred to them while writing does not occur to them while testing either. The remedy for this is not abandoning the mindset but multiplying the viewpoint: review, having a second person write scenarios, systematic application of technique. The next topic is devoted to these techniques.

Summary

• On the same function, the happy-path checks passed five for five while the checks asking boundary and gap questions revealed five separate defects.
• Confirmation bias is searching for evidence consistent with a claim instead of testing the claim; how the question is framed determines which evidence gets sought.
• Questioning questions are generated systematically from the boundary, gap, type, order, repetition, scale, and authorization classes.
• Once a defect is fixed, the test that found it turns into a sentry; a repeated suite’s defect-finding return diminishes, and a new defect calls for a new question class.
• False positives and missed defects lower a test suite’s information value in both directions; all tests passing does not show that the wrong thing was not done.

Next Step

Up to this point, testing has been treated as an activity where one person is alone with one function. In reality, when a test is written, who writes it, and how quickly its result reaches someone depends on the way of working. The same check suite carries different value when it is run once at the end of a release versus when it is run on every change. The next lesson examines how the delivery model determines testing practice and what the feedback point changes.