Testing Frameworks

The dependency is locked and the installed version is now fixed. Even so, the same code, the same lock file and the same tests can pass in one run and fail in another. The observation of this lesson is the test result row: the base is [], so no test fails, but the evidence column shows D- O+ E-. The result changes when the order changes.

M10/K05 asked about the limit of an assurance: what a test catches and what it misses. Here the question is different. What determines whether a test passes? There the limit was the coverage of the assurance; here the limit is an input outside the test itself. The core’s determinant.rb file is given again exactly; the block is source and is pulled in with require_relative.

# determinant.rb — a run model and a helper that classifies what determines a result

# Run: the three inputs a result may depend on. Everything outside these three is code.
Run = Data.define(:lock, :seed, :environment)

BASE = Run.new(
  lock: { "formatter" => "1.4.2", "parser" => "2.0.1" },
  seed: 1,
  environment: { clock: Time.utc(2026, 10, 16, 9, 0), cores: 4, variables: {} }
)

# Each variant changes a single input; the others stay as in the base.
VARIANTS = {
  dependency: BASE.with(lock: { "formatter" => "1.5.0", "parser" => "2.1.0" }),
  order: BASE.with(seed: 2),
  environment: BASE.with(environment: { clock: Time.utc(2027, 3, 1, 9, 0), cores: 8,
                                        variables: { "WORKER_COUNT" => "3" } })
}

# Rule: the computation runs first in the base, then once in each variant. The first variant
# that changes the result (dependency, order, environment, in that order) is the determinant;
# if none changes it, code.
def determinant(computation)
  base = computation.call(BASE)
  changed = VARIANTS.to_h { |name, run| [name, computation.call(run) != base] }
  kind = changed.key(true)&.to_s || "code"
  evidence = changed.map { |name, c| "#{name[0].upcase}#{c ? '+' : '-'}" }.join(" ")
  [kind, evidence, base]
end

The Three Parts of a Framework

A test framework consists of three parts. A test definition is a named block that is a runnable trial. An assertion compares the expected value with the actual one and raises an error if they do not match. A test runner runs the definitions, catches the assertion error and writes the result to a table. Common test frameworks are built on these three parts too. Here all three are built in pure Ruby, because the observation lies inside the runner and the runner must be visible.

# framework.rb — the three parts of a test framework: definition, assertion, runner
class AssertionFailed < StandardError; end

# Assertion: silent if the expected and the actual value are the same, otherwise raises an error.
def assertion(expected, actual)
  raise AssertionFailed, "assertion failed" unless expected == actual
end

# Runner: runs each test, catches the assertion error, writes the result to the table.
def run_suite(tests)
  tests.to_h do |name, test|
    test.call
    [name, :passed]
  rescue AssertionFailed
    [name, :failed]
  end
end

The file is source. The file below runs three tests. The second one expects integer division wrongly.

# small.rb — the three parts come together
require_relative "framework"

tests = {
  "addition" => -> { assertion(5, 2 + 3) },
  "division" => -> { assertion(3.5, 7 / 2) },
  "decimal" => -> { assertion(3.5, 7 / 2.0) }
}
p run_suite(tests)

{"addition" => :passed, "division" => :failed, "decimal" => :passed}

7 / 2 is integer division and gives 3; the second assertion did not hold, the runner caught the error and wrote the division test as :failed. The other two passed. Note that the runner catches only AssertionFailed. Any other error, for example division by zero, is not an assertion result and does not enter the table.

# error.rb — an error that is not an assertion error does not enter the table
require_relative "framework"

begin
  run_suite("division by zero" => -> { 1 / 0 })
rescue => e
  p e.class
end

ZeroDivisionError

The runner did not catch ZeroDivisionError; the error escaped. This distinction matters: a test failing and a test erroring out are different events, and this lesson measures only the first. PG13. The framework built here shows the three parts; the expectation library, the setup and the cleanup steps of a real framework lie outside this model.

Two Writing Styles

Common frameworks offer the same three parts in two different writing styles. In the first, tests are methods of a class whose names start with test_, and an assertion is a method call. In the second, tests are named examples inside a definition block, and an assertion is an expectation chain that starts from the actual value. The file below writes the same two tests in both styles and hands both to the same runner.

# styles.rb — the same two tests, two writing styles, one runner
require_relative "framework"

# First style: a test class; every method starting with test_ is a test.
class DivisionTests
  def test_integer
    assertion(3, 7 / 2)
  end

  def test_decimal
    assertion(3.5, 7 / 2)
  end
end

class_tests = DivisionTests.public_instance_methods(false).grep(/\Atest_/).sort.to_h do |name|
  [name.to_s, -> { DivisionTests.new.public_send(name) }]
end

# Second style: a definition block and an expectation chain.
Expectation = Struct.new(:actual) do
  def to_be(expected)
    assertion(expected, actual)
  end
end

def expect(actual)
  Expectation.new(actual)
end

class Definition
  attr_reader :tests

  def initialize(subject)
    @subject = subject
    @tests = {}
  end

  def example(name, &block)
    @tests["#{@subject} #{name}"] = block
  end
end

def define(subject, &body)
  definition = Definition.new(subject)
  definition.instance_eval(&body)
  definition.tests
end

block_tests = define("division") do
  example("integer") { expect(7 / 2).to_be(3) }
  example("decimal") { expect(7 / 2).to_be(3.5) }
end

p run_suite(class_tests)
p run_suite(block_tests)

{"test_decimal" => :failed, "test_integer" => :passed}
{"division integer" => :passed, "division decimal" => :failed}

The two tables give the same result: the test that expects 3.5 from integer division failed in both styles. The difference lies in the writing. In the first style the test name is the method name, and the tests are gathered in a class. In the second style the test name is a string built from the subject and the example name. In both cases the runner sees the same name → block table; this is why the order and the seed effect measured in the following sections belong to the runner. PG13a. The two styles were compared here in a single example with two tests; the setup, nested grouping and reporting differences of real frameworks lie outside this comparison.

The Observation’s Runner

The runner in the observation row is smaller but does one more thing: it shuffles the tests with the seed and gives all of them the same settings table. The file below carries the TESTS and run_tests definitions taken from observations.rb exactly.

# tests.rb — the tests and the runner of the observation row
# Test runner: tests run in an order shuffled by the seed and share the same settings.
TESTS = {
  "price_format" => ->(settings) { "12 #{settings[:currency]}" == "12 TRY" },
  "currency_conversion" => ->(settings) { settings[:currency] = "USD"; true },
  "invoice_total" => ->(settings) { settings[:currency] == "TRY" }
}

def run_tests(run, tests = TESTS)
  settings = { currency: "TRY" }
  order = tests.keys.shuffle(random: Random.new(run.seed))
  order.reject { |name| tests[name].call(settings) }
end

There are three tests. price_format looks at the currency in the settings and expects TRY. currency_conversion sets the currency in the settings to USD and returns true. invoice_total expects TRY again. The runner returns the names of the failing tests. The file below prints the shuffled order and the failing tests side by side for six seeds.

# seeds.rb — six seeds, six orders
require_relative "determinant"
require_relative "tests"

(1..6).each do |seed|
  order = TESTS.keys.shuffle(random: Random.new(seed))
  puts "#{seed} #{order.inspect} #{run_tests(BASE.with(seed: seed)).inspect}"
end

1 ["price_format", "invoice_total", "currency_conversion"] []
2 ["invoice_total", "currency_conversion", "price_format"] ["price_format"]
3 ["currency_conversion", "price_format", "invoice_total"] ["price_format", "invoice_total"]
4 ["currency_conversion", "price_format", "invoice_total"] ["price_format", "invoice_total"]
5 ["price_format", "currency_conversion", "invoice_total"] ["invoice_total"]
6 ["price_format", "currency_conversion", "invoice_total"] ["invoice_total"]

The order of seed 1 puts currency_conversion last; since no test runs after it, no test fails. In seed 2 currency_conversion is in the middle, and price_format, which comes after it, fails. Seeds 3 and 4 gave the same order: currency_conversion first, and the two tests after it also failed. Seeds 5 and 6 share an order too; currency_conversion is in the middle and only invoice_total, which comes after it, fails. Four different orders came out of six seeds, and the result of each order is the same as long as the seed stays fixed.

Shared State

The flaw is not in the order but in what makes the order visible as a flaw. currency_conversion changes the shared settings table. This change is the input of every test that runs after it. That the table is shared among the tests can be shown directly.

# shared.rb — the setting a test wrote is the input of the test that comes after it
require_relative "tests"

settings = { currency: "TRY" }
TESTS["currency_conversion"].call(settings)
p settings
p TESTS["price_format"].call(settings)
p TESTS["price_format"].call({ currency: "TRY" })

{currency: "USD"}
false
true

After currency_conversion the settings became USD. The same price_format test returned false with these settings and true with fresh settings. The test code is the same; what changed is the starting state of the test. This is shared state, and it is the most common source of a flaky test: a test is fine, but it depends on who ran before it. PG14. The measurement holds only for a model that shares a settings table; shared state through the file system, the network or a global variable behaves the same way, but it was not measured here.

Which Test Fails on How Many Seeds

The result of a single seed is a diagnosis; the count over six seeds gives a profile. The file below counts the number of tests each seed fails, the number of seeds on which each test fails and the number of seeds on which at least one test fails.

# counts.rb — the distribution of failures by seed
require_relative "determinant"
require_relative "tests"

results = (1..6).map { |s| run_tests(BASE.with(seed: s)) }
p results.map(&:size)
p results.flatten.tally
p results.count { |f| !f.empty? }

[0, 1, 2, 2, 1, 1]
{"price_format" => 3, "invoice_total" => 4}
5

In five of the six runs at least one test failed; only seed 1 passed cleanly. currency_conversion never failed, but it is the test that makes the other two fail. invoice_total failed on more seeds because currency_conversion came before it more often; price_format failed less. This table does not point at the culprit, it points at the victim. The failing test is not the faulty test itself; the faulty test passes silently and makes someone else fail. How often a test fails measures not its own soundness but who runs before it. For this reason, when reading the code of a failing test, the first question should be what the failing test shares. PG15. The count is for the six seeds of a single suite; it does not follow from this measurement that the failure frequency would have the same proportions in a real team’s suite.

Fixing the order does not solve this problem, it hides it. If seed 1 were used in every run, this suite would never fail and the shared state would stay unnoticed. Shuffling does not produce a defect; it makes an existing dependency visible. And to the extent that it can make it visible, printing the seed of the failing test turns shuffling from a suspicion into a diagnosis.

Reproducing with the Seed

A flaky test is a bug report if it can be reproduced at the moment it fails; if it cannot be reproduced, it is a rumor. When the runner fixes the seed, the failing test fails in the same order every time. For this reason a good runner prints the seed: when a run fails, the seed is the key to reproduction.

# replay.rb — reproduce a failing test with the seed
require_relative "determinant"
require_relative "tests"

results = (1..6).map { |s| run_tests(BASE.with(seed: s)) }
p results.count([])
p run_tests(VARIANTS[:order])
p run_tests(VARIANTS[:order]) == run_tests(BASE.with(seed: 2))
p determinant(->(r) { run_tests(r) })

1
["price_format"]
true
["order", "D- O+ E-", []]

On only one of the six seeds did no test fail: seed 1 passed while seed 2 failed. Since VARIANTS[:order] is seed 2, it gave ["price_format"], and the second call with the same seed gave the same result as the first. The last line shows the classification: the result comes from order, the evidence is D- O+ E-, the base is []. The rule wrote the base result as []; the failing test is not visible in the base run, only in the order variant. The determinant of this lesson is order. PG16. Seed 1 showing “passed” does not show that the tests are sound; it shows only that in that order the test that corrupts the settings table fell to the end.

The Bounding Measurement

The class is tested from two directions. The first: what happens if each test runs with its own copy of the settings? The second: the rule tries only two seeds, so do these two seeds catch every flakiness?

# isolation.rb — each test gets its own copy of the settings
require_relative "determinant"
require_relative "tests"

def isolated_run_tests(run, tests = TESTS)
  settings = { currency: "TRY" }
  order = tests.keys.shuffle(random: Random.new(run.seed))
  order.reject { |name| tests[name].call(settings.dup) }
end

p (1..6).map { |s| isolated_run_tests(BASE.with(seed: s)) }
p determinant(->(r) { isolated_run_tests(r) })

[[], [], [], [], [], []]
["code", "D- O- E-", []]

All six seeds gave [] and this time the rule said code, with the evidence D- O- E-. The same three tests, the same six seeds; the only difference is that each test takes its own copy. The class moved from order to code: isolation removes the effect of the order, because the tests are no longer each other’s input. The limit of isolation is the depth of the copy.

# shallow.rb — dup is shallow
settings = { list: [] }
copy = settings.dup
copy[:list] << 1
p settings
p settings[:list].equal?(copy[:list])

{list: [1]}
true

dup copied the table but not the array inside it: the two tables hold the same array, and the change in the copy passed to the original. Because the settings table carries only simple values, the isolation of this lesson worked. For nested settings dup is not enough. PG17. The isolation measurement holds only for a flat table.

The second direction is the rule’s own blind spot. The rule tries the base and the order variant, that is, seed 1 and seed 2. A suite can be built that passes on seed 2 but fails on seed 5. It is enough to replace the price_format test with a test that never looks at the settings.

# blind.rb — where the rule tries only seeds 1 and 2
require_relative "determinant"
require_relative "tests"

BLIND_TESTS = TESTS.merge("price_format" => ->(_settings) { true })

p determinant(->(r) { run_tests(r, BLIND_TESTS) })
results = (1..6).map { |s| run_tests(BASE.with(seed: s), BLIND_TESTS) }
p results

["code", "D- O- E-", []]
[[], [], ["invoice_total"], ["invoice_total"], ["invoice_total"], ["invoice_total"]]

The rule classified this suite as code, because seeds 1 and 2 both gave []. But on four of the six seeds invoice_total failed. The result “not order” was stated only for the two tried seeds. The suite still depends on shared state; the rule did not see it because it fails in the orders it did not look at. The remedy is not to change the rule but to interpret the measurement: a test suite that the rule calls code is not considered isolated until the seed range is widened. The two measurements above should be read together. The isolated runner gave [] on all six seeds; the tested suite gave [] only on seeds 1 and 2. In the first, [] was seen on all six seeds, in the second on only two. The same output is supported by a different number of seeds, and the value of the measurement comes from that number. PG18. Six seeds gave only four of the possible orders; as the number of tests grows, the number of possible orders grows quickly and a few seeds cannot exhaust them.

Summary

• A test framework has three parts: the test definition, the assertion and the runner. A failing test is an assertion error; an erroring test is another kind of error.
• The two writing styles, a test class with test_ methods and a definition block with an expectation chain, give the runner the same name → block table; the same two tests gave the same result in both styles.
• The determinant of this lesson is order: in the test result row the evidence is D- O+ E-, the base is [], and with seed 2 it is ["price_format"].
• The source of a flaky test is shared state: currency_conversion changes the shared settings and makes the test that runs after it fail. If the seed is printed, the failing test is reproduced.
• When each test takes its own copy of the settings, all six seeds give [] and the rule says code; dup is shallow.
• The rule tries only seeds 1 and 2; it classifies as code a suite that passes on seed 2 but fails on seed 5.

Next Step

The seed chooses the order and isolation removes it. But a test can depend on another input too: today’s clock. The next lesson measures why a test that reads the clock will fail tomorrow, and how faking and test data remove this dependency.