--- title: 'Testing Frameworks' source: 'https://academia.sh/en/courses/ruby-production/testing-frameworks' course: 'The Ruby Ecosystem and Production' language: en updated: '2026-10-21T05:03:44+00:00' license: 'CC BY-SA 4.0' objectives: - 'Separates the three parts of a test framework (definition, assertion, runner) and shows that the test-class and definition-block styles give the runner the same table' - 'Shows that the runner shuffles tests with a seed and that a failing test is reproduced with the seed' - 'Measures that the source of a flaky test is shared state' - 'Separates that isolation moves the class from order to code and that the rule tries only two seeds' --- # Testing Frameworks Why do the same tests sometimes fail? The lesson builds the three parts of a test framework in pure Ruby and measures that the seed chooses the order and that shared state makes a test fail. The dependency is locked and the installed version is now fixed. Even so, the same code, the same lock file and the same tests can pass in one run and fail in another. The observation of this lesson is the `test result` row: the base is `[]`, so no test fails, but the evidence column shows `D- O+ E-`. The result changes when the order changes. M10/K05 asked about the limit of an assurance: what a test catches and what it misses. Here the question is different. What determines whether a test passes? There the limit was the coverage of the assurance; here the limit is an input outside the test itself. The core's `determinant.rb` file is given again exactly; the block is source and is pulled in with `require_relative`. ```ruby # determinant.rb — a run model and a helper that classifies what determines a result # Run: the three inputs a result may depend on. Everything outside these three is code. Run = Data.define(:lock, :seed, :environment) BASE = Run.new( lock: { "formatter" => "1.4.2", "parser" => "2.0.1" }, seed: 1, environment: { clock: Time.utc(2026, 10, 16, 9, 0), cores: 4, variables: {} } ) # Each variant changes a single input; the others stay as in the base. VARIANTS = { dependency: BASE.with(lock: { "formatter" => "1.5.0", "parser" => "2.1.0" }), order: BASE.with(seed: 2), environment: BASE.with(environment: { clock: Time.utc(2027, 3, 1, 9, 0), cores: 8, variables: { "WORKER_COUNT" => "3" } }) } # Rule: the computation runs first in the base, then once in each variant. The first variant # that changes the result (dependency, order, environment, in that order) is the determinant; # if none changes it, code. def determinant(computation) base = computation.call(BASE) changed = VARIANTS.to_h { |name, run| [name, computation.call(run) != base] } kind = changed.key(true)&.to_s || "code" evidence = changed.map { |name, c| "#{name[0].upcase}#{c ? '+' : '-'}" }.join(" ") [kind, evidence, base] end ``` ## The Three Parts of a Framework A **test framework** consists of three parts. A **test definition** is a named block that is a runnable trial. An **assertion** compares the expected value with the actual one and raises an error if they do not match. A **test runner** runs the definitions, catches the assertion error and writes the result to a table. Common test frameworks are built on these three parts too. Here all three are built in pure Ruby, because the observation lies inside the runner and the runner must be visible. ```ruby # framework.rb — the three parts of a test framework: definition, assertion, runner class AssertionFailed < StandardError; end # Assertion: silent if the expected and the actual value are the same, otherwise raises an error. def assertion(expected, actual) raise AssertionFailed, "assertion failed" unless expected == actual end # Runner: runs each test, catches the assertion error, writes the result to the table. def run_suite(tests) tests.to_h do |name, test| test.call [name, :passed] rescue AssertionFailed [name, :failed] end end ``` The file is source. The file below runs three tests. The second one expects integer division wrongly. ```ruby # small.rb — the three parts come together require_relative "framework" tests = { "addition" => -> { assertion(5, 2 + 3) }, "division" => -> { assertion(3.5, 7 / 2) }, "decimal" => -> { assertion(3.5, 7 / 2.0) } } p run_suite(tests) ``` ``` {"addition" => :passed, "division" => :failed, "decimal" => :passed} ``` `7 / 2` is integer division and gives `3`; the second assertion did not hold, the runner caught the error and wrote the `division` test as `:failed`. The other two passed. Note that the runner catches only `AssertionFailed`. Any other error, for example division by zero, is not an assertion result and does not enter the table. ```ruby # error.rb — an error that is not an assertion error does not enter the table require_relative "framework" begin run_suite("division by zero" => -> { 1 / 0 }) rescue => e p e.class end ``` ``` ZeroDivisionError ``` The runner did not catch `ZeroDivisionError`; the error escaped. This distinction matters: a test **failing** and a test **erroring out** are different events, and this lesson measures only the first. **PG13.** The framework built here shows the three parts; the expectation library, the setup and the cleanup steps of a real framework lie outside this model. ## Two Writing Styles Common frameworks offer the same three parts in two different writing styles. In the first, tests are methods of a class whose names start with `test_`, and an assertion is a method call. In the second, tests are named examples inside a definition block, and an assertion is an expectation chain that starts from the actual value. The file below writes the same two tests in both styles and hands both to the same runner. ```ruby # styles.rb — the same two tests, two writing styles, one runner require_relative "framework" # First style: a test class; every method starting with test_ is a test. class DivisionTests def test_integer assertion(3, 7 / 2) end def test_decimal assertion(3.5, 7 / 2) end end class_tests = DivisionTests.public_instance_methods(false).grep(/\Atest_/).sort.to_h do |name| [name.to_s, -> { DivisionTests.new.public_send(name) }] end # Second style: a definition block and an expectation chain. Expectation = Struct.new(:actual) do def to_be(expected) assertion(expected, actual) end end def expect(actual) Expectation.new(actual) end class Definition attr_reader :tests def initialize(subject) @subject = subject @tests = {} end def example(name, &block) @tests["#{@subject} #{name}"] = block end end def define(subject, &body) definition = Definition.new(subject) definition.instance_eval(&body) definition.tests end block_tests = define("division") do example("integer") { expect(7 / 2).to_be(3) } example("decimal") { expect(7 / 2).to_be(3.5) } end p run_suite(class_tests) p run_suite(block_tests) ``` ``` {"test_decimal" => :failed, "test_integer" => :passed} {"division integer" => :passed, "division decimal" => :failed} ``` The two tables give the same result: the test that expects `3.5` from integer division failed in both styles. The difference lies in the writing. In the first style the test name is the method name, and the tests are gathered in a class. In the second style the test name is a string built from the subject and the example name. In both cases the runner sees the same name → block table; this is why the order and the seed effect measured in the following sections belong to the runner. **PG13a.** The two styles were compared here in a single example with two tests; the setup, nested grouping and reporting differences of real frameworks lie outside this comparison. ## The Observation's Runner The runner in the observation row is smaller but does one more thing: it shuffles the tests with the seed and gives all of them the same settings table. The file below carries the `TESTS` and `run_tests` definitions taken from `observations.rb` exactly. ```ruby # tests.rb — the tests and the runner of the observation row # Test runner: tests run in an order shuffled by the seed and share the same settings. TESTS = { "price_format" => ->(settings) { "12 #{settings[:currency]}" == "12 TRY" }, "currency_conversion" => ->(settings) { settings[:currency] = "USD"; true }, "invoice_total" => ->(settings) { settings[:currency] == "TRY" } } def run_tests(run, tests = TESTS) settings = { currency: "TRY" } order = tests.keys.shuffle(random: Random.new(run.seed)) order.reject { |name| tests[name].call(settings) } end ``` There are three tests. `price_format` looks at the currency in the settings and expects `TRY`. `currency_conversion` sets the currency in the settings to `USD` and returns true. `invoice_total` expects `TRY` again. The runner returns the names of the failing tests. The file below prints the shuffled order and the failing tests side by side for six seeds. ```ruby # seeds.rb — six seeds, six orders require_relative "determinant" require_relative "tests" (1..6).each do |seed| order = TESTS.keys.shuffle(random: Random.new(seed)) puts "#{seed} #{order.inspect} #{run_tests(BASE.with(seed: seed)).inspect}" end ``` ``` 1 ["price_format", "invoice_total", "currency_conversion"] [] 2 ["invoice_total", "currency_conversion", "price_format"] ["price_format"] 3 ["currency_conversion", "price_format", "invoice_total"] ["price_format", "invoice_total"] 4 ["currency_conversion", "price_format", "invoice_total"] ["price_format", "invoice_total"] 5 ["price_format", "currency_conversion", "invoice_total"] ["invoice_total"] 6 ["price_format", "currency_conversion", "invoice_total"] ["invoice_total"] ``` The order of seed 1 puts `currency_conversion` last; since no test runs after it, no test fails. In seed 2 `currency_conversion` is in the middle, and `price_format`, which comes after it, fails. Seeds 3 and 4 gave the same order: `currency_conversion` first, and the two tests after it also failed. Seeds 5 and 6 share an order too; `currency_conversion` is in the middle and only `invoice_total`, which comes after it, fails. Four different orders came out of six seeds, and the result of each order is the same as long as the seed stays fixed. ## Shared State The flaw is not in the order but in what makes the order visible as a flaw. `currency_conversion` changes the shared settings table. This change is the input of every test that runs after it. That the table is shared among the tests can be shown directly. ```ruby # shared.rb — the setting a test wrote is the input of the test that comes after it require_relative "tests" settings = { currency: "TRY" } TESTS["currency_conversion"].call(settings) p settings p TESTS["price_format"].call(settings) p TESTS["price_format"].call({ currency: "TRY" }) ``` ``` {currency: "USD"} false true ``` After `currency_conversion` the settings became `USD`. The same `price_format` test returned `false` with these settings and `true` with fresh settings. The test code is the same; what changed is the starting state of the test. This is **shared state**, and it is the most common source of a **flaky test**: a test is fine, but it depends on who ran before it. **PG14.** The measurement holds only for a model that shares a settings table; shared state through the file system, the network or a global variable behaves the same way, but it was not measured here. ## Which Test Fails on How Many Seeds The result of a single seed is a diagnosis; the count over six seeds gives a profile. The file below counts the number of tests each seed fails, the number of seeds on which each test fails and the number of seeds on which at least one test fails. ```ruby # counts.rb — the distribution of failures by seed require_relative "determinant" require_relative "tests" results = (1..6).map { |s| run_tests(BASE.with(seed: s)) } p results.map(&:size) p results.flatten.tally p results.count { |f| !f.empty? } ``` ``` [0, 1, 2, 2, 1, 1] {"price_format" => 3, "invoice_total" => 4} 5 ``` In five of the six runs at least one test failed; only seed 1 passed cleanly. `currency_conversion` never failed, but it is the test that makes the other two fail. `invoice_total` failed on more seeds because `currency_conversion` came before it more often; `price_format` failed less. This table does not point at the culprit, it points at the victim. The failing test is not the faulty test itself; the faulty test passes silently and makes someone else fail. **How often a test fails measures not its own soundness but who runs before it.** For this reason, when reading the code of a failing test, the first question should be what the failing test shares. **PG15.** The count is for the six seeds of a single suite; it does not follow from this measurement that the failure frequency would have the same proportions in a real team's suite. Fixing the order does not solve this problem, it hides it. If seed 1 were used in every run, this suite would never fail and the shared state would stay unnoticed. Shuffling does not produce a defect; it makes an existing dependency visible. And to the extent that it can make it visible, printing the seed of the failing test turns shuffling from a suspicion into a diagnosis. ## Reproducing with the Seed A flaky test is a bug report if it can be reproduced at the moment it fails; if it cannot be reproduced, it is a rumor. When the runner fixes the seed, the failing test fails in the same order every time. For this reason a good runner **prints** the seed: when a run fails, the seed is the key to reproduction. ```ruby # replay.rb — reproduce a failing test with the seed require_relative "determinant" require_relative "tests" results = (1..6).map { |s| run_tests(BASE.with(seed: s)) } p results.count([]) p run_tests(VARIANTS[:order]) p run_tests(VARIANTS[:order]) == run_tests(BASE.with(seed: 2)) p determinant(->(r) { run_tests(r) }) ``` ``` 1 ["price_format"] true ["order", "D- O+ E-", []] ``` On only one of the six seeds did no test fail: seed 1 passed while seed 2 failed. Since `VARIANTS[:order]` is seed 2, it gave `["price_format"]`, and the second call with the same seed gave the same result as the first. The last line shows the classification: the result comes from `order`, the evidence is `D- O+ E-`, the base is `[]`. The rule wrote the base result as `[]`; the failing test is not visible in the base run, only in the `order` variant. The determinant of this lesson is **order**. **PG16.** Seed 1 showing "passed" does not show that the tests are sound; it shows only that in that order the test that corrupts the settings table fell to the end. ## The Bounding Measurement The class is tested from two directions. The first: what happens if each test runs with its own copy of the settings? The second: the rule tries only two seeds, so do these two seeds catch every flakiness? ```ruby # isolation.rb — each test gets its own copy of the settings require_relative "determinant" require_relative "tests" def isolated_run_tests(run, tests = TESTS) settings = { currency: "TRY" } order = tests.keys.shuffle(random: Random.new(run.seed)) order.reject { |name| tests[name].call(settings.dup) } end p (1..6).map { |s| isolated_run_tests(BASE.with(seed: s)) } p determinant(->(r) { isolated_run_tests(r) }) ``` ``` [[], [], [], [], [], []] ["code", "D- O- E-", []] ``` All six seeds gave `[]` and this time the rule said `code`, with the evidence `D- O- E-`. The same three tests, the same six seeds; the only difference is that each test takes its own copy. The class moved from **order** to **code**: **isolation removes the effect of the order**, because the tests are no longer each other's input. The limit of isolation is the depth of the copy. ```ruby # shallow.rb — dup is shallow settings = { list: [] } copy = settings.dup copy[:list] << 1 p settings p settings[:list].equal?(copy[:list]) ``` ``` {list: [1]} true ``` `dup` copied the table but not the array inside it: the two tables hold the same array, and the change in the copy passed to the original. Because the settings table carries only simple values, the isolation of this lesson worked. For nested settings `dup` is not enough. **PG17.** The isolation measurement holds only for a flat table. The second direction is the rule's own blind spot. The rule tries the base and the `order` variant, that is, seed 1 and seed 2. A suite can be built that passes on seed 2 but fails on seed 5. It is enough to replace the `price_format` test with a test that never looks at the settings. ```ruby # blind.rb — where the rule tries only seeds 1 and 2 require_relative "determinant" require_relative "tests" BLIND_TESTS = TESTS.merge("price_format" => ->(_settings) { true }) p determinant(->(r) { run_tests(r, BLIND_TESTS) }) results = (1..6).map { |s| run_tests(BASE.with(seed: s), BLIND_TESTS) } p results ``` ``` ["code", "D- O- E-", []] [[], [], ["invoice_total"], ["invoice_total"], ["invoice_total"], ["invoice_total"]] ``` The rule classified this suite as `code`, because seeds 1 and 2 both gave `[]`. But on four of the six seeds `invoice_total` failed. **The result "not order" was stated only for the two tried seeds.** The suite still depends on shared state; the rule did not see it because it fails in the orders it did not look at. The remedy is not to change the rule but to interpret the measurement: a test suite that the rule calls `code` is not considered isolated until the seed range is widened. The two measurements above should be read together. The isolated runner gave `[]` on all six seeds; the tested suite gave `[]` only on seeds 1 and 2. In the first, `[]` was seen on all six seeds, in the second on only two. The same output is supported by a different number of seeds, and the value of the measurement comes from that number. **PG18.** Six seeds gave only four of the possible orders; as the number of tests grows, the number of possible orders grows quickly and a few seeds cannot exhaust them. ## Summary - A test framework has three parts: the test definition, the assertion and the runner. A failing test is an assertion error; an erroring test is another kind of error. - The two writing styles, a test class with `test_` methods and a definition block with an expectation chain, give the runner the same name → block table; the same two tests gave the same result in both styles. - The determinant of this lesson is **order**: in the `test result` row the evidence is `D- O+ E-`, the base is `[]`, and with seed 2 it is `["price_format"]`. - The source of a flaky test is shared state: `currency_conversion` changes the shared settings and makes the test that runs after it fail. If the seed is printed, the failing test is reproduced. - When each test takes its own copy of the settings, all six seeds give `[]` and the rule says `code`; `dup` is shallow. - The rule tries only seeds 1 and 2; it classifies as `code` a suite that passes on seed 2 but fails on seed 5. ## Next Step The seed chooses the order and isolation removes it. But a test can depend on another input too: today's clock. The next lesson measures why a test that reads the clock will fail tomorrow, and how faking and test data remove this dependency.