Testing

The previous lesson measured the advantage of the persistent stay of a fragment and the cost of its going stale. To trust a persistent value, one has to be sure that the code producing that value works correctly, and the way to check this is a test. A test also depends on the stay of a value: how far does the state a test sets up reach by the end of the test? If a record left by one test mixes into the starting state of the next, the tests are not independent of each other and their results change with their order.

In this lesson the measuring axis stays the same: how many requests does this value stay for? The four stays: this request, next request, session, persistent. The instance variable of a controller belongs to one request; the store of an application belongs to all requests. The isolation of a test depends on whether these two stays mix between tests.

Core

The measurements rest on the same small request loop. The file is the course core and is given as it is.

# stay.rb — a small request loop and a helper that classifies how many requests a value stays for

# Flash: a value written in this request is carried to the next request only.
class Flash
  def initialize(incoming)
    @incoming = incoming
    @fresh = {}
  end

  def [](key) = @fresh.fetch(key) { @incoming[key] }

  def []=(key, value)
    @fresh[key] = value
  end

  def passed_on = @fresh
end

# Every request gets a new context: params, flash and the request store start empty each time.
class Context
  attr_reader :app, :params, :session, :flash, :request_store

  def initialize(app, session)
    @app = app
    @params = {}
    @session = session
    @flash = Flash.new(session.delete(:flash) || {})
    @request_store = {}
  end
end

# The application store is the same across all requests and all clients;
# the session is chosen per client.
class App
  attr_reader :store

  def initialize
    @sessions = Hash.new { |h, client| h[client] = {} }
    @store = { products: {}, schema: { products: %i[name price] }, templates: {},
               queue: [], cache: {}, counter: 0 }
  end

  def request(client)
    context = Context.new(self, @sessions[client])
    result = yield context
    context.session[:flash] = context.flash.passed_on unless context.flash.passed_on.empty?
    result
  end
end

# Rule: request 1 (client A) writes the value; requests 2 and 3 (A) and request 4 (B) read it.
# If B sees it, persistent; if A's third request sees it, session; if only the second
# request sees it, next; if none sees it, this_request.
def stay(write, read)
  app = App.new
  first = app.request("A") { |b| write.call(b); read.call(b) }
  raise ArgumentError, "value could not be read in the request that wrote it" if first.nil?

  seen = [["A", 2], ["A", 3], ["B", 4]].map do |client, order|
    app.request(client) { |b| read.call(b) }.nil? ? "--" : "#{client}#{order}"
  end
  kind = if seen[2] != "--" then "persistent"
         elsif seen[1] != "--" then "session"
         elsif seen[0] != "--" then "next"
         else "this_request"
         end
  [kind, ["A1", *seen].join(" "), first]
end

This block is source; the file is written to disk and pulled in with require_relative. The core is not a framework but a model of one. It carries one difference: Product.lookup in the core returns the stored object itself, while in many mappers every read builds a new object from the row. The measurements in this lesson do not depend on the observable results of this difference.

Reading the Controller from Outside

When a controller runs an action it leaves what it did in two places: in the value it returns and in its instance variables. The second is the state carried to the view, and a unit test can read it directly. A unit test examines a single part, here a single controller, separated from the other parts. In Ruby an instance variable is read from outside with instance_variable_get. The excerpt gathers Product and ProductsController, familiar from earlier lessons, and the controller helper in a single file.

# testing.rb — Product, ProductsController and the controller helper
require_relative "stay"

module Product
  def self.insert(b, attributes)
    errors = []
    errors << "name cannot be blank" if attributes[:name].to_s.empty?
    errors << "price cannot be negative" if attributes[:price].to_i.negative?
    return { errors: } unless errors.empty?

    table = b.app.store[:products]
    id = table.size + 1
    table[id] = attributes.merge(id:)
    { id:, errors: }
  end

  def self.lookup(b, id)
    queries = (b.request_store[:queries] ||= {})
    queries.fetch([:products, id]) do
      b.request_store[:query_count] = b.request_store.fetch(:query_count, 0) + 1
      queries[[:products, id]] = b.app.store[:products][id]
    end
  end
end

class ProductsController
  def self.before(*names) = (@filters ||= []).concat(names)
  def self.filters = @filters || []

  before :load_product

  def initialize(context)
    @context = context
  end

  def run(action)
    self.class.filters.each { |filter| send(filter) }
    public_send(action)
  end

  def show = @product

  private

  def load_product
    @product = Product.lookup(@context, @context.params[:id])
  end
end

def controller(b) = b.request_store[:controller] ||= ProductsController.new(b)

def report(name, kind, evidence, value)
  printf("%-14s%-14s%-13s%s\n", name, kind, evidence, value.inspect)
end

This block is source as well. The first measurement reads the instance variable of the controller after the action and classifies its stay.

# variable.rb — how many requests does the controller variable stay for
require_relative "testing"

pen = ->(b) { Product.insert(b, name: "pen", price: 12) }
write = ->(b) { pen.call(b); b.params[:id] = 1; controller(b).run(:show) }
read = ->(b) { controller(b).instance_variable_get(:@product)&.fetch(:name) }
report("controller", *stay(write, read))

controller    this_request  A1 -- -- --  "pen"

The instance variable was seen only in the request in which the action ran. In the next request controller(b) builds a new controller and its @product is empty. This is the this request stay: the controller is built from the request store and goes away with the request. This is also a convenience for a test, because the state a test reads cannot have come from the previous test. The record is different: Product.insert writes to the store of the application, and that store outlives requests. Separating the two stays is the first condition of isolation. AS9. Here we built the controller directly and called it with run; no routing or middleware came into play, so the measurement is a unit test.

A Small Verification Helper

It is possible to write a test without needing a test tool: it is enough to compare the expected and the actual value and raise an exception on a mismatch. The lesson does not introduce a test tool; it only shows the smallest form of a verification and measures what happens when it fails.

# verification.rb — the smallest verification helper
require_relative "testing"

class AssertionError < StandardError; end

def verify(expected, actual)
  raise AssertionError, "expected #{expected.inspect}, got #{actual.inspect}" unless expected == actual

  :passed
end

This block is source as well. The helper is called twice in a request that creates a product; in the second call the expected value is given wrongly.

# failure.rb — a passing and a failing verification
require_relative "verification"

app = App.new
result = app.request("A") do |b|
  Product.insert(b, name: "pen", price: 12)
  b.params[:id] = 1
  controller(b).run(:show)
end
p verify("pen", result[:name])
begin
  verify("eraser", result[:name])
rescue AssertionError => e
  puts "#{e.class}: #{e.message}"
end

:passed
AssertionError: expected "eraser", got "pen"

The first verification passed. The second raised an exception because the expected value differed from the actual value, and the message of the exception wrote both values. This is basically what a test tool does: report the difference as an error and stop the rest of the test with that error. The rest, that is, finding the tests, running them in order and summarizing the results, is added on top of this core.

Unit Test and Feature Test

A unit test examines a single part. A feature test examines the path a user would follow from beginning to end: a request arrives, is processed, and a response returns. The difference between the two is scope, and as the scope grows, finding the cause of a failure becomes harder. The same behavior is measured at two scopes: a product is created and found. In the unit, Product and the controller are called directly. In the feature, there is a single call that carries the path passing through a request.

# scope.rb — a unit test and a feature test examine the same behavior at different scopes
require_relative "verification"

# unit: only Product
app = App.new
app.request("A") do |b|
  Product.insert(b, name: "pen", price: 12)
  p verify("pen", Product.lookup(b, 1)[:name])
end

# feature: the path through a request, controller included
app = App.new
app.request("A") { |b| Product.insert(b, name: "pen", price: 12) }
response = app.request("B") do |b|
  b.params[:id] = 1
  controller(b).run(:show)
end
p verify("pen", response[:name])

:passed
:passed

Both tests passed, but one only called Product; the other followed the path up to the request of another client. A feature test verifies more things because it examines the combination of one part with another, and when it breaks it does not show what broke. A unit test verifies fewer things and when it breaks its place is clear. The two do not replace each other.

A Failing Test Does Not Undo Its Writes

When a verification fails, the exception stops the rest of the test. But is the write the test has made up to that moment stopped, or undone? Measure it in a test that shares the application: the test first creates a product, then fails with a wrong expectation. After that the store of the application is looked at.

# residue.rb — does a failing test undo the product it wrote
require_relative "verification"

shared = App.new
begin
  shared.request("A") do |b|
    Product.insert(b, name: "pen", price: 12)
    verify("eraser", Product.lookup(b, 1)[:name])
  end
rescue AssertionError => e
  puts e.class
end
p shared.store[:products].size

AssertionError
1

The verification raised an exception, but the product stayed in the store. The exception cuts the flow of the test; it does not delete a write already made to the store of the application. This means that an error has two results: the test itself fails, and the residue it leaves enters the starting state of later tests because of the persistent stay of the store. This is one of the reasons the tests that come after a failed test may break as well. Leaving the cleanup to the test itself is not a solution, because the cleanup code can also stay behind the flow cut by the failing test; the more reliable way is to rebuild its own application for each test. In this core the cost of building is one App.new call; when the cost is this low, the gain of sharing does not exist in this core.

If the Order Changes the Result

The bounding measurement below will show the broken isolation with two identifiers; here we measure the same breakage through whether a test passes or not. Two tests share the same application: one expects the identifier of the first product to be 1, the other only creates a product. The tests are run in two different orders, and a new application is built in each run.

# order_test.rb — the same two tests, in two orders, in a shared application
require_relative "verification"

expecting = ->(b) { verify(1, Product.insert(b, name: "pen", price: 12)[:id]) }
writing = ->(b) { Product.insert(b, name: "eraser", price: 5) }

{ "expecting, writing" => [expecting, writing], "writing, expecting" => [writing, expecting] }.each do |name, ordering|
  app = App.new
  results = ordering.map do |test|
    app.request("A") { |b| test.call(b) }
    :passed
  rescue AssertionError
    :failed
  end
  puts "#{name}: #{results.inspect}"
end

expecting, writing: [:passed, :passed]
writing, expecting: [:passed, :failed]

The same two tests passed in one order and failed in the other. Each of the tests looks correct on its own; the problem is that the two share the application. A failure that depends on order is a sign that the test does not examine a behavior but the state the tests leave to each other.

Tests That Share the Cache

A measurement shows that isolation holds for the cache as well. Two tests share the same application. The first test writes a value into the cache of the application. The second test assumes the cache is empty and wants to fill the same key with its own value.

# cache_leak.rb — does the second test that shares the cache see its own value
require_relative "verification"

shared = App.new
shared.request("A") { |b| b.app.store[:cache]["k"] ||= "first test's value" }
read = shared.request("A") { |b| b.app.store[:cache]["k"] ||= "second test's value" }
p read
isolated = App.new.request("A") { |b| b.app.store[:cache]["k"] ||= "second test's value" }
p isolated

"first test's value"
"second test's value"

In the shared application the second test read the value of the first, not its own; in the isolated application it read its own value. Because of its persistent stay, the cache is a leakage channel between tests, and this leakage does not tell that the test failed: the second test may pass, but it has not examined what it expected to.

Relation to the Production Environment

M12/K05 measured whether the test environment goes out to production: the configuration of an environment is part of a process and does not leave a process. The same principle holds one step further here. A test builds its own application; the store and sessions of that application live only in the process of that test, inside the objects that test built. This gives a test safety: a test does not read or change the records of the production application.

The Bounding Measurement

The situation in which isolation breaks is the one in which two tests share the same application. The first measurement is the situation in which each test builds its own application: in two separate App.new instances, what is the identifier of the first product?

# isolation.rb — if each test builds its own application
require_relative "testing"

first = App.new.request("A") { |b| Product.insert(b, name: "pen", price: 12)[:id] }
second = App.new.request("A") { |b| Product.insert(b, name: "eraser", price: 5)[:id] }
puts "separate apps: #{first} and #{second}"

separate apps: 1 and 1

Both tests received the identifier 1 as their first record: one can run without the other, and the order does not affect the result. The second measurement sets up two tests that share the same application.

# sharing.rb — if two tests share the same application
require_relative "testing"

shared = App.new
first = shared.request("A") { |b| Product.insert(b, name: "pen", price: 12)[:id] }
second = shared.request("A") { |b| Product.insert(b, name: "eraser", price: 5)[:id] }
puts "shared app: #{first} and #{second}"

shared app: 1 and 2

The second test, which would have received 1 when run independently, received 2 after the first test. If a test expects 1, it passes only when it runs first in the shared application; when run in another order or alone, its result changes. This can be called leakage between tests: because the store of the application is in the persistent stay, the value a test wrote changes the state the next test reads. The solution is isolation: each test builds its own application, the store of that application and its clients, from scratch. This holds for the fragment cache of the previous lesson as well. If tests share a cache, the fragment written by one test becomes the fragment the next test sees, and in the same way as stale content it changes the result of the test.

Summary

• The instance variable of a controller is in the this request stay; in the next request a new controller is built and the variable is empty.
• A unit test examines a single part by reading the instance variable from outside; the smallest verification helper raises an exception if the expected and actual values differ.
• A feature test examines a path from beginning to end; a unit test verifies less but shows the place when it breaks.
• Each test building its own application provides isolation: the first identifier is 1 and 1 in separate applications, and 1 and 2 in a shared application.
• Because the store of the application is in the persistent stay, a shared application leads to leakage between tests: a failing test leaves residue, and the result depends on the order.
• A shared cache is a leakage channel as well; the second test may pass even though it has not examined what it expected to.

Next Step

The last thing to be verified in an application is its common security defaults. The next lesson asks why the anti-forgery token is kept in the session and measures the stay of this token.