Migrations

The previous lesson showed that a saved product is read by another client too. In that measurement the product had two attributes: name and price. What happens when a stock count is to be added to a product as well? The shape of the table, that is, the schema, does not change as a hand-edited file but through recorded steps. These steps are called migrations: a migration is a named and ordered step that carries the schema from one state to another. Is a column added to the schema present in the old rows?

The axis has not changed: how many requests does this value stay for? A column lives in the schema, and the schema, like the records, is in the store of the app. For this reason the expected stay is persistent. The lesson measures this and then shows two limits of a migration: what happens if the same migration runs twice, and whether rows written before the migration see the new column. Where a change reaches was measured separately in M12/K02; the question there was “which target does the change reach”, and here the question is how long the change that did reach stays.

The Measurement Core

The first file is the request loop and the stay classifier, identical in every lesson of the course. This block is the source; the file is written to disk and pulled in with require_relative.

# stay.rb — a small request loop and a helper that classifies how many requests a value stays for

# Flash: a value written in this request is carried to the next request only.
class Flash
  def initialize(incoming)
    @incoming = incoming
    @fresh = {}
  end

  def [](key) = @fresh.fetch(key) { @incoming[key] }

  def []=(key, value)
    @fresh[key] = value
  end

  def passed_on = @fresh
end

# Every request gets a new context: params, flash and the request store start empty each time.
class Context
  attr_reader :app, :params, :session, :flash, :request_store

  def initialize(app, session)
    @app = app
    @params = {}
    @session = session
    @flash = Flash.new(session.delete(:flash) || {})
    @request_store = {}
  end
end

# The application store is the same across all requests and all clients;
# the session is chosen per client.
class App
  attr_reader :store

  def initialize
    @sessions = Hash.new { |h, client| h[client] = {} }
    @store = { products: {}, schema: { products: %i[name price] }, templates: {},
               queue: [], cache: {}, counter: 0 }
  end

  def request(client)
    context = Context.new(self, @sessions[client])
    result = yield context
    context.session[:flash] = context.flash.passed_on unless context.flash.passed_on.empty?
    result
  end
end

# Rule: request 1 (client A) writes the value; requests 2 and 3 (A) and request 4 (B) read it.
# If B sees it, persistent; if A's third request sees it, session; if only the second
# request sees it, next; if none sees it, this_request.
def stay(write, read)
  app = App.new
  first = app.request("A") { |b| write.call(b); read.call(b) }
  raise ArgumentError, "value could not be read in the request that wrote it" if first.nil?

  seen = [["A", 2], ["A", 3], ["B", 4]].map do |client, order|
    app.request(client) { |b| read.call(b) }.nil? ? "--" : "#{client}#{order}"
  end
  kind = if seen[2] != "--" then "persistent"
         elsif seen[1] != "--" then "session"
         elsif seen[0] != "--" then "next"
         else "this_request"
         end
  [kind, ["A1", *seen].join(" "), first]
end

The excerpt of this lesson is a single method. A migration, in its plainest form, adds a column name to a table in the schema. The schema is in the hash store[:schema]: the key is the table name and the value is the list of column names.

# migrate.rb — schema change: a column is added to a table
require_relative "stay"

def migrate(b, table, column) = b.app.store[:schema][table] << column

This block is source as well. The core is not a framework but a model of a framework; this method stands for a single forward migration. Most migration mechanisms write each migration as a class or a file and can define a backward direction next to the forward one. Below, helpers of this lesson build the same distinction on a small scale.

The Stay of a Column

The writing lambda adds the column stock to the table products, and the reading lambda looks for this column in the schema.

# measure.rb — stay of the schema column observation
require_relative "migrate"

write = ->(b) { migrate(b, :products, :stock) }
read = ->(b) { b.app.store[:schema][:products].find { |c| c == :stock } }
p stay(write, read)

["persistent", "A1 A2 A3 B4", :stock]

The stay is persistent: the fourth request of B found the column :stock in the schema too. The evidence column contains all four requests and none has --. The schema is in the store of the app, and the app serves all clients; a migration run by one client changes the schema that the others see as well. DL6. This measurement is within a single process: whether the migration remains after the process restarts cannot be measured in the in-memory store of the core.

Who runs the migration does not change the stay. If the same column is added in a request of client B, the next request of A sees it.

# other_client.rb — the migration runs for one client, the other client reads
require_relative "migrate"

app = App.new
app.request("B") { |b| migrate(b, :products, :stock) }
app.request("A") { |b| p b.app.store[:schema][:products] }

[:name, :price, :stock]

The schema belongs to the app, not to a client: a schema changed in the request of one client is visible in every later request in its changed form, whichever client the request comes from. This rests on the same cause as the stay of the records in the previous lesson; both live in store. DL9. Here the migration ran inside a request because the core has no other place; where a migration runs in a real application is not the subject of this measurement.

Forward and Backward

Migrations have two directions. The forward migration changes the schema, and the rollback restores the change to its former state. The core has only the forward direction; the rollback is a helper of this lesson. First the state of the schema before and after the migration is examined, then its state after the rollback.

# forward_back.rb — forward and backward migration
require_relative "migrate"

def revert(b, table, column) = b.app.store[:schema][table].delete(column)

app = App.new
app.request("A") do |b|
  schema = b.app.store[:schema]
  p schema[:products]
  migrate(b, :products, :stock)
  p schema[:products]
  revert(b, :products, :stock)
  p schema[:products]
end

[:name, :price]
[:name, :price, :stock]
[:name, :price]

The three output lines are three states: before the migration, after the forward migration, and after the rollback. The rollback returned the schema to the first state. That a migration has two directions says that the schema is a sequence of versions: each version is built on the previous one and forms a path that moves on in one direction and can go back in the other. This lesson measures only the schema side; the values inside the rows are asked separately in the bounding measurement below.

How many requests does the effect of the rollback stay for? The measurement below adds the column, writes a value into a row, rolls the column back, and reads the schema and the row separately. Then it passes the same rollback through the stay rule.

# rollback_stay.rb — the rolled-back column and the value in the row
require_relative "migrate"

def revert(b, table, column) = b.app.store[:schema][table].delete(column)

app = App.new
app.request("A") do |b|
  migrate(b, :products, :stock)
  b.app.store[:products][1] = { name: "pen", price: 12, id: 1, stock: 5 }
  revert(b, :products, :stock)
  p b.app.store[:schema][:products].include?(:stock)
  p b.app.store[:products][1].key?(:stock)
end

write = lambda do |b|
  migrate(b, :products, :stock)
  revert(b, :products, :stock)
end
read = ->(b) { b.app.store[:schema][:products].include?(:stock) }
p stay(write, read)

false
true
["persistent", "A1 A2 A3 B4", false]

The schema no longer contains the column :stock, but the row keeps carrying the key and the value 5. In this model the rollback touches only the schema; it does not delete the data in the rows. The stay line shows that the rolled-back schema is persistent as well: even the fourth request of B reads false, so the absence of the column was not limited to the request of A. In a real migration mechanism, whether a rollback also deletes the value in the rows depends on the mechanism and has not been measured here. In this model the step that restored the schema did not restore the data.

A New Table and a Missing Table

migrate only adds a column to an existing table. What happens if the same call is made for a table that is not in the schema? The measurement prints only the class of the exception that is caught.

# missing_table.rb — adding a column to a table that does not exist
require_relative "migrate"

def create_table(b, table, *columns)
  b.app.store[:schema][table] = columns
end

app = App.new
app.request("A") do |b|
  begin
    migrate(b, :orders, :quantity)
  rescue => e
    p e.class
  end
  create_table(b, :orders, :product_id, :quantity)
  p b.app.store[:schema].keys
  p b.app.store[:schema][:orders]
end

NoMethodError
[:products, :orders]
[:product_id, :quantity]

Adding a column to a table that does not exist gave NoMethodError: that name was not in the schema, and the append method was called on nil. Creating a table is a different kind of migration. create_table wrote the table into the schema together with its column list, and the schema now contains two tables. In the next lesson, Order records will use this table. DL7. Here orders is only defined in the schema; no row is written to the table.

Why Migrations Are Numbered

What happens to the schema if the same forward migration runs twice? migrate appends to the end of a list and does not remember what it appended.

# twice.rb — the same migration twice
require_relative "migrate"

app = App.new
app.request("A") do |b|
  migrate(b, :products, :stock)
  migrate(b, :products, :stock)
  p b.app.store[:schema][:products]
end

[:name, :price, :stock, :stock]

:stock is in the schema twice. The model itself has no version trace: no record remembers which migration has run, and it cannot be told whether the second call repeats the first or is a new migration. A real migration mechanism gives every migration a number for exactly this reason and keeps in a version table which numbers have been applied. The helper of this lesson builds the same idea on a small scale: a migration whose number has been applied does not run a second time.

# numbered.rb — helper that remembers applied migrations
require_relative "migrate"

def numbered_migration(b, number, table, column)
  applied = (b.app.store[:applied] ||= [])
  return false if applied.include?(number)

  migrate(b, table, column)
  applied << number
  true
end

app = App.new
app.request("A") do |b|
  p numbered_migration(b, 1, :products, :stock)
  p numbered_migration(b, 1, :products, :stock)
  p b.app.store[:schema][:products]
  p b.app.store[:applied]
end

write = ->(b) { numbered_migration(b, 1, :products, :stock) }
read = ->(b) { b.app.store[:applied]&.first }
p stay(write, read)

true
false
[:name, :price, :stock]
[1]
["persistent", "A1 A2 A3 B4", 1]

The first call gave true and ran the migration; the second call with the same number gave false and did not touch the schema. :stock appears in the schema once. The list of applied migrations is [1]: the migration number lives in the store of the app like the schema, and its stay is persistent, because even the fourth request of B sees the first item of the list. The version record remembers that the migration ran, not what its result was. DL8. Here the number is an integer and migrations are called one by one, in order; what happens if the order is broken has not been measured.

What the record verifies can be asked separately. Adding the same column under two different numbers shows whether the record follows the number or the content of the migration.

# two_numbers.rb — the same column, two numbers
require_relative "migrate"

def numbered_migration(b, number, table, column)
  applied = (b.app.store[:applied] ||= [])
  return false if applied.include?(number)

  migrate(b, table, column)
  applied << number
  true
end

app = App.new
app.request("A") do |b|
  p numbered_migration(b, 1, :products, :stock)
  p numbered_migration(b, 2, :products, :stock)
  p b.app.store[:schema][:products]
  p b.app.store[:applied]
end

true
true
[:name, :price, :stock, :stock]
[1, 2]

The second call gave true and :stock entered the schema a second time. The record follows the number and does not follow what the migration does: a migration that brings the same change under another number is not blocked. The protection that numbering gives rests on the numbers being given correctly. Whether real mechanisms keep a checksum of the content of the migration file has not been measured here.

The Bounding Measurement

A column added to the schema is persistent. But does a row written before the migration have this column? The stay of the schema and the stay of the data are separate questions. The measurement below writes a row directly to the table, runs the migration, and looks at both the schema and the row.

# boundary.rb — a row written before the migration
require_relative "migrate"

app = App.new
app.request("A") do |b|
  b.app.store[:products][1] = { name: "pen", price: 12, id: 1 }
  migrate(b, :products, :stock)
  p b.app.store[:schema][:products].include?(:stock)
  p b.app.store[:products][1].key?(:stock)
  b.app.store[:products][2] = { name: "notebook", price: 30, id: 2, stock: 5 }
  p b.app.store[:products].values.map { |row| row.key?(:stock) }
end

true
false
[false, true]

The schema contains the column :stock, but the row written before the migration does not carry the key :stock: key? gave false. The row written after the migration has the key. The schema and the row are not one thing: the schema says what rows should look like and does not change the existing rows. In this model old rows never carry the new key; in a relational database the new column appears in old rows as empty (NULL) or with a default value. In both cases, old rows must be filled separately to get a meaningful value. For this reason two jobs are kept apart: the schema migration that adds the column and the data migration that writes a value into old rows. The helper below writes a default into every row that lacks the key.

# backfill.rb — data migration that writes the new column into old rows
require_relative "migrate"

def backfill_column(b, table, column, value)
  b.app.store[table].each_value { |row| row[column] = value unless row.key?(column) }
end

app = App.new
app.request("A") do |b|
  b.app.store[:products][1] = { name: "pen", price: 12, id: 1 }
  migrate(b, :products, :stock)
  p b.app.store[:products].values.map { |row| row.key?(:stock) }
  backfill_column(b, :products, :stock, 0)
  p b.app.store[:products][1]
end

write = lambda do |b|
  b.app.store[:products][1] = { name: "pen", price: 12, id: 1 }
  migrate(b, :products, :stock)
end
read = ->(b) { b.app.store[:products][1]&.key?(:stock) }
p stay(write, read)

[false]
{name: "pen", price: 12, id: 1, stock: 0}
["persistent", "A1 A2 A3 B4", false]

First the old row gave false; after the backfill the row has stock: 0. The last line puts two stays side by side: this reading lambda looks for the column key in the row and finds false, but because false is a value, the stay rule counts it as seen and reads the same value in every request. As long as no key is added to the row after the migration, the fourth request of B reads false too. The same migration left a persistent column in the schema and a persistent absence in the row. These two must be kept apart: if the reader asks the schema, :stock exists; if it asks the row, it does not. The stay rule does not show this difference, because the rule reads the behavior of a value; whether an absence is an error or a default is decided by the code that reads the row.

Summary

• A migration is a named step that carries the schema from one state to another; its forward direction adds a column and its rollback direction removes it.
• The schema column, the observation of this lesson, is a value whose stay is persistent: the column :stock was read in the schema in the next two requests of A and in the fourth request of B.
• The core model has no version trace; when the same migration ran twice, :stock was written twice, which is why migrations are numbered and the list of applied ones is kept.
• A migration changes the schema and does not change existing rows: a row written before the migration has no new key, and filling is a separate step.
• In this model the rollback restores the schema and does not touch the data in the rows. The record follows the number, not the content of the migration; the same content under a different number is applied again.
• The stay of the column and the stay of the data are separate things; both are persistent, but one preserves the existence of the column and the other the absence of the value in the row.

Next Step

The schema and the rows are now defined. But a record must obey rules before it enters the table, and a record can be linked to another record. Where does the error of a rejected record stay? The next lesson reads relationships and validations with the same measure.