Local Database

In the previous lesson the repository ordered the sources, and the disk tier was a ready-made map. On the platform, a local database stands behind this tier, and its schema grows together with the application. In this lesson we look at how the disk tier gets filled and what happens to old rows when the schema changes. The question of the lesson is the same as in the earlier lessons: where does the answer to this read come from?

// Origin.kt — classifies which tier a read's answer came from
class Tier(val origin: String, private val entries: Map<String, String>) {
    init {
        require(origin in listOf("memory", "disk", "network")) { "invalid tier: $origin" }
    }

    fun lookup(key: String): String? = entries[key]
}

class Read(val label: String, val key: String, chain: List<Tier>, fallback: String) {
    private val hit: Tier? = chain.firstOrNull { it.lookup(key) != null }
    val value: String = hit?.lookup(key) ?: fallback
    val origin: String = hit?.origin ?: "default"

    fun show() {
        println("$label: key=$key | value=$value -> $origin")
    }
}

fun countOrigins(reads: List<Read>) {
    for (name in listOf("memory", "disk", "network", "default")) {
        println("$name: ${reads.count { it.origin == name }}")
    }
}

The Entity, the Data Access Object, and the Database

The local database library consists of three parts. The entity is the class that describes a row of a table; each of its fields is a column. The data access object (DAO) is the interface of the queries and writes made against that table. The database class brings the entities and the schema version together and exposes the data access objects. From here on we will say only DAO.

// Platform code: AppDatabase
@Entity(tableName = "cart_item")
data class CartItemEntity(
    @PrimaryKey val productId: String,
    val quantity: Int,
    val stock: Int = 0,
)

@Dao
interface CartItemDao {
    @Query("SELECT * FROM cart_item WHERE productId = :id")
    suspend fun find(id: String): CartItemEntity?

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun save(item: CartItemEntity)
}

@Database(entities = [CartItemEntity::class], version = 2)
abstract class AppDatabase : RoomDatabase() {
    abstract fun cartItemDao(): CartItemDao
}

The version number is the identity of the schema. When the new version of the application adds a column, the number goes from 1 to 2, and the rows written by the old version stay on disk. This snippet is platform code, and here we make no claim about output, duration, or file path. The model builds the rule of the platform; the model is not the platform itself.

In the model, the database is a disk tier, and the rows are key and value pairs. A schema change affects the keys of these pairs.

A Migration Carries Rows

In version 1, the quantity of the cart was in the cart.quantity column. In version 2, this column became cart.count, and a new column, product.stock, was added. A migration is the rule that carries the rows of the old version to the schema of the new version. On the platform, a class writes it, and schema commands sit inside it.

// Platform code: Migration1To2
val migration1To2 = object : Migration(1, 2) {
    override fun migrate(db: SupportSQLiteDatabase) {
        db.execSQL("ALTER TABLE cart_item RENAME COLUMN quantity TO count")
        db.execSQL("ALTER TABLE cart_item ADD COLUMN stock INTEGER NOT NULL DEFAULT 0")
    }
}

ALTER TABLE cart_item RENAME COLUMN quantity TO count;
ALTER TABLE cart_item ADD COLUMN stock INTEGER NOT NULL DEFAULT 0;

On the platform, the DEFAULT 0 expression writes zero into old rows. In the model, we will separate this behavior into two distinct choices. The migration function renames the keys and, if asked, writes the value of the new column into the old rows.

// Migrate.kt — carries old rows to the new schema, fills the new column if asked
fun migrate(
    rows: Map<String, String>,
    renames: Map<String, String>,
    backfill: Map<String, String>,
): Map<String, String> =
    rows.map { (key, value) -> (renames[key] ?: key) to value }.toMap() + backfill

The screen reads four values according to the schema of version 2.

// Screen.kt — four values read according to schema version 2
fun readCartScreen(disk: Tier): List<Read> {
    val chain = listOf(disk)
    return listOf(
        Read("profile-name", "profile.name", chain, "unknown"),
        Read("product-price", "product.price", chain, "unknown"),
        Read("cart-count", "cart.count", chain, "0"),
        Read("product-stock", "product.stock", chain, "0"),
    )
}

We first measure the state before the migration. Disk holds three rows written with version 1, and nobody has transformed them yet.

// Main.kt — without a migration, the query of version 2 looks at the rows of version 1
fun main() {
    val v1Rows = mapOf("profile.name" to "Ayse", "product.price" to "120", "cart.quantity" to "2")
    val reads = readCartScreen(Tier("disk", v1Rows))
    reads.forEach { it.show() }
    countOrigins(reads)
}

profile-name: key=profile.name | value=Ayse -> disk
product-price: key=product.price | value=120 -> disk
cart-count: key=cart.count | value=0 -> default
product-stock: key=product.stock | value=0 -> default
memory: 0
disk: 2
network: 0
default: 2

profile.name and product.price are columns whose names did not change, and they were read from disk. The key under which the quantity sat on disk was cart.quantity; because the query looked for cart.count, the row was not found even though it exists, and the answer fell to the default. The data was not lost; the new schema does not see it under its old name.

Now we apply the migration. There is only a rename, and the new column is not filled.

// Main.kt — the migration carries the renamed column
fun main() {
    val v1Rows = mapOf("profile.name" to "Ayse", "product.price" to "120", "cart.quantity" to "2")
    val migrated = migrate(v1Rows, mapOf("cart.quantity" to "cart.count"), emptyMap())
    val reads = readCartScreen(Tier("disk", migrated))
    reads.forEach { it.show() }
    countOrigins(reads)
}

profile-name: key=profile.name | value=Ayse -> disk
product-price: key=product.price | value=120 -> disk
cart-count: key=cart.count | value=2 -> disk
product-stock: key=product.stock | value=0 -> default
memory: 0
disk: 3
network: 0
default: 1

The quantity is now read from disk, and its value is 2. This lesson’s origin is disk: the answer for a migrated row belongs to disk. product.stock, on the other hand, is not on disk because it was never written in version 1.

DL1. The model builds the migration as a one-step mapping: it renames a key and, if asked, adds a value. The migration passing through intermediate versions, the table being recreated, and the transaction boundaries that protect the consistency of the data during a migration are absent from the model.

The Bounding Measurement

What happens to a new column in old rows? If the migration does not fill it, it does not exist in the old row, and as we saw in the measurement above, that field falls to the default. The default value of a column is the fourth parameter of Read. If the migration fills the column with a rule such as DEFAULT 0, the situation changes.

// Main.kt — the same value, two migrations: an unfilled column is default, a filled column is disk
fun main() {
    val v1Rows = mapOf("profile.name" to "Ayse", "product.price" to "120", "cart.quantity" to "2")
    val renames = mapOf("cart.quantity" to "cart.count")
    val plans = listOf(
        "without-backfill" to emptyMap<String, String>(),
        "with-backfill" to mapOf("product.stock" to "0"),
    )
    for ((name, backfill) in plans) {
        println("migration: $name")
        val migrated = migrate(v1Rows, renames, backfill)
        readCartScreen(Tier("disk", migrated)).filter { it.key == "product.stock" }.forEach { it.show() }
    }
}

migration: without-backfill
product-stock: key=product.stock | value=0 -> default
migration: with-backfill
product-stock: key=product.stock | value=0 -> disk

In both migrations the value of product.stock is 0. The origin, however, differs: in one it is the fallback value of the reader, and in the other it is a row written on disk. The rule does not see the value; it sees the first tier that holds the key. The benefit of this is that one can tell apart the rows in which a column was written from the rows in which it only appears to be. The person building the model sees that the backfill did not change the value, but it changed the owner of the answer.

What this measurement does not say is which migration is right. Writing a value into old rows grows the database and lengthens the migration; not writing it requires a fallback value everywhere the field is read. The rule does not weigh this cost.

DL2. The model treats a migration as either applied or not applied. A migration being interrupted halfway, a migration definition being incomplete, or the database being rolled back to an old version are absent from the model; in these cases the platform raises an error, and the model does not build that error.

Summary

• This lesson’s origin: disk. The read of a migrated row is answered from the disk tier.
• The entity carries the row, the data access object carries the query, and the database class carries the schema version.
• A migration carries old rows to the new schema; a row that was not carried is not found when it is searched under the new name and falls to the default: disk 2, default 2.
• With the rename migration, the quantity is read from disk with the value 2: disk 3, default 1.
• A new column that is not filled falls to the default, and a filled one becomes disk; in both cases the value is 0 and the origin differs.

Next Step

In this lesson, disk was a database with rows and columns. Not every piece of data is structured enough to enter a table: small preferences such as theme, language, and notification are written under a single key. The next lesson, Key–Value Storage, measures the origin from which a preference that was never written is read and how the origin changes when the user writes it.