On-Device Tests

In the previous lesson, test doubles took the place of the network, and no piece of the test ran on a device. The stub had its own answer measured; it did not measure how a real screen talks to a real system. Only a test that actually runs the screen can say that. This lesson’s question is the course’s question: in how many samples does this measurement exceed its budget? The four answers are the same: within, minority, majority, all. The unit of work measured is the work a test does before it starts and while it runs.

The core is the same as in the previous lessons: Budget carries a limit, and Overrun carries the samples of a measurement and how many of them strictly exceed the limit.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

What an Instrumented Test Runs

An instrumented test loads the app and the test code onto an Android device and runs them together with the system there. A unit test runs on the development machine, in the pure Kotlin layer; an instrumented test uses a real screen, a real lifecycle, and the system’s own services. This catches the defects a unit test cannot: a screen losing its state when it is rotated, a permission dialog failing to open, a database not actually being created on the system.

The place where tests run is of two kinds. A physical device is the hardware that will end up in a user’s hands; sensors, the camera, and manufacturer-specific system behavior are verified only there. An emulator is a software imitation of a device; it is easy to set up and to wipe, but it only imitates hardware behavior. In both, a test runner loads the app, runs the tests in order, and collects the result.

This way of running has a cost: each test has to bring up the screen it checks on its own. The test starts a screen, loads the necessary data, puts the user in a signed-in state, and only then moves on to its actual assertion. This preparation is called the test fixture, and the work the preparation does we call fixtureWork. On the platform, a test that starts a screen from scratch is written like this.

// Platform code: CartScreenTest
@RunWith(AndroidJUnit4::class)
class CartScreenTest {
    @Test
    fun showsEmptyCart() {
        ActivityScenario.launch(CartActivity::class.java).use { scenario ->
            scenario.onActivity { activity ->
                assertEquals(0, activity.itemCount())
            }
        }
    }
}

The launch call restarts the screen in every test; the use block closes it when the test ends. The model sets the rule of this measurement; the model is not the tool itself. In the model, the start-up work is counted; no screen is actually started.

TE1. The model does not run an instrumented test; there is no device or emulator in this environment. The setup work is a sum of steps: starting the screen is 6, loading the seed data is 4, signing in is 2. These numbers are inputs of the model, not a measurement of a device.

Setup Paid in Every Test

The model builds two functions. The first starts the screen from scratch in every test: the work of a test is the sum of the setup work and the work of the test’s own body. The second does the setup once per test class: only the first test pays for the setup, and the later ones find the screen ready.

// Fixture.kt — a test fixture that starts the screen from scratch: setup work is counted in steps
class Fixture(val steps: List<Int>) {
    val fixtureWork: Int = steps.sum()
}

fun freshPerTest(fixture: Fixture, bodies: List<Int>): List<Int> =
    bodies.map { fixture.fixtureWork + it }

fun sharedPerClass(fixture: Fixture, bodies: List<Int>): List<Int> =
    bodies.mapIndexed { index, body -> (if (index == 0) fixture.fixtureWork else 0) + body }

Both functions produce a list of work from a list of test bodies; these lists are the samples that will be given to Overrun. Think of eight tests whose bodies have work of 1, 2, 1, 3, 2, 1, 2, and 3. The budget limits the work a test may do to 10 units. This number is not a platform limit; it is the model’s counterpart of the target a team sets so that a test stays short.

// Main.kt
fun main() {
    val fixture = Fixture(listOf(6, 4, 2))
    val bodies = listOf(1, 2, 1, 3, 2, 1, 2, 3)
    println("fixture work: ${fixture.fixtureWork}")

    val work = freshPerTest(fixture, bodies)
    println("work per test: $work")

    val overrun = Overrun("setup-per-test", Budget("work", 10), work)
    overrun.line()
    countOverruns(listOf(overrun))
}

fixture work: 12
work per test: [13, 14, 13, 15, 14, 13, 14, 15]
setup-per-test: budget=work<=10 | over=8/8 -> all
within: 0
minority: 0
majority: 0
all: 1

The setup work alone is 12 units and is above the budget of 10 units. However small the bodies are, eight of eight tests went above the budget: the measurement is all. Here all gives a hint: the cause of the overrun is not the body of any test. The smallest body is 1 unit and it still reaches 13. The cause is the setup, and it is paid again in every test.

Loosening the budget would close the overrun: a budget of 16 units would leave all eight tests within. But that does not change the fact that the setup is expensive; it only pushes it out of the measurement. The rule does not see the magnitude of the overrun, and it does not see why the setup is expensive. The prose has to say that.

The Parts of the Setup

The 12-unit setup in the model consists of three parts, and each part corresponds to a separate job in a real test. Starting the screen is asking the system for a screen, bringing its lifecycle to its first states, and drawing it; for this reason it takes 6 units, the most expensive of the three. Loading the seed data is writing the records the screen will show into the database, and it is 4 units. Signing in brings the user to the state the screen expects, and it is 2 units.

None of these three parts belongs to the test’s actual assertion. If the test says “when the cart is empty, the empty message is visible”, the assertion is a single line. The previous three jobs only make the assertion possible. This is why the setup work is counted in a separate column from the test body.

There are two ways to shrink the setup. The first is to make the parts cheaper: loading the seed data with fewer records, or giving the sign-in through a fake authenticator. The second is to share the parts; doing it once per class is one form of this. The first gives up the realism of the test, the second the independence of the tests.

Reading the Platform Pieces Line by Line

In the CartScreenTest piece, the line @RunWith(AndroidJUnit4::class) declares that the test will run with an instrumented runner. ActivityScenario.launch(CartActivity::class.java) asks the system to actually start the screen, that is, the activity, and returns a scenario object. The .use { scenario -> ... } block closes the scenario when the block ends, whatever happens; a screen that is not closed affects the next test. The scenario.onActivity { activity -> ... } block gives access to the activity on the screen’s own thread. assertEquals(0, activity.itemCount()) is the assertion itself: there are zero items in the cart.

In the SeedOnce piece that we will see in the bounding measurement, the companion object holds members tied to the class in Kotlin. The @BeforeClass annotation asks for the method to run once before the first test in the class. @JvmStatic makes the method be generated as a real static method in the form the runner expects. seedDatabase() is the expensive part of the setup, and in the model it corresponds to the 4-unit seed load.

Physical Device or Emulator

The side that sets the budget also has to choose where the tests run. An emulator is quick to set up and can be reset with a single command; in a CI pipeline this means every run starts from a clean state. A physical device catches defects tied to hardware, but it does not guarantee a clean state in every run. A permission or an account left by a previous test can remain on the device.

The setup work measured in this lesson is independent of this choice. In both, the screen is started, data is loaded, and a sign-in is made. What changes is the cost of the work per unit, and the model counts the cost in steps, not in duration.

The Test Pyramid and the Budget

The expense of instrumented tests determines the shape of the test suite. Unit tests are fast and run in large numbers; instrumented tests are slow and run in small numbers. This distribution is called the test pyramid: at the base are many unit tests, in the middle are tests that bring components together, and at the top are a few end-to-end instrumented tests. A suite that turns the pyramid upside down, that is, that checks everything on a device, multiplies the setup cost we measured in this lesson by thousands of tests.

A team sets the budget separately for each test. The budget of a unit test can be a few steps; the budget of an instrumented test is set according to the setup. The same 10 units is a very different expectation in two different kinds of test. This is why, throughout the course, the budget of every measurement is written together with a reason.

Conditions of a Shared Setup

Sharing the setup per class depends on some conditions. The order of the tests must not affect the result; if a test benefits from the state another test left, it breaks when the order changes or when it runs alone. Tests that change the shared state must restore it when they finish. On the platform, this is done with a cleanup method that runs after every test. Finally, the shared setup must be read-only: the seed data is loaded once and no test changes it.

If one of these three conditions cannot be met, sharing the setup raises speed but lowers stability. The cart measurement showed this balance: in the shared cart, seven tests found something at the start. Cleanup also has a cost, and the cleanup done after every test takes back part of the gain the setup created.

Common Misconceptions

The first misconception is the thought that “an instrumented test replaces a unit test”. An instrumented test uses the real system and is therefore more expensive. Checking the edge cases of a function on a device means paying for the setup again for every edge. Edge cases are checked in unit tests, screen flows on a device.

The second misconception is the thought that “sharing the setup is always good”. A shared setup does not reduce the load of the first test; it only makes it be paid once. But shared state ties the tests to each other. Seven tests finding something at the start in the cart measurement was a concrete sign of this.

The third misconception is the thought that “closing an overrun by enlarging the budget is enough”. Raising the budget to 16 would move the measurement from all to within, but the cost of the setup would not change at all. A budget rests on a product decision; a budget changed to close an overrun destroys the meaning of the measurement.

The Bounding Measurement

The bounding measurement here is moving to the setup per class. The reasoning is this: in the first measurement the cause of the overrun was the setup, and the only thing to do to confirm it is to change where the setup is paid. The bodies, the budget, and the number of samples stay the same. If the overrun moves from all to minority, the cause really was the setup.

Now do the setup once per test class. The bodies of the tests, the budget, and the setup stay the same; only when the setup is paid changes. On the platform, this is done with a preparation method that runs once per class.

// Platform code: SeedOnce
class CartScreenTest {
    companion object {
        @BeforeClass
        @JvmStatic
        fun seedOnce() {
            seedDatabase()
        }
    }
}

The preparation method runs once before the first test in the class. In the model, this is the setup work being paid only in the first test.

// Main.kt — the setup is paid once per class
fun main() {
    val fixture = Fixture(listOf(6, 4, 2))
    val bodies = listOf(1, 2, 1, 3, 2, 1, 2, 3)

    val work = sharedPerClass(fixture, bodies)
    println("work per test: $work")

    val overrun = Overrun("setup-per-class", Budget("work", 10), work)
    overrun.line()
    countOverruns(listOf(overrun))
}

work per test: [13, 2, 1, 3, 2, 1, 2, 3]
setup-per-class: budget=work<=10 | over=1/8 -> minority
within: 0
minority: 1
majority: 0
all: 0

The same eight tests, the same budget. Seven tests stayed between 1 and 3 units; only the first test rose to 13. The measurement moved from all to minority. The overrun did not close entirely: the setup did not disappear, it was only piled onto a single test. The rule reads this as “one test exceeds”; that the first test carries the load of the setup has to be said again by the prose.

This gain is not free. When the setup is shared, the tests share the same screen, the same data, and the same system state. If one test puts an item in the cart, the next test finds the cart full. A measurement shows this: each test adds a certain number of items to the cart, and we count how many items are left in the cart when the next test starts. The budget is zero, because every test must start with an empty cart.

// Main.kt — shared state spoils the start of the next test
fun leftoverBefore(adds: List<Int>, cleanup: Boolean): List<Int> {
    val cart = mutableListOf<String>()
    return adds.map { count ->
        val leftover = cart.size
        repeat(count) { cart.add("item") }
        if (cleanup) cart.clear()
        leftover
    }
}

fun main() {
    val adds = listOf(1, 0, 2, 1, 0, 1, 1, 0)
    val shared = leftoverBefore(adds, false)
    val cleaned = leftoverBefore(adds, true)
    println("no cleanup: $shared")
    println("with cleanup: $cleaned")

    val clean = Budget("leftover", 0)
    val overruns = listOf(
        Overrun("shared-cart", clean, shared),
        Overrun("cleaned-cart", clean, cleaned),
    )
    overruns.forEach { it.line() }
    countOverruns(overruns)
}

no cleanup: [0, 1, 1, 3, 4, 4, 5, 6]
with cleanup: [0, 0, 0, 0, 0, 0, 0, 0]
shared-cart: budget=leftover<=0 | over=7/8 -> majority
cleaned-cart: budget=leftover<=0 | over=0/8 -> within
within: 1
minority: 0
majority: 1
all: 0

The shared cart is majority: apart from the first test, seven tests find something at the start. If cleanup is done after every test, the measurement is within, but cleanup is work too. Sharing the setup makes tests faster; cleaning the shared state takes back part of this gain. Which method to choose depends on whether the team puts the independence of tests or their duration first. The rule does not make this decision; it only counts the result of the decision.

TE2. The model builds the cart as a list and does the cleanup in a single step. In a real database, cleanup is an operation that varies with the number of tables and foreign keys; the model does not have this difference.

Summary

• This lesson’s overrun: all. Because every test started the screen from scratch, the 12-unit setup made eight of eight tests exceed the budget of 10 units.
• The lesson’s criterion, in how many tests the work of a test exceeds the budget: with setup in every test it came out all, with setup per class it came out minority.
• The cause of the overrun was not the test body; although the smallest body was 1 unit, the test rose to 13. The cause was the setup, and it was paid again in every test.
• Sharing the setup piled the load onto the first test, and the overrun moved from all to minority, but the setup did not disappear.
• Shared state spoiled the start of later tests: with no cleanup the measurement came out majority, with cleanup after every test it came out within.

Next Step

In this lesson the screen was started from scratch, and the question of which part of the screen the test would check was never asked. A real UI test has to find a node on the screen, and there is a way to find it. The next lesson, UI Tests, measures in how many samples looking a node up by its text and looking it up by its test tag exceed the budget.