--- title: 'Test Stability' source: 'https://academia.sh/en/courses/testing-performance-and-release/test-stability' course: 'Testing, Performance and Release' language: en updated: '2026-10-21T05:02:43+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines a flaky test as a result that changes from run to run on the same code' - 'Measures in how many runs a test that waits a fixed number of rounds exceeds its waiting budget against the ready round' - 'Compares the effect on the budget of lengthening the wait and of polling on a condition' - 'Recognizes the idling resource concept in a platform piece as a means of synchronization' --- # Test Stability A test that sometimes passes and sometimes fails on the same code is flaky. The lesson measures in how many runs a test that waits a fixed number of rounds exceeds its budget against the ready round, and how polling on a condition closes the overrun. In the previous lesson the test could find the node on the screen. But if it goes looking before the screen is ready, the same test passes in one run and fails in another. The unit of work measured in this lesson is a waiting round: a round the test spends while waiting for the screen to be ready. The course's question does not change: **in how many samples does this measurement exceed its budget?** The answers are again within, minority, majority, and all. The core is the same as in the previous lessons. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` ## The Anatomy of a Flaky Test A flaky test is a test that sometimes passes and sometimes fails although the code has not changed. The cause is mostly not in the test but in the things the test puts in order: the drawing of the screen, a network response, the end of a coroutine. The test does not know when these will happen, and it makes a guess: "it will be ready after two seconds". The guess holds in most runs and does not hold in one. This problem gets mixed up with three separate questions. In M30/K05 we asked whether a result changes with repetition or with order. In M13/K04 we asked what determines a result. Here neither is present: we do not name flakiness as an overrun category, and we do not search for its source. The measurement only counts how many runs the overrun spreads across. Flakiness is not a category; it is the mechanism that produces the overrun, that is, the way of waiting. The tool used on the platform against this problem is the idling resource, which provides synchronization. The resource reports to the test whether the app is busy, and the test postpones its action until the resource goes idle. ```kotlin // Platform code: BusyCounterResource class BusyCounterResource : IdlingResource { private val busy = AtomicInteger(0) private var callback: IdlingResource.ResourceCallback? = null override fun getName(): String = "busy-counter" override fun isIdleNow(): Boolean = busy.get() == 0 override fun registerIdleTransitionCallback(callback: IdlingResource.ResourceCallback) { this.callback = callback } fun begin() { busy.incrementAndGet() } fun end() { if (busy.decrementAndGet() == 0) callback?.onTransitionToIdle() } } ``` `begin` is called when a job starts, and `end` when a job finishes. If `isIdleNow` returns true, the test moves on. The test does not guess a duration; it asks the app itself. The model does not run these tools; instead it reads the round in which the screen became ready from a list and counts the rounds spent according to how the test does its waiting. **TE1.** In the model, the ready round is a fixed list; in a real run this value depends on the state of the network, the processor, and the queue at that moment. The list represents the ready rounds of eight runs and is not generated randomly. ## A Test That Waits a Fixed Number of Rounds The helper file defines the ready rounds and the two forms of waiting. ```kotlin // Stability.kt — ready rounds and two forms of waiting val readyTable: List = listOf(3, 4, 3, 8, 4, 3, 9, 4) fun pollRounds(run: Int): Int = readyTable[run] fun fixedWaitRounds(run: Int, wait: Int): Int = wait fun passesAfterFixedWait(run: Int, wait: Int): Boolean = pollRounds(run) <= wait ``` `pollRounds(run)` gives the round in which the screen became ready in that run. `fixedWaitRounds` is the test's fixed guess: it waits `wait` rounds whatever happens. `passesAfterFixedWait` says whether the test passes: it passes if the screen is ready before `wait` rounds have gone by. The test assumes the screen will be ready in five rounds. The ready rounds of eight runs are measured; the budget is the 5 rounds the test waits. ```kotlin // Main.kt fun main() { val runs = (0 until 8).toList() val ready = runs.map { pollRounds(it) } println("ready rounds: $ready") val failed = runs.count { !passesAfterFixedWait(it, 5) } println("wait 5 rounds, failed runs: $failed") val overrun = Overrun("fixed-wait", Budget("round", 5), ready) overrun.line() countOverruns(listOf(overrun)) } ``` ```text ready rounds: [3, 4, 3, 8, 4, 3, 9, 4] wait 5 rounds, failed runs: 2 fixed-wait: budget=round<=5 | over=2/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` In two of the eight runs the screen became ready after more than five rounds: the fourth run in 8 rounds, the seventh run in 9. The test fails in these two runs and passes in the other six. The measurement is minority. A test that passes in six of eight is not considered reliable: the two failing runs are noise for the team. The rule knows the number of overruns, but it cannot tell whether that noise is a regression or the ready round. ## Lengthening the Wait The first remedy that comes to mind is to lengthen the wait. If the test waits nine rounds, the screen is ready in every run and the test never fails. But this has a price: when the screen is ready in three rounds, the test still waits nine. Set the budget this time to the number of rounds a run may spend, that is, 5. ```kotlin // Main.kt — raising the wait to nine fun main() { val runs = (0 until 8).toList() val failed = runs.count { !passesAfterFixedWait(it, 9) } println("wait 9 rounds, failed runs: $failed") val spent = runs.map { fixedWaitRounds(it, 9) } println("rounds spent: $spent") val overrun = Overrun("long-wait", Budget("round", 5), spent) overrun.line() countOverruns(listOf(overrun)) } ``` ```text wait 9 rounds, failed runs: 0 rounds spent: [9, 9, 9, 9, 9, 9, 9, 9] long-wait: budget=round<=5 | over=8/8 -> all within: 0 minority: 0 majority: 0 all: 1 ``` The test became stable, but it exceeded the budget in every run: all. The flakiness is gone, and a constant cost took its place. A fixed wait makes you choose one of two kinds of failure: either a test that fails in some runs or a test that is slow in every run. **TE2.** The waiting round is a common unit in the model. In a real test, a round is not a single fixed duration but a waiting step of variable length; the model reduces it to a unit of work. ## The Sources of Flakiness For a test to be flaky the code does not have to be wrong; it is enough that the timing relationship between the test and the app is set up wrongly. Flakiness caused by timing has three typical scenes. The first is the UI thread. The app draws the interface on its own thread, while the test runs on another thread and may try to read the result right after tapping a button. If the drawing has not finished yet, the read sees the old state. The second is background work. A response from the network, a database write, or the end of a coroutine is not under the test's control. Because the test does not know when it will finish, it makes a guess, and the guess may not hold. The third is animation and transitions. While a screen opens, nodes stay in temporary states until they find their places. If the test taps a node in the middle of a transition, the node may not be tappable yet. The common point of these three scenes is that the test assumes an event has finished. The right remedy is to ask whether the event has finished, not to assume it. In the model, the round in which the screen became ready is the counterpart of this event, and the test either guesses it or asks for it. ## Synchronization Options There are four ways to synchronize a test with the app, and each carries a different cost. A fixed wait is the simplest and was used in the first two measurements of this lesson. The test stops for a certain duration or number of rounds. Its problem is this: if the wait is short the test breaks, and if it is long the test waits for nothing in every run. Polling is the test checking a condition at short intervals. When the condition is met the test moves on; if an upper limit is reached the test fails. This is the wait on a condition that we use in the last measurement. The cost of polling is that every check does its own work. An idling resource is the app itself saying "I am busy" or "I am idle". It differs from polling in that the app, not the test, knows the condition. When the app starts a job it increments the counter, when it finishes it decrements it; the test waits until the counter returns to zero. The resource is the class shown in the platform piece. For coroutines, a test scheduler uses a virtual clock and advances waits without any real time passing. Because the test can control time, this virtual clock is the most stable option, but it can be used only in work limited to coroutines and timers; the network and drawing do not know this clock. In this lesson the virtual clock is not measured, because the library's test package is not installed; it is only named. ## Reading the Idling Resource Line by Line `BusyCounterResource` implements the resource interface that the test framework recognizes. `busy` is an `AtomicInteger`: the counter can be safely incremented and decremented from several threads. `getName()` gives the name of the resource as it will appear in error messages. `isIdleNow()` is the question the test framework asks: if the counter is zero, the app is idle. `registerIdleTransitionCallback` lets the framework register a callback; it is stored to notify the framework when the resource goes idle. `begin()` is called when the app starts a job and increments the counter. `end()` decrements the counter when the job finishes; if the counter drops to zero, `onTransitionToIdle()` is called and the test framework stops waiting. The most common mistake is to make the `begin` call and forget the `end` call on an error path. Then the counter never returns to zero and the test waits forever. The remedy is to put the `end` call in a `finally` block. ## How the Budget Is Set The waiting budget answers two different questions, and mixing them up is a common mistake. The first question is "how many rounds is the test willing to wait at most". The second question is "in how many rounds at most does the app become ready". In a fixed wait, these two questions are answered with a single number: the test guesses in how many rounds at most the app will be ready and waits that long. If the guess is based on observed values it looks sufficient, but observed values do not cover new runs. In a wait on a condition, the two are separated. The test moves on when ready; the upper limit comes into play only when the app is never ready. In this lesson we set the upper limit to the worst observed round: 9. Eight runs represent a short period, so a real team sets the upper limit above the worst observed round, with a safety margin. This margin is a product decision. ## Measuring Stability Before repairing a flaky test, you need to know how flaky it is. Running the test in a single run is not enough; you have to run the same test many times by repetition and count how many times it failed. In this lesson, eight runs and the list of ready rounds are the model of this repetition. Failing in two of eight runs is a sign of flakiness for the team. The measurement also shows the effect of the repair. After the repair, the same number of runs is repeated, and if the number of failed runs is zero, the repair has held. But zero failed runs does not mean the test is stable; it only means it did not break in these runs. The evidence gets stronger as the number of runs increases. For this reason, the comparison before and after the repair uses the same number of runs. ## What to Do with a Flaky Test When a flaky test is found there are three ways. The first is to quarantine the test: the result is reported but does not stop the pipeline. Quarantine is a temporary measure and is put in place together with a deadline. The second is to find the root cause and repair it: tying the timing to a condition, cleaning the shared state, faking the dependency. The third is to delete the test; if a test gives a result that nobody trusts, it is more honest for it to give no result at all. If quarantine becomes a permanent solution, the test suite slowly fills with noise and the team gets used to red results; a red result that people are used to also hides a real defect. The team can track the number of tests in quarantine with a budget as well; this course's measure asks exactly this. ## Common Misconceptions The first misconception is the thought that "retrying a flaky test solves the problem". A retry makes the test pass, but it hides the overrun: a test that tries twice in one run pays the cost of two runs and erases the flakiness from the reports. What this lesson measures is exactly this hidden overrun. The second misconception is the thought that "the test is responsible for a flaky test". App code that leaves the timing undefined can also be the cause of flakiness. If the app does not report that it is ready, the test has to guess however well it is written. The idling resource ties these two sides together. The third misconception is the thought that "setting the wait duration once and forgetting it is enough". A nine-round wait may pass all the runs today; tomorrow the hardware or the app changes and a run appears that is ready in the tenth round. Fixed values age; a wait on a condition does not. ## The Bounding Measurement The bounding measurement here is polling on a condition. The reasoning is this: in the fixed wait there were two bad options, and both came from the guess. Replacing the guess with a condition makes it possible to measure the same runs again without changing the ready rounds or the budget at all. The budget is set this time to the worst observed round, because in a wait on a condition the goal is not to guess the overrun but to stay within the worst observed case. The third remedy is to tie the wait to a condition, not to a guess. In every round the test checks whether the screen is ready and moves on when it is; the idling resource does this on the platform. In this case the rounds spent equal the ready round of that run. The budget is also set not by failure but by the worst observed ready round: 9 rounds. ```kotlin // Main.kt — polling on a condition fun main() { val runs = (0 until 8).toList() val spent = runs.map { pollRounds(it) } println("rounds spent: $spent") val failed = runs.count { !passesAfterFixedWait(it, pollRounds(it)) } println("failed runs: $failed") val overrun = Overrun("condition-polling", Budget("round", 9), spent) overrun.line() countOverruns(listOf(overrun)) } ``` ```text rounds spent: [3, 4, 3, 8, 4, 3, 9, 4] failed runs: 0 condition-polling: budget=round<=9 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` Every run finished in its own ready round, no run failed, and the measurement stayed within the budget. This remedy did not pay the price that the previous two remedies paid: fast runs waited briefly, and slow runs waited for the slow screen. The limit of this result is in the budget. The 9-round budget comes from the worst observed round. If a run that is ready in the tenth round appears, the measurement has to be redone. Polling on a condition does not remove flakiness; it removes the mechanism that produces it, that is, the guess. ## Summary - This lesson's overrun: minority. The test that waited five rounds could not catch up with the ready round in two of the eight runs. - Flakiness is not an overrun category but the mechanism that produces the overrun; the measurement only counts how many runs the overrun spreads across. - Raising the wait to nine reset the failed runs to zero but carried the rounds spent above the budget in every run: all. - Polling on a condition finished every run in its own ready round and stayed within the 9-round budget. - An idling resource reports to the test whether the app is busy; the test does not guess the duration, it asks the app. ## Next Step The test is written, made stable, and running. But something can be said about the code itself without running a test. The next lesson, Static Analysis, measures in how many modules the rule violations that a linter finds per module exceed the budget.