--- title: Benchmarking source: 'https://academia.sh/en/courses/testing-performance-and-release/benchmarking' course: 'Testing, Performance and Release' language: en updated: '2026-10-21T05:02:43+00:00' license: 'CC BY-SA 4.0' objectives: - 'Explains that a benchmark consists of repeated iterations rather than a single number' - 'Measures from the model how warm-up iterations inflate the overrun count' - 'Shows how the regression between two versions looks once warm-up is dropped' - 'Distinguishes that the rule cannot see a regression when the budget is chosen wider than it' --- # Benchmarking Comparing two versions of a piece of work requires repeating the measurement. The lesson measures how warm-up iterations distort the measurement, how a regression looks once they are dropped, and how the rule misses it when the budget is chosen wider than the regression. In the previous lesson we measured the request count once on a stream and accepted the result as certain. A real measurement is not a single number. When you run the same work back to back, the first runs take a different time from the later ones: classes are loaded, code is compiled, caches fill. We call these first iterations **warm-up**. When a measurement includes warm-up, it measures not the steady state of the work but the road travelled until the steady state is reached. The question of this lesson is this: how do we tell whether a change slowed the work down, that is, whether there is a **regression**, while warm-up distorts the measurement. The answer has two parts: separating warm-up from the measurement and choosing the budget narrow enough to catch the regression. The measure is the course's measure again: in how many of its iterations the work exceeds the budget. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Budget` carries the name and limit of a measurement, and `Overrun` counts the samples that exceed the limit and gives one of four overruns. This file is the same in every lesson of the course. ## The Iterative Measurement Model In the model the cost of a piece of work is its number of work steps. The work has two versions: `old` and `new`. After a change, the new version does three more steps in every run; this is our regression. Warm-up is the extra steps added to the first iterations, and it shrinks as the iterations progress. ```kotlin // Bench.kt — iterative measurement of a piece of work; the first iterations carry a warm-up share fun workOf(version: String, size: Int): Int = when (version) { "old" -> size * 2 "new" -> size * 2 + 3 else -> size } fun iterations(version: String, size: Int, runs: Int, warmup: Int): List = (0 until runs).map { run -> workOf(version, size) + if (run < warmup) (warmup - run) * 4 else 0 } fun dropWarmup(samples: List, warmup: Int): List = samples.drop(warmup) ``` `workOf` gives the steady-state cost of a version. `iterations` repeats this cost `runs` times and adds a decreasing warm-up share to the first `warmup` iterations. `dropWarmup` removes the warm-up iterations from the measurement. For a piece of work of five units, the steady cost of the `old` version is ten and that of the `new` version is thirteen. We first measure the old version with eight iterations, warm-up included. Let the budget be twelve steps. ```kotlin // Main.kt — the old version is measured with the warm-up iterations included fun main() { val samples = iterations("old", 5, 8, 3) println("old: $samples") val overrun = Overrun("old", Budget("step", 12), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text old: [22, 18, 14, 10, 10, 10, 10, 10] old: budget=step<=12 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The steady cost of the version is below the twelve-step budget, but the first three iterations exceeded it because of warm-up: 3/8, **minority**. The old version does work that fits the budget; the measurement wrote three overruns against it. This is warm-up distorting the measurement. **PD1.** In the model the warm-up share decreases linearly and the number of iterations is known exactly. In reality, when warm-up ends depends on the work, the device and the state of the system at that moment; it is not possible to say after which iteration the steady state is reached without measuring. The model counts how many work steps each iteration does; it does not set up how long an iteration takes. ## Dropping Warm-Up and Seeing the Regression To compare, we measure both versions with warm-up dropped and hold them to the same budget. ```kotlin // Main.kt — the two versions are compared with warm-up dropped, under the same budget fun main() { val warmup = 3 val oldSamples = dropWarmup(iterations("old", 5, 8, warmup), warmup) val newSamples = dropWarmup(iterations("new", 5, 8, warmup), warmup) println("old: $oldSamples") println("new: $newSamples") val overruns = listOf( Overrun("old", Budget("step", 12), oldSamples), Overrun("new", Budget("step", 12), newSamples), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text old: [10, 10, 10, 10, 10] new: [13, 13, 13, 13, 13] old: budget=step<=12 | over=0/5 -> within new: budget=step<=12 | over=5/5 -> all within: 1 minority: 0 majority: 0 all: 1 ``` With warm-up dropped the measurement became clean: the old version is **within**, the new version is **all**. The three overruns written against the old version in the previous measurement were warm-up's share; in this measurement the new version exceeding in five of five shows a real regression. The difference in the steady state is three steps, and the budget was chosen at twelve steps so that it passes right through this difference. On the platform these two distinctions are set up with a rule object that repeats the measurement. In the piece below, the number of warm-up iterations and the number of iterations to measure are given explicitly. ```kotlin // Platform code: repeated benchmark import androidx.benchmark.macro.CompilationMode import androidx.benchmark.macro.StartupMode import androidx.benchmark.macro.StartupTimingMetric import androidx.benchmark.macro.junit4.MacrobenchmarkRule import androidx.test.ext.junit.runners.AndroidJUnit4 import org.junit.Rule import org.junit.Test import org.junit.runner.RunWith @RunWith(AndroidJUnit4::class) class StartupBenchmark { @get:Rule val benchmarkRule = MacrobenchmarkRule() @Test fun coldStartup() = benchmarkRule.measureRepeated( packageName = "com.example.app", metrics = listOf(StartupTimingMetric()), iterations = 5, startupMode = StartupMode.COLD, compilationMode = CompilationMode.Partial(warmupIterations = 3), ) { pressHome() startActivityAndWait() } } ``` This piece is platform code, and here we claim no output, duration or ratio. The rule object repeats the measurement five times with `measureRepeated` and runs the warm-up iterations separately, in the number specified; which durations come out depends on the device and the app. The model sets up the rule of this measurement; the model is not the tool itself. When the tool counts warm-up as finished is information the model does not show. ## The Iteration Count Changes the Measurement If warm-up is not dropped, the overrun count also depends on the number of iterations, because the warm-up share is fixed and its weight shrinks as the iteration count grows. We measure the same old version, warm-up included, with four, eight and sixteen iterations. ```kotlin // Main.kt — the same version is measured, warm-up included, with three different iteration counts fun main() { val overruns = listOf(4, 8, 16).map { runs -> Overrun("runs-$runs", Budget("step", 12), iterations("old", 5, runs, 3)) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text runs-4: budget=step<=12 | over=3/4 -> majority runs-8: budget=step<=12 | over=3/8 -> minority runs-16: budget=step<=12 | over=3/16 -> minority within: 0 minority: 2 majority: 1 all: 0 ``` The version is the same, the warm-up is the same, the number of overrunning iterations is three. Only the iteration count changed, and the overrun moved from majority to minority. For this reason, in a measurement that includes warm-up the iterations decide the result; in a measurement that drops warm-up the steady state decides it. For a benchmark to be reliable, both are fixed together: how many iterations will be measured and how many will be counted as warm-up and dropped. ## Micro and Macro Benchmarks Benchmarking is done at two levels. At the micro level a single function or a small piece of code is run repeatedly in an isolated environment; the aim is to separate the cost of that piece from other work. At the macro level a flow of the app, for example startup or the opening of a screen, is measured end to end; the aim is to see the result the user experiences. The two do not replace each other. The micro measurement may look improved while the macro measurement stays the same, because the measured piece is a small part of the flow. The macro measurement may show a regression while the micro measurement comes out clean, because the problem lies in the interaction of the pieces. In this lesson the model measures a single unit of work; a real project uses both levels. ## The Baseline A reference point is needed to catch a regression. This point is the measurement of the accepted version of the work, and it is kept as the **baseline**. The new measurement is compared with this value; if the difference exceeds a certain limit, it counts as a regression. In the lesson's model the baseline is the steady-state cost of the old version. The baseline is itself a measurement and is affected by warm-up and noise. Taking the baseline with warm-up included spoils all later comparisons. For this reason the baseline is computed only from steady iterations and recorded together with the conditions in which it was measured. ## Why Warm-Up Exists Warm-up means the first runs of the work are more expensive than the later ones. In the first run the code may not be compiled yet, the caches are empty, the resources used are opened for the first time. After a few iterations these costs are paid and the work reaches its steady state. In the model, warm-up is a decreasing share added to the first iterations; the shape of real warm-up varies with the work. Dropping warm-up from the measurement means choosing to measure the steady state. This choice is right, because to catch a regression the work itself must be measured, not the luck of the first run. But a measurement that drops warm-up does not see the cost the user meets first. For this reason a measurement of startup and a measurement of regression are kept separate: in the first, warm-up is the subject of the measurement; in the second, it is noise. ## The Setup Must Stay Fixed In a benchmark the two versions are measured not side by side but one after the other in the same setup. If the setup changes, for example if a measurement is taken on another device or in another system state, the difference may come not from the regression but from the setup. In this lesson the setup is the model itself and is the same in every measurement; in a real project the same assurance is provided only by fixing the device, the system state and the iteration count. This fixedness also determines how measurement results are stored. When a baseline is recorded, which version it was taken on, with how many iterations, and whether warm-up was dropped or not is written beside it. Without this information two numbers cannot be compared, because the lesson showed that the same version can give two different overruns depending on the iteration count. Where the budget comes from must also be asked. In this lesson twelve steps was placed between ten, the steady cost of the old version, and thirteen, the cost of the new version, and decided in advance which version would stay within it. In a real project the budget is set by adding a small margin to the measurement of the accepted version; the margin must be larger than the width of the noise and smaller than the size of the regression. If the margin is too large the rule cannot see the regression; if it is too small it raises a false alarm in every measurement. When reporting results, the distribution of the numbers is given instead of a single number: the best iteration, the worst iteration and the middle one. In this lesson the number of work steps per iteration is the samples themselves and is printed under every measurement. The single line the rule writes summarizes the overrun count; when the samples are read, one sees where warm-up ended and where the regression began. The decision made as a result of a benchmark must also be written down. If a change introduced a regression there are three ways: revert the change, accept the regression and raise the baseline, or compensate for the regression somewhere else. If the choice and its reason are not recorded, a later measurement cannot explain why the new baseline is what it is. In this lesson the three-step difference was clearly a regression; the product decision lies between accepting it or not, and the rule does not give that. The order of the iterations also carries information. Because the warm-up share is concentrated in the first iterations, the beginning of the sample series measures something different from its end. Shuffling the series and taking its average mixes warm-up into the steady state and combines two separate phenomena in a single number. Samples printed in order show, without eyeballing, the iteration where warm-up ended, and are the basis for choosing how many iterations to drop. ## Common Misconceptions **"Measuring once is enough."** A single measurement cannot separate warm-up from noise. The lesson showed this by the same version giving different overruns depending on the iteration count. **"Dropping warm-up makes the measurement look nicer."** When warm-up is dropped, what the measurement targets must be clear. The user pays the warm-up cost when opening the app for the first time; a measurement that wants to measure this cost does not drop warm-up, it takes it as a separate measurement. **"A small difference is a regression."** While noise exists, a small difference may come from something other than a regression. In this lesson the model is noiseless, so the three-step difference is certain. In a real measurement, iterations are multiplied to be sure the difference is larger than the noise. **"The rule finds the regression."** The rule only counts the samples that exceed the budget. If the budget is above the regression, the rule cannot see it; the lesson's bounding measurement showed this. ## The Bounding Measurement The most important limit of the rule is this: if the budget is chosen wider than the regression, the rule cannot see the regression. We show it with two trials. In the first, warm-up is included and the budget is twenty steps. In the second, warm-up is dropped and the budget is thirteen steps. ```kotlin // Main.kt — under a wide budget the regression escapes the rule fun main() { val warmup = 3 val oldAll = iterations("old", 5, 8, warmup) val newAll = iterations("new", 5, 8, warmup) val oldSteady = dropWarmup(oldAll, warmup) val newSteady = dropWarmup(newAll, warmup) val overruns = listOf( Overrun("old-warm", Budget("step", 20), oldAll), Overrun("new-warm", Budget("step", 20), newAll), Overrun("old-steady", Budget("step", 13), oldSteady), Overrun("new-steady", Budget("step", 13), newSteady), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text old-warm: budget=step<=20 | over=1/8 -> minority new-warm: budget=step<=20 | over=2/8 -> minority old-steady: budget=step<=13 | over=0/5 -> within new-steady: budget=step<=13 | over=0/5 -> within within: 2 minority: 2 majority: 0 all: 0 ``` Under the twenty-step budget both versions are **minority**; under the thirteen-step budget both are **within**. Although the new version has slowed by three steps, the rule wrote the same overrun to both versions in both cases. In the measurement with warm-up, the new version's overrunning iteration count is two and the old version's is one: the difference is one iteration, and from the sample count alone it cannot be told whether it came from the regression or from warm-up. In the measurement with warm-up dropped, thirteen steps is exactly the new version's limit, and a sample equal to the limit does not count as an overrun. The rule sees the number of overruns; it does not see their size, that is, the three-step difference. The same function can give two overruns under two budgets; unless the budget is set at a value below the regression, the regression stays inside the rule. **PD2.** The regression in the model is the same three steps in every iteration. Real regressions do not arrive the same in every iteration; some appear only on certain inputs or certain devices. It cannot be concluded that a regression not visible in one measurement setup will not be visible in another. ## Summary - This lesson's overrun: minority. The steady cost of the old version did not exceed the budget, but the first three warm-up iterations did, and three of the eight iterations went past the budget. - Warm-up iterations measure not the steady state of the work but the road to reaching it. For comparison, warm-up is dropped from the measurement. - Once warm-up was dropped, the old version came out within and the new version all; the three-step regression became visible. - In a measurement that includes warm-up, the same version came out majority or minority depending on the iteration count. - If the budget is chosen wider than the regression, the rule writes the same overrun to both versions: it sees the number of overruns, not their size. ## Next Step This lesson fixed what a measurement measures. What remains is how the measured thing enters the package to be published. The next lesson, `Code Shrinking and Obfuscation`, measures which classes keep rules hold alive while unused classes are stripped from the app package, and how a broad rule defeats shrinking.