--- title: 'Testing, Performance and Release' source: 'https://academia.sh/en/courses/testing-performance-and-release' language: en updated: '2026-10-21T05:02:42+00:00' license: 'CC BY-SA 4.0' lessons: 19 outcomes: - 'Builds unit and on-device tests layer by layer' - 'Fixes startup time and smoothness problems by measuring them' - 'Publishes the app package to store tracks' - 'Distinguishes whether a measurement exceeds its budget in no sample, in a minority of samples, in a majority, or in all of them' --- # Testing, Performance and Release Every step that takes an app to production compares a measurement with a budget: a unit test the steps a function takes, a startup measurement the work done before the first screen, a rendering measurement the work done in a frame, a staged rollout the crashes in each stage. A single measurement misleads; the same measurement is taken over several samples. The course measures every mechanism with a single question — in how many samples does this measurement exceed its budget? Unit, on-device and UI tests, test stability, static analysis, startup time, rendering smoothness, memory, network and battery profiling, benchmarking, code shrinking, app packages, version management, continuous integration, testing tracks, staged rollout, store review and post-release monitoring all pass through the same frame. ## Testing Unit tests for pure Kotlin layers, mocking dependencies, on-device tests, component-based and classic UI tests, flakiness caused by waiting and concurrency, and static analysis. ### Unit Tests Source: https://academia.sh/en/courses/testing-performance-and-release/unit-tests A unit test pins the number a function in the pure Kotlin layer produces for a set of inputs. The lesson classifies in how many samples a measurement exceeds its budget with four answers: within, minority, majority, all. The previous course counted how many copies of an event reached a recipient. That course's measurement axis does not continue here. In this course the app is taken to production, and every step toward production compares a measurement with a limit: a unit test compares the steps a function takes, an instrumented test compares the number of rounds of a wait loop, a startup measurement compares the work done before the first screen, and a staged rollout compares the crashes in each stage. A single measurement misleads, because on a single input everything can look fine. So the same measurement is taken on several inputs, and each input becomes a sample. The course's one question starts with this lesson: **in how many samples does this measurement exceed its budget?** A budget is the limit a measurement is expected not to cross. The answer has four forms, and their names do not change throughout the course. If no sample exceeds the budget, the measurement is **within**. If some samples exceed it but fewer than half of the samples do, it is **minority**. If at least half of the samples exceed it but not all, it is **majority**. If every sample exceeds it, it is **all**. The umbrella name for the four answers is overrun; when we speak of a measurement's overrun, we mean one of these four answers. #### The Rule of the Measurement The label does not give the overrun; the rule does. The core below defines a budget (`Budget`) and a measurement (`Overrun`) with a label, a budget, and a list of samples. It takes the number of samples and the number of samples that strictly exceed the limit, then asks the question in order. A sample equal to the limit does not count as exceeding. Exactly half also counts as majority: if four of eight samples exceed the budget, the measurement is majority, not minority. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `countOverruns` counts the four overruns in a fixed order and prints an overrun even when its count is zero. The rule misses two things. The first is magnitude: a sample that exceeds the budget by one step counts the same as a sample that exceeds it by ten steps. The second is cause: a slow algorithm, an extra call, and a budget set too tight can all produce the same overrun. The prose of the lesson states the magnitude and the cause; the rule does not. Writing a list of samples by hand shows the rule, but it measures nothing; it is a demonstration, not a measurement. In this course, every measurement that carries an overrun takes its samples from the return values of a model function: the function produces a unit of work from an input, and the list of samples is the numbers the function returns for each input. Depending on the lesson, the unit of work is a step, a round, a request, or a crash; it is never a duration. The budget is an input of the model as well: when a lesson sets a budget, it states the reasoning, and it does not present the budget as a platform threshold. **TE1.** The model does not run the test framework itself; there is no framework in this environment. In the model, what a test measures is the number of steps a function takes for one input. The test class, on the other hand, is a platform piece and prints nothing. #### What a Unit Test Pins Down A unit test runs a function in the pure Kotlin layer of the app and compares its result with an expectation. The function that runs is the test target (system under test), and the call that states the expectation is the assertion. The pure Kotlin layer knows nothing about the screen, the file system, the network, or the clock; the function always returns the same result for the same input. For that reason the test runs on the development machine in very little time, without a device or an emulator. An assertion pins a single number for a single input. The behavior of the function, however, changes with the input: a function that takes two steps on one input can take ten steps on another. Once a set of inputs is chosen, the unit test turns into a collection of assertions that pin the number for every input in that set. On the platform it looks like this. ```kotlin // Platform code: ValidatorTest import org.junit.Assert.assertEquals import org.junit.Test class ValidatorTest { @Test fun acceptsShortCode() { assertEquals(4, validateSteps("AB12")) } @Test fun stopsAtFirstBrokenCharacter() { assertEquals(2, validateSteps("A-1234")) } } ``` Each `@Test` method pins one input. The model sets the rule of this measurement; the model is not the tool itself. In the model, instead of assertions we look at the rule: how many of the samples exceed the budget. It is easy to confuse this question with two questions from earlier courses. In M30/K06 we compared a test with a release pair and asked whether this test separates the correct release from the faulty one. What was measured there was the correctness of the test, and every measurement rested on a pair made of one test and one release. There is no pair here: there is a single measurement, a single budget, and a list of samples. The question is not the correctness of the test but how many times the budget is exceeded. #### Eight Samples, Four Searches Suppose the pure Kotlin layer of an app has a product code search. The codes are in order from 1 to 16, and eight codes are searched. Four search methods are measured against the same budget: a search may take at most four steps. We do not present four steps as a platform threshold; we set it as the model's counterpart of the share a product screen reserves for one search. `scanSteps` scans from the start and takes as many steps as the position of the searched code. `halveSteps` halves the range at every step. `filterSteps` walks the whole list even after it finds the searched code. `bucketSteps` places a code into a bucket by its remainder modulo eight and searches only in that bucket. Each function returns the number of steps it took to find the searched code. ```kotlin // Main.kt fun scanSteps(items: List, target: Int): Int { var steps = 0 for (item in items) { steps++ if (item == target) return steps } return steps } fun halveSteps(items: List, target: Int): Int { var low = 0 var high = items.size - 1 var steps = 0 while (low <= high) { steps++ val mid = (low + high) / 2 when { items[mid] == target -> return steps items[mid] < target -> low = mid + 1 else -> high = mid - 1 } } return steps } fun filterSteps(items: List, target: Int): Int { var steps = 0 items.filter { steps++; it == target } return steps } fun bucketSteps(items: List, target: Int): Int = items.filter { it % 8 == target % 8 }.indexOf(target) + 1 fun main() { val codes = (1..16).toList() val targets = listOf(2, 5, 7, 9, 12, 14, 16, 3) println("searched codes: $targets") val step = Budget("step", 4) val overruns = listOf( Overrun("linear-search", step, targets.map { scanSteps(codes, it) }), Overrun("binary-search", step, targets.map { halveSteps(codes, it) }), Overrun("full-filter", step, targets.map { filterSteps(codes, it) }), Overrun("bucket-lookup", step, targets.map { bucketSteps(codes, it) }), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text searched codes: [2, 5, 7, 9, 12, 14, 16, 3] linear-search: budget=step<=4 | over=6/8 -> majority binary-search: budget=step<=4 | over=1/8 -> minority full-filter: budget=step<=4 | over=8/8 -> all bucket-lookup: budget=step<=4 | over=0/8 -> within within: 1 minority: 1 majority: 1 all: 1 ``` Each of the four measurements gave a different overrun: within 1, minority 1, majority 1, all 1. This distribution is the binding number of the course; the number of samples is 8 and the budget is 4 steps. The samples of each measurement are the list that the corresponding function returned for the eight inputs. `linear-search` is majority. The positions of codes 5, 7, 9, 12, 14, and 16 are greater than four, while 2 and 3 stay under the limit. Six samples are more than half of eight. `binary-search` is minority: the only place where sixteen codes need five steps is the last code, so only 16 exceeds. `full-filter` takes 16 steps in every sample and is all; even when the searched code is among the first, the filter walks the list to the end. `bucketSteps` puts two codes in each bucket, so no sample goes beyond two steps and the measurement is within. The difference between majority and all lies not in the number but in the inference. If it is majority, the budget fails only for some inputs, and the choice of inputs matters. If it is all, no input can meet the budget, and the function itself is outside the budget. #### Input Validation: A Budget From the Worst Case A unit test is set up not only for search but for every pure function. Take a function that validates a campaign code. The function walks the characters of the code in order and stops as soon as it sees an invalid character. A code has at most six characters, so the worst case of the function is six steps. Setting the budget at this worst case is the way to pin that the function never exceeds its own limit on any input. In this lesson the function stays in a helper file and the later measurements call it. ```kotlin // Validation.kt — walks the characters of a campaign code in order, stops at an invalid character fun validateSteps(code: String): Int { var steps = 0 for (ch in code) { steps++ if (!ch.isLetterOrDigit()) return steps } return steps } ``` We chose eight codes. One of the codes is short, two are broken right at the start, and three are long enough to walk all six characters. The list of codes will stay the same; only the budget will change. ```kotlin // Main.kt — the input validation budget is set from the worst case fun main() { val codes = listOf("AB12", "A-1234", "SUM24", "123456", "K 9", "TR2026", "ZZ", "P7Q8R9") val steps = codes.map { validateSteps(it) } println("tried codes: $codes") println("steps: $steps") val worst = Budget("step", 6) val overrun = Overrun("code-validation", worst, steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text tried codes: [AB12, A-1234, SUM24, 123456, K 9, TR2026, ZZ, P7Q8R9] steps: [4, 2, 5, 6, 2, 6, 2, 6] code-validation: budget=step<=6 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` The steps of the eight codes came out as 4, 2, 5, 6, 2, 6, 2, and 6. None went past six, so the measurement is within. The three codes that take six steps sit exactly at the limit of the budget, and a sample equal to the limit does not count as exceeding. The value of this measurement is not a number but a contract: the function will change in the future, and if one code takes seven steps, a test that said within up to that moment will break. This is why a unit test is not written once and forgotten; with every change, the same eight inputs are measured again with the same budget. **TE2.** The budget of 6 steps derives from the character count of the longest code for this function. On the platform, the product team can choose a different worst case; the model sets this choice as a budget and does not turn it into a threshold assertion. #### Common Misconceptions Three misconceptions about unit tests keep coming back, and the measurement gives each one a concrete answer. The first is the thought that "if the test is green, the function is correct". A green test says only that the assertions pinned for the chosen inputs hold. If the set of inputs is chosen narrowly, the worst case can stay outside the set. This is why we set the budget from the step count of the longest code in this lesson: if the budget does not rest on the hardest member of the set, the answer within says nothing. The second is the thought that "the tighter the budget, the better". A tight budget raises the overrun, but the function does not get worse because the overrun count went up; only the limit moved. The same function came out within at a budget of 6 steps and minority at a budget of 5 steps. The function was the same. What was measured was not the function itself but its relation to the limit. This is why a budget value is always written together with a reason, and when the reason changes, the budget is reviewed too. The third is the thought that "a unit test looks only for bugs". Here the test pins not a bug but a number. If the number moves in the next change, the test breaks and the team notices the change. Even in a bug-free function, a silent rise in the step count is the first sign that the budget is being approached; the test exists to catch that sign. What the three misconceptions share is reading more than a single result can give. The rule says only this: in how many samples the limit was strictly exceeded. Anything beyond that, that is, why the overrun happened, is the team's work. #### The Bounding Measurement Tighten the budget. The function did not change and the list of codes did not change; only the budget went down from 6 steps to 5 steps. The same eight samples will be compared with the new limit. ```kotlin // Main.kt — same function, same codes, tighter budget fun main() { val codes = listOf("AB12", "A-1234", "SUM24", "123456", "K 9", "TR2026", "ZZ", "P7Q8R9") val steps = codes.map { validateSteps(it) } val tight = Budget("step", 5) val overrun = Overrun("code-validation-tight", tight, steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text code-validation-tight: budget=step<=5 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The same function, the same eight inputs, and the same steps: 4, 2, 5, 6, 2, 6, 2, 6. When the budget drops to 5, the three codes that take six steps exceed it. Three of eight samples exceed, and because three is under half, the measurement is minority. The count was zero in the previous measurement. Not a single line changed in the function. This is not a contradiction; it is the measurement itself. The overrun is a property not of the function alone but of the function and the budget together. The same function can be within at one budget, minority at another, and majority or all at an even tighter one. This is why, when a measurement's overrun is stated, the budget is stated too: "code validation is within" alone is not information; "code validation is within at a budget of 6 steps" is information. The two things the rule misses apply here as well. It misses magnitude: three codes that exceed a five-step budget by one step are, in the eyes of the rule, the same as three codes that exceed it by ten steps. It misses cause as well: in this measurement the cause of the overrun is not the function, because the function stayed the same; the cause is that the budget was set tighter. Someone writing a unit test should, when reading a measurement's overrun, look first not at the function but at what the budget was set from. Two lessons come out of the four searches. The first is that the same budget can give four overruns on four functions: keeping the budget fixed tells the functions apart. The second is that the same function can give two overruns at two budgets: keeping the function fixed tells the budgets apart. A unit test pins something in both directions. By fixing the set of inputs it locks the function, and by fixing the budget it locks the expectation. #### Summary - This lesson's overrun: within. Input validation with a budget of six steps did not exceed the budget in any of the eight codes. - The lesson's criterion, in how many samples a measurement exceeds its budget: the four searches came out within 1, minority 1, majority 1, all 1; the number of samples was 8 and the budget was 4 steps. - A unit test pins a number for a set of inputs, not for a single input; a function in the pure Kotlin layer gives the same result for the same input. - The same function and the same eight codes moved from within to minority when the budget dropped from 6 steps to 5 steps: the overrun is a property of the function and the budget together, not of the function alone. - The rule sees how many samples overrun, not by how much or why; the prose states both. #### Next Step The functions measured in this lesson touched no outside resource. A real repository, however, calls a network resource, and a unit test does not want that call. The next lesson, Mocking, measures in how many samples the call count exceeds the budget when a dependency is replaced with its test double. ### Mocking Source: https://academia.sh/en/courses/testing-performance-and-release/mocking Mocking puts a test double in place of a real dependency in a test. The lesson counts how many times a repository goes to the network source in each operation and shows what a test double does and does not let you measure. The functions measured in the previous lesson touched no outside resource: an input came in and a step count came out. A real repository, however, goes to a network source, and a unit test does not want that trip. A test that reaches the network gets slow, depends on the state of the network, and can give two different results for the same input. The fix is to replace the dependency with its test double. In this lesson the course's question stays the same: **in how many samples does this measurement exceed its budget?** The four answers are the same too: within, minority, majority, all. The unit of work measured in this lesson is the number of calls that go to a network source. The core is the same as in the previous lesson. `Budget` carries the name and the limit of a budget, `Overrun` takes the label, the budget, and the list of samples of a measurement; it counts the samples that strictly exceed the limit and picks the overrun. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### The Four Kinds of Test Double Every place where the class under test connects to the outside is called a dependency. The object that takes the place of this dependency in a test is called a test double. A test double has four kinds by role, and each goes by its own name. A dummy only fills a parameter; it is never called and gives no answer. A stub gives a predetermined answer: whatever it is asked, it returns its ready answer. A spy records and counts calls; it can keep the real behavior or give a ready answer. A mock knows the expected calls in advance and breaks the test if an expected call does not arrive or if too many arrive. In this lesson's model we will build the spy and the stub; we will see the mock in the platform piece. Because the concepts overlap, drawing a line helps: the stub decides the answer, the spy counts the call. The stub carries the answer to "what was answered", the spy carries the answer to "how many times was it asked". The number measured in this lesson is the second: how many times a repository goes to the network source. #### Two Calls, One Budget Think of an order repository. When it reads an order, the repository goes to the network source twice, first for a freshness check and then to fetch the actual data. Whether this behavior is a bug is a product decision; the model takes the behavior as it is and counts it. The network source is an interface. In production a class that goes to the network implements it; in a test a spy that counts calls takes its place. ```kotlin // Source.kt — network source contract, a spy that counts calls, and a cache that gives ready answers as a stub interface RemoteSource { fun fetch(key: String): String } class SpySource(private val replies: Map) : RemoteSource { var requestCount: Int = 0 private set override fun fetch(key: String): String { requestCount++ return replies[key] ?: "none" } } class CacheStub(private val entries: Map) { fun lookup(key: String): String? = entries[key] } ``` `SpySource` increases the `requestCount` counter by one on every `fetch` call; the counter is read from outside but written only inside the class. `CacheStub` only returns its ready answers and counts no call. Now the repository. ```kotlin // OrderRepository.kt — calls the network source twice per operation: one is a freshness check class OrderRepository( private val source: RemoteSource, private val cache: CacheStub = CacheStub(emptyMap()), ) { fun load(key: String): String { val cached = cache.lookup(key) if (cached != null) return cached source.fetch(key) return source.fetch(key) } } ``` The repository looks at the cache first. If the cache gives an answer, it does not go to the network; if it does not, it goes twice. The default cache is empty, that is, it never gives an answer. The measuring function also goes into a helper file: for each operation it builds a separate spy, runs the repository, and returns the call count seen in that operation. ```kotlin // Probe.kt — measures the number of calls that go to the network source in each operation fun callsPerOperation(keys: List, stub: CacheStub = CacheStub(emptyMap())): List = keys.map { key -> val spy = SpySource(mapOf(key to "network")) OrderRepository(spy, stub).load(key) spy.requestCount } ``` This function produces a list of numbers from a list of inputs; its return value is what will be given to `Overrun` as the samples. Eight orders will be read and the budget is one call per operation. This budget is the model's counterpart of a product decision that wants the screen to go to the network once when it opens an order; neither the platform nor a network provider sets such a limit. **TE1.** The model does not run the mocking library; it is not present in this environment. The spy and the stub are hand-written classes, not objects produced by the library. The budget of 1 call per operation is an input of the model. ```kotlin // Main.kt fun main() { val keys = listOf( "order-1", "order-2", "order-3", "order-4", "order-5", "order-6", "order-7", "order-8", ) val calls = callsPerOperation(keys) println("call counts: $calls") val overrun = Overrun("repo-calls", Budget("call", 1), calls) overrun.line() countOverruns(listOf(overrun)) } ``` ```text call counts: [2, 2, 2, 2, 2, 2, 2, 2] repo-calls: budget=call<=1 | over=8/8 -> all within: 0 minority: 0 majority: 0 all: 1 ``` All eight operations made two calls, and each exceeded the budget by one call. The measurement is all. All says that the function is outside the budget: changing the input does not change the result, because the source of the overrun is not a property of an input but the repository's own design. The rule does not see magnitude, and here the difference is one call; the prose says that this one call equals one network trip. #### Call Verification A unit test can pin the same number. The spy counts the call, and the test verifies that the counter has the expected value. With a library that builds mocks, it is written like this. ```kotlin // Platform code: OrderRepositoryTest class OrderRepositoryTest { @Test fun loadAsksSourceOnce() { val source = mock() whenever(source.fetch("order-1")).thenReturn("ok") OrderRepository(source).load("order-1") verify(source, times(1)).fetch("order-1") } } ``` The piece does two jobs. The `whenever` call builds a stub: whenever `fetch("order-1")` is asked, it returns `"ok"`. The `verify` call is a call verification: the source must have been asked exactly once. Because the repository asks the source twice, this test breaks. The mock the library produces is the platform counterpart of the model's `SpySource` and `requestCount`, but it is not the same thing. The model sets the rule of this measurement; the model is not the tool itself. The break of the test is an overrun: the expected count is one, the actual count is two. There is a difference when comparing the rule with an assertion. An assertion breaks or does not break for a single operation; the rule says in how many of eight operations it broke. The assertion gives an error, the rule gives prevalence. Someone who sees a failure in a test and wants to measure it runs the same test on eight inputs and reads the overrun. #### What the Library's Lines Do Each of the five lines in the platform piece does a separate job, and knowing which line corresponds to which kind of test double is the way to understand what the test checks. `mock()` builds a fake implementation of the interface, generated at run time. In this object no method does real work; every call returns without a reply and is recorded. In the model, the hand-written `SpySource` takes its place, but unlike it, this object carries a log rather than a counter. The line `whenever(source.fetch("order-1")).thenReturn("ok")` builds the stub. The first half states which call is meant, and the second half states which answer will be given to that call. Once the stub is built, it returns the same answer however many times the same call arrives. `OrderRepository(source).load("order-1")` is the class under test itself. The mock is given to the constructor as the dependency; the repository does not know it is fake. Giving the dependency from outside, that is, injecting it through the constructor, is a required condition for mocking. If the repository built the source inside itself, the test would have no place to step in. `verify(source, times(1)).fetch("order-1")` is the verification line. The library looks at the log and checks whether the call `fetch("order-1")` was made exactly once. If another number were written instead of `times(1)`, the expectation would change too. The failure of this line is the direct result of the two calls in the repository. #### The Limit of the Mock Call verification is a powerful tool, and for exactly that reason it needs care. When a test pins in which order and how many times the repository calls the source, it has pinned the implementation, not the behavior. If the repository one day drops the freshness check and lowers the call count to one, the behavior has not changed, but a test that expects two calls breaks. The test team takes this break for a bug and has to repair the test. The reason we set the budget at one in this lesson rests on a change in the opposite direction: a product decision wants one call, and the test pins that decision. The difference lies in who the decision comes from. If a budget rests on a product decision, its overrun is a finding; if it merely copies the current implementation, its overrun is noise. The measurement cannot tell the two apart; the team does. #### Which Dependency Is Doubled With Which Kind The question to ask when choosing the kind of test double is what the test expects from that dependency. If the test needs only an answer, a stub is enough: let the cache return a value and let the repository pass that value on. If the test wants to check whether a call was made, a spy or a mock is needed: was the network reached, how many times. If the dependency exists only to fill the constructor's signature, a dummy is enough and no behavior is written. If the choice is made wrongly, the test either becomes needlessly complicated or falls short. A test that wants to check the call count cannot see the count if it uses a stub; a test that looks only for an answer breaks on the verification of unrelated calls if it builds a mock. In this lesson we used two kinds: the network source was doubled with a spy to count the call, and the cache was doubled with a stub to give an answer. Each dependency was doubled with the kind that fits its role. #### Common Misconceptions The first misconception is the thought that "every dependency should be doubled". Mocking cuts the dependency off from real behavior. If the dependency is fast, deterministic, and free of side effects, such as a pure calculation function, doubling it only lengthens the test and keeps it from checking the real behavior. The second misconception is the thought that "a mock is faster than the real object, so it is always better". A mock is fast, but what it checks is the behavior of the mock. We saw this in the measurement where the stub gave an old answer: the call count was zero, the answer was wrong. The third misconception is the thought that "the call count is the measure of performance". The call count is the number of network trips; not every trip lasts the same. Two calls can exceed a budget and the total duration can still stay short. In this lesson the number is used as a unit of work, not as a duration. #### The Bounding Measurement The bounding measurement here is the cache with a stub. The reasoning is this: the first measurement coming out all showed that the overrun came not from the input but from the design. The way to confirm this is to change the design and measure the same input again. The input, the budget, and the number of samples stay fixed; only the cache changes. That way, the difference in the result cannot be put down to another factor. Now read the same eight operations with a stub. The stub cache gives a ready answer for every key. When the repository finds an answer in the cache, it does not go to the network. ```kotlin // Main.kt — with a stub cache the network calls drop to zero fun main() { val keys = listOf( "order-1", "order-2", "order-3", "order-4", "order-5", "order-6", "order-7", "order-8", ) val stub = CacheStub(keys.associateWith { "cache" }) val calls = callsPerOperation(keys, stub) println("call counts: $calls") val overrun = Overrun("stubbed-repo", Budget("call", 1), calls) overrun.line() countOverruns(listOf(overrun)) } ``` ```text call counts: [0, 0, 0, 0, 0, 0, 0, 0] stubbed-repo: budget=call<=1 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` The same repository, the same eight keys, the same budget. The network calls dropped to zero in all eight, and the measurement moved from all to within. The behavior of the repository did not change here: the repository still goes twice if the cache gives no answer. What changed is that the test never enters that path of the repository. This is the limit of the mocking measurement. The stub decides the answer, that is, the test measures the answer of the stub, not of the real network. Zero calls do not say that the network is fast or that the repository works well; they say only that the stub answered every question. A measurement shows this: put an old value in the stub, knowing that the network gives a new value. ```kotlin // Main.kt — the difference between the stub and the network is what the rule does not see fun main() { val keys = listOf("order-1", "order-2", "order-3", "order-4") val stale = CacheStub(keys.associateWith { "stale" }) val fresh = keys.associateWith { "fresh" } val fromStub = keys.map { OrderRepository(SpySource(fresh), stale).load(it) } val fromNetwork = keys.map { OrderRepository(SpySource(fresh)).load(it) } println("answers from the stub: $fromStub") println("network answers: $fromNetwork") val calls = callsPerOperation(keys, stale) println("call counts: $calls") val overrun = Overrun("stubbed-stale", Budget("call", 1), calls) overrun.line() countOverruns(listOf(overrun)) } ``` ```text answers from the stub: [stale, stale, stale, stale] network answers: [fresh, fresh, fresh, fresh] call counts: [0, 0, 0, 0] stubbed-stale: budget=call<=1 | over=0/4 -> within within: 1 minority: 0 majority: 0 all: 0 ``` The stub returned `stale` in all four operations; the network was saying `fresh` for the same keys. The call count is again zero, the measurement again within. The rule counts the call and does not look at whether the answer is correct. This is not a bug but the purpose of the rule; yet a test written with a stub being green does not mean the real network will give the same answer. What closes that gap is another kind of test: the instrumented tests, closer to the real dependency, that we will see in the next lesson. **TE2.** The model builds the stub as a map; a real mocking library can also have a stub give different answers by call order, which the model does not have. #### Summary - This lesson's overrun: all. The repository went to the network source twice in every operation, and eight of eight operations exceeded the budget of one call. - The lesson's criterion, the number of calls that go to the network source in one operation: the spy counts, the stub decides the answer, the dummy only fills a parameter, the mock verifies the expected call. - With the stubbed cache, the calls in the same eight operations dropped to zero and the measurement moved from all to within; the repository did not change, the path the test enters did. - A stub has its own answer measured: while the network said the new value, the stub returned the old one and the call count was zero again. - Call verification breaks or does not break for one operation; the rule says in how many operations the overrun occurred. #### Next Step In this lesson, the objects that took the place of the network source were hand-written classes, and none of them ran on a device. Instrumented tests turn this around: instead of a test double they use a real screen and a real system, but each test builds that screen from scratch. The next lesson, On-Device Tests, measures in how many tests this setup exceeds the budget. ### On-Device Tests Source: https://academia.sh/en/courses/testing-performance-and-release/on-device-tests An instrumented test runs a real screen on a device or an emulator and pays for each test's setup again. The lesson measures in how many tests the setup work exceeds the budget and what sharing the setup costs. In the previous lesson, test doubles took the place of the network, and no piece of the test ran on a device. The stub had its own answer measured; it did not measure how a real screen talks to a real system. Only a test that actually runs the screen can say that. This lesson's question is the course's question: **in how many samples does this measurement exceed its budget?** The four answers are the same: within, minority, majority, all. The unit of work measured is the work a test does before it starts and while it runs. The core is the same as in the previous lessons: `Budget` carries a limit, and `Overrun` carries the samples of a measurement and how many of them strictly exceed the limit. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### What an Instrumented Test Runs An instrumented test loads the app and the test code onto an Android device and runs them together with the system there. A unit test runs on the development machine, in the pure Kotlin layer; an instrumented test uses a real screen, a real lifecycle, and the system's own services. This catches the defects a unit test cannot: a screen losing its state when it is rotated, a permission dialog failing to open, a database not actually being created on the system. The place where tests run is of two kinds. A physical device is the hardware that will end up in a user's hands; sensors, the camera, and manufacturer-specific system behavior are verified only there. An emulator is a software imitation of a device; it is easy to set up and to wipe, but it only imitates hardware behavior. In both, a test runner loads the app, runs the tests in order, and collects the result. This way of running has a cost: each test has to bring up the screen it checks on its own. The test starts a screen, loads the necessary data, puts the user in a signed-in state, and only then moves on to its actual assertion. This preparation is called the test fixture, and the work the preparation does we call `fixtureWork`. On the platform, a test that starts a screen from scratch is written like this. ```kotlin // Platform code: CartScreenTest @RunWith(AndroidJUnit4::class) class CartScreenTest { @Test fun showsEmptyCart() { ActivityScenario.launch(CartActivity::class.java).use { scenario -> scenario.onActivity { activity -> assertEquals(0, activity.itemCount()) } } } } ``` The `launch` call restarts the screen in every test; the `use` block closes it when the test ends. The model sets the rule of this measurement; the model is not the tool itself. In the model, the start-up work is counted; no screen is actually started. **TE1.** The model does not run an instrumented test; there is no device or emulator in this environment. The setup work is a sum of steps: starting the screen is 6, loading the seed data is 4, signing in is 2. These numbers are inputs of the model, not a measurement of a device. #### Setup Paid in Every Test The model builds two functions. The first starts the screen from scratch in every test: the work of a test is the sum of the setup work and the work of the test's own body. The second does the setup once per test class: only the first test pays for the setup, and the later ones find the screen ready. ```kotlin // Fixture.kt — a test fixture that starts the screen from scratch: setup work is counted in steps class Fixture(val steps: List) { val fixtureWork: Int = steps.sum() } fun freshPerTest(fixture: Fixture, bodies: List): List = bodies.map { fixture.fixtureWork + it } fun sharedPerClass(fixture: Fixture, bodies: List): List = bodies.mapIndexed { index, body -> (if (index == 0) fixture.fixtureWork else 0) + body } ``` Both functions produce a list of work from a list of test bodies; these lists are the samples that will be given to `Overrun`. Think of eight tests whose bodies have work of 1, 2, 1, 3, 2, 1, 2, and 3. The budget limits the work a test may do to 10 units. This number is not a platform limit; it is the model's counterpart of the target a team sets so that a test stays short. ```kotlin // Main.kt fun main() { val fixture = Fixture(listOf(6, 4, 2)) val bodies = listOf(1, 2, 1, 3, 2, 1, 2, 3) println("fixture work: ${fixture.fixtureWork}") val work = freshPerTest(fixture, bodies) println("work per test: $work") val overrun = Overrun("setup-per-test", Budget("work", 10), work) overrun.line() countOverruns(listOf(overrun)) } ``` ```text fixture work: 12 work per test: [13, 14, 13, 15, 14, 13, 14, 15] setup-per-test: budget=work<=10 | over=8/8 -> all within: 0 minority: 0 majority: 0 all: 1 ``` The setup work alone is 12 units and is above the budget of 10 units. However small the bodies are, eight of eight tests went above the budget: the measurement is all. Here all gives a hint: the cause of the overrun is not the body of any test. The smallest body is 1 unit and it still reaches 13. The cause is the setup, and it is paid again in every test. Loosening the budget would close the overrun: a budget of 16 units would leave all eight tests within. But that does not change the fact that the setup is expensive; it only pushes it out of the measurement. The rule does not see the magnitude of the overrun, and it does not see why the setup is expensive. The prose has to say that. #### The Parts of the Setup The 12-unit setup in the model consists of three parts, and each part corresponds to a separate job in a real test. Starting the screen is asking the system for a screen, bringing its lifecycle to its first states, and drawing it; for this reason it takes 6 units, the most expensive of the three. Loading the seed data is writing the records the screen will show into the database, and it is 4 units. Signing in brings the user to the state the screen expects, and it is 2 units. None of these three parts belongs to the test's actual assertion. If the test says "when the cart is empty, the empty message is visible", the assertion is a single line. The previous three jobs only make the assertion possible. This is why the setup work is counted in a separate column from the test body. There are two ways to shrink the setup. The first is to make the parts cheaper: loading the seed data with fewer records, or giving the sign-in through a fake authenticator. The second is to share the parts; doing it once per class is one form of this. The first gives up the realism of the test, the second the independence of the tests. #### Reading the Platform Pieces Line by Line In the `CartScreenTest` piece, the line `@RunWith(AndroidJUnit4::class)` declares that the test will run with an instrumented runner. `ActivityScenario.launch(CartActivity::class.java)` asks the system to actually start the screen, that is, the activity, and returns a scenario object. The `.use { scenario -> ... }` block closes the scenario when the block ends, whatever happens; a screen that is not closed affects the next test. The `scenario.onActivity { activity -> ... }` block gives access to the activity on the screen's own thread. `assertEquals(0, activity.itemCount())` is the assertion itself: there are zero items in the cart. In the `SeedOnce` piece that we will see in the bounding measurement, the `companion object` holds members tied to the class in Kotlin. The `@BeforeClass` annotation asks for the method to run once before the first test in the class. `@JvmStatic` makes the method be generated as a real static method in the form the runner expects. `seedDatabase()` is the expensive part of the setup, and in the model it corresponds to the 4-unit seed load. #### Physical Device or Emulator The side that sets the budget also has to choose where the tests run. An emulator is quick to set up and can be reset with a single command; in a CI pipeline this means every run starts from a clean state. A physical device catches defects tied to hardware, but it does not guarantee a clean state in every run. A permission or an account left by a previous test can remain on the device. The setup work measured in this lesson is independent of this choice. In both, the screen is started, data is loaded, and a sign-in is made. What changes is the cost of the work per unit, and the model counts the cost in steps, not in duration. #### The Test Pyramid and the Budget The expense of instrumented tests determines the shape of the test suite. Unit tests are fast and run in large numbers; instrumented tests are slow and run in small numbers. This distribution is called the test pyramid: at the base are many unit tests, in the middle are tests that bring components together, and at the top are a few end-to-end instrumented tests. A suite that turns the pyramid upside down, that is, that checks everything on a device, multiplies the setup cost we measured in this lesson by thousands of tests. A team sets the budget separately for each test. The budget of a unit test can be a few steps; the budget of an instrumented test is set according to the setup. The same 10 units is a very different expectation in two different kinds of test. This is why, throughout the course, the budget of every measurement is written together with a reason. #### Conditions of a Shared Setup Sharing the setup per class depends on some conditions. The order of the tests must not affect the result; if a test benefits from the state another test left, it breaks when the order changes or when it runs alone. Tests that change the shared state must restore it when they finish. On the platform, this is done with a cleanup method that runs after every test. Finally, the shared setup must be read-only: the seed data is loaded once and no test changes it. If one of these three conditions cannot be met, sharing the setup raises speed but lowers stability. The cart measurement showed this balance: in the shared cart, seven tests found something at the start. Cleanup also has a cost, and the cleanup done after every test takes back part of the gain the setup created. #### Common Misconceptions The first misconception is the thought that "an instrumented test replaces a unit test". An instrumented test uses the real system and is therefore more expensive. Checking the edge cases of a function on a device means paying for the setup again for every edge. Edge cases are checked in unit tests, screen flows on a device. The second misconception is the thought that "sharing the setup is always good". A shared setup does not reduce the load of the first test; it only makes it be paid once. But shared state ties the tests to each other. Seven tests finding something at the start in the cart measurement was a concrete sign of this. The third misconception is the thought that "closing an overrun by enlarging the budget is enough". Raising the budget to 16 would move the measurement from all to within, but the cost of the setup would not change at all. A budget rests on a product decision; a budget changed to close an overrun destroys the meaning of the measurement. #### The Bounding Measurement The bounding measurement here is moving to the setup per class. The reasoning is this: in the first measurement the cause of the overrun was the setup, and the only thing to do to confirm it is to change where the setup is paid. The bodies, the budget, and the number of samples stay the same. If the overrun moves from all to minority, the cause really was the setup. Now do the setup once per test class. The bodies of the tests, the budget, and the setup stay the same; only when the setup is paid changes. On the platform, this is done with a preparation method that runs once per class. ```kotlin // Platform code: SeedOnce class CartScreenTest { companion object { @BeforeClass @JvmStatic fun seedOnce() { seedDatabase() } } } ``` The preparation method runs once before the first test in the class. In the model, this is the setup work being paid only in the first test. ```kotlin // Main.kt — the setup is paid once per class fun main() { val fixture = Fixture(listOf(6, 4, 2)) val bodies = listOf(1, 2, 1, 3, 2, 1, 2, 3) val work = sharedPerClass(fixture, bodies) println("work per test: $work") val overrun = Overrun("setup-per-class", Budget("work", 10), work) overrun.line() countOverruns(listOf(overrun)) } ``` ```text work per test: [13, 2, 1, 3, 2, 1, 2, 3] setup-per-class: budget=work<=10 | over=1/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The same eight tests, the same budget. Seven tests stayed between 1 and 3 units; only the first test rose to 13. The measurement moved from all to minority. The overrun did not close entirely: the setup did not disappear, it was only piled onto a single test. The rule reads this as "one test exceeds"; that the first test carries the load of the setup has to be said again by the prose. This gain is not free. When the setup is shared, the tests share the same screen, the same data, and the same system state. If one test puts an item in the cart, the next test finds the cart full. A measurement shows this: each test adds a certain number of items to the cart, and we count how many items are left in the cart when the next test starts. The budget is zero, because every test must start with an empty cart. ```kotlin // Main.kt — shared state spoils the start of the next test fun leftoverBefore(adds: List, cleanup: Boolean): List { val cart = mutableListOf() return adds.map { count -> val leftover = cart.size repeat(count) { cart.add("item") } if (cleanup) cart.clear() leftover } } fun main() { val adds = listOf(1, 0, 2, 1, 0, 1, 1, 0) val shared = leftoverBefore(adds, false) val cleaned = leftoverBefore(adds, true) println("no cleanup: $shared") println("with cleanup: $cleaned") val clean = Budget("leftover", 0) val overruns = listOf( Overrun("shared-cart", clean, shared), Overrun("cleaned-cart", clean, cleaned), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text no cleanup: [0, 1, 1, 3, 4, 4, 5, 6] with cleanup: [0, 0, 0, 0, 0, 0, 0, 0] shared-cart: budget=leftover<=0 | over=7/8 -> majority cleaned-cart: budget=leftover<=0 | over=0/8 -> within within: 1 minority: 0 majority: 1 all: 0 ``` The shared cart is majority: apart from the first test, seven tests find something at the start. If cleanup is done after every test, the measurement is within, but cleanup is work too. Sharing the setup makes tests faster; cleaning the shared state takes back part of this gain. Which method to choose depends on whether the team puts the independence of tests or their duration first. The rule does not make this decision; it only counts the result of the decision. **TE2.** The model builds the cart as a list and does the cleanup in a single step. In a real database, cleanup is an operation that varies with the number of tables and foreign keys; the model does not have this difference. #### Summary - This lesson's overrun: all. Because every test started the screen from scratch, the 12-unit setup made eight of eight tests exceed the budget of 10 units. - The lesson's criterion, in how many tests the work of a test exceeds the budget: with setup in every test it came out all, with setup per class it came out minority. - The cause of the overrun was not the test body; although the smallest body was 1 unit, the test rose to 13. The cause was the setup, and it was paid again in every test. - Sharing the setup piled the load onto the first test, and the overrun moved from all to minority, but the setup did not disappear. - Shared state spoiled the start of later tests: with no cleanup the measurement came out majority, with cleanup after every test it came out within. #### Next Step In this lesson the screen was started from scratch, and the question of which part of the screen the test would check was never asked. A real UI test has to find a node on the screen, and there is a way to find it. The next lesson, UI Tests, measures in how many samples looking a node up by its text and looking it up by its test tag exceed the budget. ### UI Tests Source: https://academia.sh/en/courses/testing-performance-and-release/ui-tests A UI test finds a node on a screen, interacts with it, and verifies the result. The lesson measures that looking a node up by its text walks the semantics tree from the start, while looking it up by its test tag finishes in one step. In the previous lesson every test started the screen from scratch, and the measurement counted the cost of the setup. Once the screen is up, the test has to talk to it: it finds a node, taps it, and reads the result. This conversation has a cost too, and it is the subject of this lesson. The course's question is the same: **in how many samples does this measurement exceed its budget?** The four answers do not change: within, minority, majority, all. The unit of work measured is the step taken to find one node. The core is the same as in the previous lessons: a budget, the label of a measurement and its list of samples, and the number of samples that strictly exceed the limit. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### How the Screen Looks to a Test A UI test reads the meaning of the screen, not its pixels. In a component-based UI, this meaningful form of the screen is called the semantics tree. Each node in the tree carries a text, a role, or an action: a heading, a button, the name of a product. Accessibility services build on the same tree, although they apply their own rules for merging nodes. The job of the test is to find a node in this tree, apply an action to it, and verify the node's later state. There are two ways to find a node. The first is to look it up by its text: you find the node by the words the user sees. This is a natural way, because the test looks at what the user looks at. The second is to look it up by its test tag: the developer gives the node a tag, and the test asks for that tag. The tag is invisible to the user; it serves only the test. On the platform the two look like this. ```kotlin // Platform code: CartComposeTest class CartComposeTest { @get:Rule val composeRule = createComposeRule() @Test fun deleteButtonIsShown() { composeRule.setContent { CartScreen() } composeRule.onNodeWithTag("delete-notebook").assertIsDisplayed() composeRule.onNodeWithText("Delete notebook").assertIsDisplayed() } } ``` `setContent` sets up the screen, `onNodeWithTag` finds the node by tag, and `onNodeWithText` finds it by text. In the classic view system the same job is done with a different tool. ```kotlin // Platform code: CartViewTest class CartViewTest { @Test fun deleteButtonIsShownInViews() { onView(withId(R.id.deleteNotebook)).check(matches(isDisplayed())) } } ``` `onView` finds a view in the view tree by its id or its text. Both pieces belong to platform code. The model builds the rule of this measurement; the model is not the tool itself. In the model, the tree is a list of nodes, and a lookup is counted by the steps taken in this list. **TE1.** The model does not run the UI testing library; there is no screen or tree in this environment. The semantics tree is represented by a flat list, and a lookup walks the list from the start. The real tool walks the tree in depth order; the model reduces this order to the order of the list. #### Looking a Node Up by Its Text The tree of a shopping cart screen has twelve nodes: the heading, the back button, two products with a quantity and a delete button for each, the total, the coupon code, the apply-coupon button, and the proceed-to-checkout button. Each node has a text and a unique tag. The helper file builds the tree and defines the lookup by text and the lookup by tag. ```kotlin // Semantics.kt — the semantics tree of the cart screen, lookup by text and by tag class Node(val text: String, val tag: String) fun cartTree(extra: Int = 0): List { val head = listOf( Node("My cart", "title"), Node("Back", "back"), Node("Notebook", "item-notebook"), Node("Notebook quantity", "quantity-notebook"), Node("Delete notebook", "delete-notebook"), Node("Pencil", "item-pencil"), Node("Pencil quantity", "quantity-pencil"), Node("Delete pencil", "delete-pencil"), ) val offers = (1..extra).map { Node("Offer $it", "offer-$it") } val tail = listOf( Node("Total", "total"), Node("Coupon code", "coupon"), Node("Apply coupon", "apply-coupon"), Node("Proceed to checkout", "checkout"), ) return head + offers + tail } fun pickByTags(tree: List, tags: List): List = tags.map { tag -> tree.first { it.tag == tag } } fun findSteps(tree: List, text: String): Int { var steps = 0 for (node in tree) { steps++ if (node.text == text) return steps } return steps } fun tagSteps(byTag: Map, tag: String): Int = if (byTag.containsKey(tag)) 1 else 0 ``` `findSteps` walks the tree from the start and stops when the text matches. The steps taken equal the node's position in the tree: the heading is found in 1 step, the proceed-to-checkout button in 12. `cartTree` can optionally add offer nodes to the tree; for now we add none. Eight tests will look up eight nodes: back, delete notebook, pencil, pencil quantity, total, apply coupon, proceed to checkout, and the heading. The budget is 5 steps: this is the model's counterpart of a team decision that wants a node to be found within five nodes from the top of the screen, and it is not a platform threshold. ```kotlin // Main.kt fun main() { val tree = cartTree() val picks = pickByTags( tree, listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title"), ) val steps = picks.map { findSteps(tree, it.text) } println("steps: $steps") val overrun = Overrun("lookup-by-text", Budget("step", 5), steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text steps: [2, 5, 6, 7, 9, 11, 12, 1] lookup-by-text: budget=step<=5 | over=5/8 -> majority within: 0 minority: 0 majority: 1 all: 0 ``` Five of the eight lookups exceeded the budget: the pencil, pencil quantity, total, apply-coupon, and proceed-to-checkout nodes sit beyond the first five nodes of the screen. The measurement is majority. The heading, the back button, and the delete-notebook button stayed within the budget because they are at the start of the tree. The overrun comes from the position of the node, not from its text: if a node with the same text were at the start of the tree, the lookup would be short. The rule misses two things in this measurement as well. It does not see the magnitude: the proceed-to-checkout node exceeds the budget by seven steps and the pencil by one, and both count as one. It does not see the cause: the overrun comes from the walking order of the tree and from how the nodes are arranged for the user. #### The Nodes of the Semantics Tree Each node in the semantics tree is the summary of an element on the screen, as it appears to tests and to accessibility services. A node can carry a text, declare a role, offer an action, and receive a test tag. Several elements nested on the screen, for example the heading and the description inside a card, can by default merge into a single node in a component-based UI. The test looks up this merged node; the tree before merging can be requested separately. This detail changes how many steps a lookup takes, because the merged tree carries fewer nodes. In our model there are twelve nodes, and each is a separate element. On a real screen, the number of nodes can be far smaller or far larger depending on the structure of the screen. What the measurement teaches is not the number but the relationship: if the lookup depends on the position of the node, the overrun grows as the screen grows. #### Reading the Platform Pieces Line by Line In the `CartComposeTest` piece, the `@get:Rule` annotation makes the declarative UI's test rule be set up before every test and shut down afterward. `createComposeRule()` produces this rule. The line `composeRule.setContent { CartScreen() }` loads the screen under test into the rule; the screen is set up inside an empty activity, so it is not necessary to start the app's own screen as in the previous lesson. `onNodeWithTag("delete-notebook")` finds the node whose tag is `delete-notebook`. `assertIsDisplayed()` verifies that the node is visible on the screen. `onNodeWithText("Delete notebook")` finds the same node by its text. In the `CartViewTest` piece, `onView(withId(R.id.deleteNotebook))` finds the view with the given id in the classic view system. `check(matches(isDisplayed()))` verifies that the view is visible. An id is a label that stands in for the text: it is defined once in a resource file and is independent of translation. If we want to add an interaction, an action such as `performClick()` is added to the same lookup chain. The first link of the chain is always the node lookup; the action and the verification work on the node that was found. For this reason the cost of the lookup is added to the cost of every interaction in the test. #### The Fragility of Lookup by Text Lookup by text is not only slow; it breaks when the text changes. If the label of the button turns from "Delete notebook" into "Delete", every test that looks it up by text breaks. If the app is published in several languages, the test passes in only one language and cannot find the node in the others. A tag has neither problem, because a tag is independent of the user's language. On the other hand, the text is what the user really sees. A test that looks up by text implicitly checks that the button appears to the user with the right words. A test that looks up by tag does not check this, and the difference is a choice: the team decides whether it wants the fast lookup or the correctness of the visible text. #### Naming the Tags The value of a tag lies in its staying stable. The model chose the form `action-element` for the tags: `delete-notebook`, `quantity-pencil`, `apply-coupon`. This name is tied to the function of the element, not to the layout of the screen. Even if the button is moved to another place on the screen, the tag stays the same. A name that describes the layout, for example `second-row-right-button`, becomes misleading when the screen changes. A tag also has to be unique. If every row in a list carries the same tag, a lookup that expects a single node finds multiple matches and the test fails with an error. For this reason, in lists a value that tells the rows apart, for example the product code, is added to the tag. #### The Relationship with Accessibility The semantics tree is a structure used not only by tests but also by screen readers. For this reason a test that looks up by text searches for a text close to the one the screen reader will read; a test that cannot find a node whose text is wrong also catches an accessibility problem indirectly. A tag, on the other hand, serves only tests and may be invisible to the screen reader. This is the reason for using the two lookups together: the tag finds quickly and stably, and the text verifies what the user hears. #### Common Misconceptions The first misconception is the thought that "putting a tag on every node makes the test more robust". Tags add a test-only burden to production code, and a forgotten tag breaks the test. A tag is put only on nodes whose text cannot be trusted: repeated texts, dynamic texts, and icons. The second misconception is the thought that "lookup by text is always slow". In the measurement the slowness came from the order of the node. If the node being sought is at the start of the tree, lookup by text is short as well: the heading was found in one step and the back button in two. The overrun depends on the position, not on the form of the lookup. The third misconception is the thought that "when the screen grows, the test has to be rewritten". In tests that look up by tag, growth did not change the cost of the lookup. What has to be rewritten on a growing screen is the lookup of the nodes that remain without a tag. #### The Bounding Measurement The bounding measurement here is the lookup by tag. The reasoning is this: in the first measurement the overrun came from the lookup depending on the position in the tree. A lookup that finds the node from a map removes the position. If the nodes, the budget, and the number of samples stay the same, only the form of the lookup explains the difference in the result. Look up the same screen, this time by test tag. The tag is unique, so it is found in a map with a single glance. The map itself has to be built once: the tree is walked once and each node is put into the map under its tag. This work is paid once per screen. ```kotlin // Main.kt — the same screen, lookup by tag fun main() { val tree = cartTree() val picks = pickByTags( tree, listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title"), ) val byTag = tree.associateBy { it.tag } println("map setup: ${tree.size} steps, once per screen") val steps = picks.map { tagSteps(byTag, it.tag) } println("steps: $steps") val overrun = Overrun("lookup-by-tag", Budget("step", 5), steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text map setup: 12 steps, once per screen steps: [1, 1, 1, 1, 1, 1, 1, 1] lookup-by-tag: budget=step<=5 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` The same screen, the same eight nodes, the same budget. Every lookup finished in one step, and the measurement moved from majority to within. What changed is not the screen under test but the way the node is reached. Lookup by tag gives up one thing as well: the tag is invisible to the user. If the label on a button is wrong, a test that looks up by text catches the defect and a test that looks up by tag does not. On the platform, the two ways are used together: the node is found by tag, and the text is verified separately. A third measurement shows what happens as the screen grows. Add eight offer nodes before the total. Set the budget this time to the old tree's length, 12 steps: on the old screen, lookup by text finds every node within this limit. ```kotlin // Main.kt — when the screen grows, lookup by text gets more expensive and lookup by tag stays the same fun main() { val tags = listOf("back", "delete-notebook", "item-pencil", "quantity-pencil", "total", "apply-coupon", "checkout", "title") val small = cartTree() val large = cartTree(8) val smallPicks = pickByTags(small, tags) val largePicks = pickByTags(large, tags) val byTag = large.associateBy { it.tag } val smallText = smallPicks.map { findSteps(small, it.text) } val largeText = largePicks.map { findSteps(large, it.text) } val largeTag = largePicks.map { tagSteps(byTag, it.tag) } println("old screen, text: $smallText") println("large screen, text: $largeText") println("large screen, tag: $largeTag") val limit = Budget("step", 12) val overruns = listOf( Overrun("text-old-screen", limit, smallText), Overrun("text-large-screen", limit, largeText), Overrun("tag-large-screen", limit, largeTag), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text old screen, text: [2, 5, 6, 7, 9, 11, 12, 1] large screen, text: [2, 5, 6, 7, 17, 19, 20, 1] large screen, tag: [1, 1, 1, 1, 1, 1, 1, 1] text-old-screen: budget=step<=12 | over=0/8 -> within text-large-screen: budget=step<=12 | over=3/8 -> minority tag-large-screen: budget=step<=12 | over=0/8 -> within within: 2 minority: 1 majority: 0 all: 0 ``` On the old screen, lookup by text was within the budget of 12 steps. When the screen grew by eight nodes, the total, apply-coupon, and proceed-to-checkout nodes moved beyond the limit, and the measurement fell to minority. Lookup by tag is still one step on the large screen. As the screen grows, the overrun of lookup by text increases; the overrun of lookup by tag is independent of position. **TE2.** The model places the offer nodes in front of the total; on a real screen, the place of new nodes in the tree depends on the screen design. It is also assumed that the map covers every node: if two nodes carry the same tag, the real tool fails with an error, and the model does not. #### Summary - This lesson's overrun: majority. The test that looked up nodes by text went past the first five steps of the tree in five of the eight lookups. - The lesson's criterion, in how many lookups the steps taken to find a node exceed the budget: the overrun comes not from the text of the node but from its position in the tree. - Lookup through a test tag and a map is one step in every sample; on the same screen the measurement moved from majority to within. The map is built once per screen. - When the screen grew by eight nodes, lookup by text went from within to minority on the budget of 12 steps, and lookup by tag stayed within. - The rule counts steps and does not look at whether a node appears correctly to the user; a test that looks up by tag does not catch a wrongly written button text. #### Next Step In this lesson the screen stood still, and the lookup only counted steps. A real screen, however, may not be ready when the test starts looking for a node. The next lesson, Test Stability, measures in how many runs a test that waits for the ready signal exceeds its waiting budget. ### Test Stability Source: https://academia.sh/en/courses/testing-performance-and-release/test-stability A test that sometimes passes and sometimes fails on the same code is flaky. The lesson measures in how many runs a test that waits a fixed number of rounds exceeds its budget against the ready round, and how polling on a condition closes the overrun. In the previous lesson the test could find the node on the screen. But if it goes looking before the screen is ready, the same test passes in one run and fails in another. The unit of work measured in this lesson is a waiting round: a round the test spends while waiting for the screen to be ready. The course's question does not change: **in how many samples does this measurement exceed its budget?** The answers are again within, minority, majority, and all. The core is the same as in the previous lessons. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### The Anatomy of a Flaky Test A flaky test is a test that sometimes passes and sometimes fails although the code has not changed. The cause is mostly not in the test but in the things the test puts in order: the drawing of the screen, a network response, the end of a coroutine. The test does not know when these will happen, and it makes a guess: "it will be ready after two seconds". The guess holds in most runs and does not hold in one. This problem gets mixed up with three separate questions. In M30/K05 we asked whether a result changes with repetition or with order. In M13/K04 we asked what determines a result. Here neither is present: we do not name flakiness as an overrun category, and we do not search for its source. The measurement only counts how many runs the overrun spreads across. Flakiness is not a category; it is the mechanism that produces the overrun, that is, the way of waiting. The tool used on the platform against this problem is the idling resource, which provides synchronization. The resource reports to the test whether the app is busy, and the test postpones its action until the resource goes idle. ```kotlin // Platform code: BusyCounterResource class BusyCounterResource : IdlingResource { private val busy = AtomicInteger(0) private var callback: IdlingResource.ResourceCallback? = null override fun getName(): String = "busy-counter" override fun isIdleNow(): Boolean = busy.get() == 0 override fun registerIdleTransitionCallback(callback: IdlingResource.ResourceCallback) { this.callback = callback } fun begin() { busy.incrementAndGet() } fun end() { if (busy.decrementAndGet() == 0) callback?.onTransitionToIdle() } } ``` `begin` is called when a job starts, and `end` when a job finishes. If `isIdleNow` returns true, the test moves on. The test does not guess a duration; it asks the app itself. The model does not run these tools; instead it reads the round in which the screen became ready from a list and counts the rounds spent according to how the test does its waiting. **TE1.** In the model, the ready round is a fixed list; in a real run this value depends on the state of the network, the processor, and the queue at that moment. The list represents the ready rounds of eight runs and is not generated randomly. #### A Test That Waits a Fixed Number of Rounds The helper file defines the ready rounds and the two forms of waiting. ```kotlin // Stability.kt — ready rounds and two forms of waiting val readyTable: List = listOf(3, 4, 3, 8, 4, 3, 9, 4) fun pollRounds(run: Int): Int = readyTable[run] fun fixedWaitRounds(run: Int, wait: Int): Int = wait fun passesAfterFixedWait(run: Int, wait: Int): Boolean = pollRounds(run) <= wait ``` `pollRounds(run)` gives the round in which the screen became ready in that run. `fixedWaitRounds` is the test's fixed guess: it waits `wait` rounds whatever happens. `passesAfterFixedWait` says whether the test passes: it passes if the screen is ready before `wait` rounds have gone by. The test assumes the screen will be ready in five rounds. The ready rounds of eight runs are measured; the budget is the 5 rounds the test waits. ```kotlin // Main.kt fun main() { val runs = (0 until 8).toList() val ready = runs.map { pollRounds(it) } println("ready rounds: $ready") val failed = runs.count { !passesAfterFixedWait(it, 5) } println("wait 5 rounds, failed runs: $failed") val overrun = Overrun("fixed-wait", Budget("round", 5), ready) overrun.line() countOverruns(listOf(overrun)) } ``` ```text ready rounds: [3, 4, 3, 8, 4, 3, 9, 4] wait 5 rounds, failed runs: 2 fixed-wait: budget=round<=5 | over=2/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` In two of the eight runs the screen became ready after more than five rounds: the fourth run in 8 rounds, the seventh run in 9. The test fails in these two runs and passes in the other six. The measurement is minority. A test that passes in six of eight is not considered reliable: the two failing runs are noise for the team. The rule knows the number of overruns, but it cannot tell whether that noise is a regression or the ready round. #### Lengthening the Wait The first remedy that comes to mind is to lengthen the wait. If the test waits nine rounds, the screen is ready in every run and the test never fails. But this has a price: when the screen is ready in three rounds, the test still waits nine. Set the budget this time to the number of rounds a run may spend, that is, 5. ```kotlin // Main.kt — raising the wait to nine fun main() { val runs = (0 until 8).toList() val failed = runs.count { !passesAfterFixedWait(it, 9) } println("wait 9 rounds, failed runs: $failed") val spent = runs.map { fixedWaitRounds(it, 9) } println("rounds spent: $spent") val overrun = Overrun("long-wait", Budget("round", 5), spent) overrun.line() countOverruns(listOf(overrun)) } ``` ```text wait 9 rounds, failed runs: 0 rounds spent: [9, 9, 9, 9, 9, 9, 9, 9] long-wait: budget=round<=5 | over=8/8 -> all within: 0 minority: 0 majority: 0 all: 1 ``` The test became stable, but it exceeded the budget in every run: all. The flakiness is gone, and a constant cost took its place. A fixed wait makes you choose one of two kinds of failure: either a test that fails in some runs or a test that is slow in every run. **TE2.** The waiting round is a common unit in the model. In a real test, a round is not a single fixed duration but a waiting step of variable length; the model reduces it to a unit of work. #### The Sources of Flakiness For a test to be flaky the code does not have to be wrong; it is enough that the timing relationship between the test and the app is set up wrongly. Flakiness caused by timing has three typical scenes. The first is the UI thread. The app draws the interface on its own thread, while the test runs on another thread and may try to read the result right after tapping a button. If the drawing has not finished yet, the read sees the old state. The second is background work. A response from the network, a database write, or the end of a coroutine is not under the test's control. Because the test does not know when it will finish, it makes a guess, and the guess may not hold. The third is animation and transitions. While a screen opens, nodes stay in temporary states until they find their places. If the test taps a node in the middle of a transition, the node may not be tappable yet. The common point of these three scenes is that the test assumes an event has finished. The right remedy is to ask whether the event has finished, not to assume it. In the model, the round in which the screen became ready is the counterpart of this event, and the test either guesses it or asks for it. #### Synchronization Options There are four ways to synchronize a test with the app, and each carries a different cost. A fixed wait is the simplest and was used in the first two measurements of this lesson. The test stops for a certain duration or number of rounds. Its problem is this: if the wait is short the test breaks, and if it is long the test waits for nothing in every run. Polling is the test checking a condition at short intervals. When the condition is met the test moves on; if an upper limit is reached the test fails. This is the wait on a condition that we use in the last measurement. The cost of polling is that every check does its own work. An idling resource is the app itself saying "I am busy" or "I am idle". It differs from polling in that the app, not the test, knows the condition. When the app starts a job it increments the counter, when it finishes it decrements it; the test waits until the counter returns to zero. The resource is the class shown in the platform piece. For coroutines, a test scheduler uses a virtual clock and advances waits without any real time passing. Because the test can control time, this virtual clock is the most stable option, but it can be used only in work limited to coroutines and timers; the network and drawing do not know this clock. In this lesson the virtual clock is not measured, because the library's test package is not installed; it is only named. #### Reading the Idling Resource Line by Line `BusyCounterResource` implements the resource interface that the test framework recognizes. `busy` is an `AtomicInteger`: the counter can be safely incremented and decremented from several threads. `getName()` gives the name of the resource as it will appear in error messages. `isIdleNow()` is the question the test framework asks: if the counter is zero, the app is idle. `registerIdleTransitionCallback` lets the framework register a callback; it is stored to notify the framework when the resource goes idle. `begin()` is called when the app starts a job and increments the counter. `end()` decrements the counter when the job finishes; if the counter drops to zero, `onTransitionToIdle()` is called and the test framework stops waiting. The most common mistake is to make the `begin` call and forget the `end` call on an error path. Then the counter never returns to zero and the test waits forever. The remedy is to put the `end` call in a `finally` block. #### How the Budget Is Set The waiting budget answers two different questions, and mixing them up is a common mistake. The first question is "how many rounds is the test willing to wait at most". The second question is "in how many rounds at most does the app become ready". In a fixed wait, these two questions are answered with a single number: the test guesses in how many rounds at most the app will be ready and waits that long. If the guess is based on observed values it looks sufficient, but observed values do not cover new runs. In a wait on a condition, the two are separated. The test moves on when ready; the upper limit comes into play only when the app is never ready. In this lesson we set the upper limit to the worst observed round: 9. Eight runs represent a short period, so a real team sets the upper limit above the worst observed round, with a safety margin. This margin is a product decision. #### Measuring Stability Before repairing a flaky test, you need to know how flaky it is. Running the test in a single run is not enough; you have to run the same test many times by repetition and count how many times it failed. In this lesson, eight runs and the list of ready rounds are the model of this repetition. Failing in two of eight runs is a sign of flakiness for the team. The measurement also shows the effect of the repair. After the repair, the same number of runs is repeated, and if the number of failed runs is zero, the repair has held. But zero failed runs does not mean the test is stable; it only means it did not break in these runs. The evidence gets stronger as the number of runs increases. For this reason, the comparison before and after the repair uses the same number of runs. #### What to Do with a Flaky Test When a flaky test is found there are three ways. The first is to quarantine the test: the result is reported but does not stop the pipeline. Quarantine is a temporary measure and is put in place together with a deadline. The second is to find the root cause and repair it: tying the timing to a condition, cleaning the shared state, faking the dependency. The third is to delete the test; if a test gives a result that nobody trusts, it is more honest for it to give no result at all. If quarantine becomes a permanent solution, the test suite slowly fills with noise and the team gets used to red results; a red result that people are used to also hides a real defect. The team can track the number of tests in quarantine with a budget as well; this course's measure asks exactly this. #### Common Misconceptions The first misconception is the thought that "retrying a flaky test solves the problem". A retry makes the test pass, but it hides the overrun: a test that tries twice in one run pays the cost of two runs and erases the flakiness from the reports. What this lesson measures is exactly this hidden overrun. The second misconception is the thought that "the test is responsible for a flaky test". App code that leaves the timing undefined can also be the cause of flakiness. If the app does not report that it is ready, the test has to guess however well it is written. The idling resource ties these two sides together. The third misconception is the thought that "setting the wait duration once and forgetting it is enough". A nine-round wait may pass all the runs today; tomorrow the hardware or the app changes and a run appears that is ready in the tenth round. Fixed values age; a wait on a condition does not. #### The Bounding Measurement The bounding measurement here is polling on a condition. The reasoning is this: in the fixed wait there were two bad options, and both came from the guess. Replacing the guess with a condition makes it possible to measure the same runs again without changing the ready rounds or the budget at all. The budget is set this time to the worst observed round, because in a wait on a condition the goal is not to guess the overrun but to stay within the worst observed case. The third remedy is to tie the wait to a condition, not to a guess. In every round the test checks whether the screen is ready and moves on when it is; the idling resource does this on the platform. In this case the rounds spent equal the ready round of that run. The budget is also set not by failure but by the worst observed ready round: 9 rounds. ```kotlin // Main.kt — polling on a condition fun main() { val runs = (0 until 8).toList() val spent = runs.map { pollRounds(it) } println("rounds spent: $spent") val failed = runs.count { !passesAfterFixedWait(it, pollRounds(it)) } println("failed runs: $failed") val overrun = Overrun("condition-polling", Budget("round", 9), spent) overrun.line() countOverruns(listOf(overrun)) } ``` ```text rounds spent: [3, 4, 3, 8, 4, 3, 9, 4] failed runs: 0 condition-polling: budget=round<=9 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` Every run finished in its own ready round, no run failed, and the measurement stayed within the budget. This remedy did not pay the price that the previous two remedies paid: fast runs waited briefly, and slow runs waited for the slow screen. The limit of this result is in the budget. The 9-round budget comes from the worst observed round. If a run that is ready in the tenth round appears, the measurement has to be redone. Polling on a condition does not remove flakiness; it removes the mechanism that produces it, that is, the guess. #### Summary - This lesson's overrun: minority. The test that waited five rounds could not catch up with the ready round in two of the eight runs. - Flakiness is not an overrun category but the mechanism that produces the overrun; the measurement only counts how many runs the overrun spreads across. - Raising the wait to nine reset the failed runs to zero but carried the rounds spent above the budget in every run: all. - Polling on a condition finished every run in its own ready round and stayed within the 9-round budget. - An idling resource reports to the test whether the app is busy; the test does not guess the duration, it asks the app. #### Next Step The test is written, made stable, and running. But something can be said about the code itself without running a test. The next lesson, Static Analysis, measures in how many modules the rule violations that a linter finds per module exceed the budget. ### Static Analysis Source: https://academia.sh/en/courses/testing-performance-and-release/static-analysis Static analysis reads code without running it and lists rule violations. The lesson measures in how many modules the number of violations per module exceeds the budget, and how a baseline file changes the count. The previous lesson made the test stable. A test verifies by running code; some code, however, can be said to be wrong without ever being run. An unused resource, a stream that was not closed, and an API called on the wrong version are examples. The linter does this job automatically. The unit of work measured in this lesson is a rule violation, and the measurement is taken per module. The course's question is the same: **in how many samples does this measurement exceed its budget?** The answers are within, minority, majority, and all; this time the sample is a module. The core is the same as in the previous lessons. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### Reading Code Without Running It Static analysis reads the source code without running it and checks it against a list of rules. Everything it finds is called a violation. A violation can be a defect, it can be a warning, or it can be a false positive of the linter: the code is correct but the rule reads it wrongly. Telling these three apart is the job of the team, not of the linter. The behavior of the linter comes from a configuration file: which rule is on and which is counted as an error is written there. The violations accepted as a baseline are kept in a separate file. ```xml ``` This piece belongs to the platform: which rule is counted as an error, which stays a warning, and which path is exempted from a rule. The model does not read the configuration; it takes the number of violations per module from a fixed table and compares it with the budget. **TE1.** In the model, the violation counts of eight modules are a fixed table; in a real project the numbers vary with the linter's version and rule set. The table represents a single check made with the same rule set. #### Violations per Module The helper file defines the modules and the number of violations. Each module has two numbers: the violations that existed before the baseline and the newly added violations. In addition, the violations for which the linter gave a false positive are counted separately in each module. ```kotlin // Lint.kt — the violation table of the modules class Module(val name: String, val existing: Int, val added: Int, val falseAlarms: Int) val modules: List = listOf( Module("app", 6, 2, 1), Module("core", 3, 0, 0), Module("data", 7, 1, 2), Module("ui", 9, 3, 1), Module("network", 2, 0, 0), Module("auth", 5, 1, 0), Module("feature-cart", 8, 0, 2), Module("feature-search", 4, 2, 1), ) fun lintFindings(m: Module): Int = m.existing + m.added fun cleanedFindings(m: Module): Int = lintFindings(m) - m.falseAlarms fun newFindings(m: Module): Int = m.added ``` `lintFindings` is the raw number of violations of the module. `cleanedFindings` is the number after the false positives have been filtered out. `newFindings` is what remains after the baseline: the newly added violations. The team allows each module to carry at most five violations. ```kotlin // Main.kt fun main() { val samples = modules.map { lintFindings(it) } println("findings: $samples") val overrun = Overrun("raw-findings", Budget("finding", 5), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text findings: [8, 3, 8, 12, 2, 6, 8, 6] raw-findings: budget=finding<=5 | over=6/8 -> majority within: 0 minority: 0 majority: 1 all: 0 ``` Six of the eight modules are above five violations. Only `core` and `network` are within the budget. The measurement is majority. The rule does not see the magnitude this time either: `ui` with twelve and `auth` with six count as having exceeded in the same way. It does not see the cause either: one module may have piled up its violations over years, and another may have been written within a week. #### Filtering Out False Positives The first reaction is that some violations are false positives. That is true: there are seven false positives across the eight modules. If they are filtered out, the count changes. The question is in how many modules this rescues the budget. ```kotlin // Main.kt — after the false positives are filtered out fun main() { val samples = modules.map { cleanedFindings(it) } println("filtered findings: $samples") val overrun = Overrun("filtered-findings", Budget("finding", 5), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text filtered findings: [7, 3, 6, 11, 2, 6, 6, 5] filtered-findings: budget=finding<=5 | over=5/8 -> majority within: 0 minority: 0 majority: 1 all: 0 ``` Five modules are still above the budget, and the measurement stayed majority. `feature-search` dropped to five violations and stayed at the limit of the budget; because only strictly exceeding counts, it has no longer exceeded. Filtering out the false positives lowered the number but did not change the class. The false positives were not the main cause of the overrun. #### The Kinds of Violations A linter does not give the same weight to everything it finds. Each rule has a severity, and the configuration file sets it. A violation at the error level stops the continuous integration pipeline; a violation at the warning level appears in the report but does not stop the pipeline; a violation at the information level is only a suggestion. The same number of violations carries very different meanings depending on the distribution of severity. In this lesson the measurement does not separate severity: all violations count as one unit. This is a simplification, and in a real project the team can track those at the error level with a separate budget. Violations also have two separate origins. Some rules catch genuinely wrong code: a stream that is not closed, a missing permission in the manifest, an unused resource. Others enforce a style preference: hardcoded text or naming. The first group aims at defects, the second at conformity. The team must know which of them it counts toward the budget; otherwise a budget overrun can be read as neither a defect nor a style issue. #### Reading the Configuration Piece Line by Line The root element `` of the `lint.xml` file defines the scope of the configuration. Each `` element names a rule, and the `id` field is the identity of the rule. The first element raises the `UnusedResources` rule to the `error` level: an unused resource now stops the pipeline. The second element leaves the `HardcodedText` rule at the `warning` level: hardcoded text appears in the report but does not stop the pipeline. The third element makes the `OldTargetApi` rule an `error` and carries an `` line inside. This line says that the files under the `legacy` directory will not be checked for this rule. This is not a baseline; it is not applying the rule to a certain path. The difference between the two is important. An exemption is a decision and stays visible in the configuration. A baseline is a record of the past and is kept separately as a file. #### The Causes of False Positives A false positive means the linter misunderstood the code. A rule looks only at the structure of the code and does not know the runtime. A resource used only through reflection makes the rule think that the resource is unused. If a function is called dynamically from another module, the rule may count it as dead code. In these cases the right remedy is not to fix the violation in the code but to exempt the line or the path deliberately. Filtering out false positives has two risks. The first is that real violations are filtered out as false positives; the team loosens the limit to lower the number. The second is that the filtering is not documented; a year later nobody knows why it was filtered out. In this lesson we kept the number of false positives explicitly separate in the table, because making the filtering visible keeps the budget honest. #### The Risks of a Baseline A baseline file is a ledger of debt, and like every ledger of debt it can grow. Three risks stand out. The first is invisible debt. The 44 violations that are hidden are not discussed in team meetings, because they are not in the report. Counting only new violations makes people forget that accumulated debt exists. For this reason the hidden number is reported together with the visible number. The second is fragility of matching. A baseline ties a violation to a file and a rule id. If the file is reorganized or the code is moved, the same violation does not match the baseline and shows up again as a new violation. It looks like a false positive, but its cause is the change in the code. The third is the renewal of the baseline. If the team one day produces the baseline from scratch, all the violations added up to that day turn into old debt. This operation should be recorded as a decision and its reason written; otherwise the baseline turns into a silent way of resetting the budget. #### Where the Check Sits in the Pipeline Static analysis can run the same rules in three places, and each has a different trade-off. If it runs on the developer's local machine, while the code is written or saved, the feedback is fastest, but there is no guarantee that everyone on the team runs with the same version and settings. If it runs in a pull request before a change is merged, it is known that everyone is checked with the same settings, and a new violation is caught before merging. In a full check that runs at night, the raw violation counts of all modules are collected. The two numbers measured in this lesson correspond to two different places. The new-violation budget suits the pull request check: a change may bring at most one new violation. The raw budget of five violations per module suits the full check that runs at night: the total debt of the module is tracked. #### Tightening the Budget over Time While violations have accumulated, it is not possible to pull the budget to zero all at once. Teams usually lower the budget gradually: five violations per module today, four after three months. With each drop the number of exceeding modules may rise, because the limit has tightened. In this lesson the effect of moving from 5 to 4 can be seen: in the filtered count `feature-search` stayed at the limit with five violations, and if the budget goes down to four, the number of exceeding modules rises to six. Tightening and improvement are not the same thing. When the number of exceeding modules goes up, the team may think the code has gotten worse; yet the limit has changed. For this reason a change of budget is stated as a separate note in the measurement report. Improvement is the number of exceeding modules going down with the same budget. #### Rule Selection As the rule set grows, the number of violations grows, but the value does not grow at the same rate. If the team turns on all the rules, every module exceeds the budget and the measurement comes out all; all no longer distinguishes anything. If all the rules are turned off, no module exceeds and the measurement says nothing. A useful rule set consists of the rules the team is really willing to fix. Keeping a rule on that will not be fixed only produces noise. One way to add a rule is a trial period. A new rule is first turned on at the warning level, the number of violations is tracked for a few weeks, and the false positive rate is observed. If the rule finds useful violations, it is raised to the error level; if it produces noise, it is turned off or narrowed with an exemption specific to a path. This order keeps the rule from appearing to the team as a sudden obstacle, and it preserves the meaning of the budget: only the rules the team has adopted count toward the budget. A violation of a rule that has not been adopted inflates the budget and makes it harder for real problems to show. For this reason reviewing the rule set is as regular a job as reviewing the budget; every quarter the team asks which rules are working and removes the ones that are not from the set. #### Common Misconceptions The first misconception is the thought that "if the number of violations is zero, the code is correct". The linter knows only the rules that are defined. A logic error, a wrong calculation, or a security hole often falls outside the rule set. Zero violations means there are no violations that the rules can find. The second misconception is the thought that "a baseline fixes the violations". We saw this in the measurement: 44 violations stayed in the baseline, the count changed, the code did not. A baseline is a measuring tool, not a repair tool. The third misconception is the thought that "the linter is useless because there are many false positives". Seven false positives are a small part of the forty-four violations. In the measurement, filtering out the false positives brought the number of exceeding modules from six to five; that is, the false positives were real but they were not the main cause of the overrun. Giving up the linter means giving up the real violations as well. The fourth misconception is the thought that "the budget must be the same in all modules". A core module and an experimental module may not be measured with the same expectation. But separating the budget per module muddles the measurement: if each module has its own limit, the number of exceeding modules no longer answers the same question. In this lesson we set a single budget and we state openly that this is a simplification. #### The Bounding Measurement The bounding measurement here is the baseline file. The reasoning is this: the raw count mixes debt and new violations. Taking the old violations out of the count re-measures the same table of modules in a way that answers another question. The budget fits this question as well: the new-violation budget is 1, because each change should be able to bring at most one new violation. It is not possible to fix the accumulated violations overnight. The common remedy is the baseline file: the linter records that day's violations and in later checks reports only the new violations. The old violations drop out of the count. The budget is also set according to the new violations: at most one per module. ```kotlin // Main.kt — only new violations with the baseline file fun main() { val samples = modules.map { newFindings(it) } println("new findings: $samples") val hidden = modules.sumOf { it.existing } val visible = modules.sumOf { newFindings(it) } println("hidden in baseline: $hidden") println("visible: $visible") val overrun = Overrun("baseline", Budget("new-finding", 1), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text new findings: [2, 0, 1, 3, 0, 1, 0, 2] hidden in baseline: 44 visible: 9 baseline: budget=new-finding<=1 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The measurement came from majority to minority: only `app`, `ui`, and `feature-search` exceed the new-violation budget. But the reason for this is not that the violations were fixed; it is that the count changed. The 44 hidden violations stand where they stood. A baseline file does not fix a violation; it changes what the linter looks at. These two measurements answer different questions: the first asks "how much debt is in this module", the second asks "what was newly added to this module". The two do not stand in for each other. **TE2.** In the model, the baseline is a single number: the earlier number of violations. A real baseline file records the violations one by one; when the code changes, a violation may not match its record and may show up again as a new violation. #### Summary - This lesson's overrun: majority. Six of the eight modules exceeded the budget of five violations per module. - Filtering out false positives brought the number of exceeding modules from six to five but left the measurement at majority; the main cause was not the false positives. - A baseline file does not fix violations, it changes the count: 44 violations were hidden, 9 new violations became visible, and the measurement came to minority. - The raw count answers "how much debt is there", and the count with a baseline answers "what was added"; the two are read together. - The rule does not see the magnitude or the cause of a violation; it only counts in how many modules the budget is exceeded. #### Next Step Testing and checking examined the correctness and the order of the code. How the app first appears to the user is a separate measurement. The next topic begins with Startup Time, a lesson that measures in how many samples the work done while the app opens exceeds its startup budget. ## Performance and Debugging Cold, warm and hot starts, the causes of dropped frames, leak detection and memory profiling, measuring network and battery use, benchmarking that catches performance regressions, and code shrinking and obfuscation. ### Startup Time Source: https://academia.sh/en/courses/testing-performance-and-release/startup-time Cold, warm and hot starts do different amounts of work before the first frame. The lesson measures in how many of eight launches the first-frame budget is exceeded, and shows that one piece of work yields two overruns under two budgets and that deferred work does not disappear. In the previous lesson static analysis looked at the code without running it; the baseline file removed the old violations from the count, and only the modules with new violations exceeded the budget. In this lesson we look at the code while it runs. The app opens on the user's device and does a certain amount of work until it draws the first frame. The question is whether that work fits a budget. The course's measure does not change: in how many samples a measurement exceeds its budget. There are four answers. If no sample exceeds the budget the overrun is **within**; if fewer than half exceed it, **minority**; if half or more exceed it but not all, **majority**; if all exceed it, **all**. A sample equal to the limit is not an overrun. In the core, eight samples produced each overrun once; in this lesson the same classification is applied to eight launches. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Budget` carries a measurement's name and limit. `Overrun` compares the samples with the limit and counts the ones that exceed it. `countOverruns` counts the four overruns in a fixed order. This file is the same in every lesson of the course and never changes again. #### Three Kinds of Start The app can open in one of three states. A **cold start** is when the process does not exist yet: the platform sets up the process, the app's startup code runs, the first screen is created and the first frame is drawn. A **warm start** is a subset of the work in a cold start: in most cases the process is still in memory and only the screen has to be recreated; if the platform has evicted the process from memory, the process is rebuilt too, but the screen benefits from its saved state. A **hot start** is when the screen sits in memory and is merely brought to the front; object and UI setup are mostly not repeated, and only the screen is redrawn. The first frame is the first drawing the user sees from the app. Startup time is the name for the time between the launch request and this frame. In this lesson we do not measure time. What we will measure is the number of steps of work done up to the first frame, and that number comes from inside the model. ```kotlin // Startup.kt — computes the work done before the first frame, in steps, by start kind class Phase(val name: String, val steps: Int) fun startupPhases(kind: String): List = when (kind) { "cold" -> listOf( Phase("process", 2), Phase("app", 1), Phase("dependency", 4), Phase("screen", 2), Phase("draw", 1), ) "warm" -> listOf(Phase("screen", 2), Phase("draw", 1)) else -> listOf(Phase("draw", 1)) } fun startupWork(kind: String): Int = startupPhases(kind).sumOf { it.steps } ``` `Phase` names one phase on the way to the first frame. `startupPhases` returns the phases for a start kind; `startupWork` adds up the steps of the phases. We look at what the three kinds look like. ```kotlin // Main.kt — lists the work the three start kinds do before the first frame fun main() { for (kind in listOf("cold", "warm", "hot")) { val parts = startupPhases(kind).joinToString(" ") { "${it.name}=${it.steps}" } println("$kind: $parts | total=${startupWork(kind)}") } } ``` ```text cold: process=2 app=1 dependency=4 screen=2 draw=1 | total=10 warm: screen=2 draw=1 | total=3 hot: draw=1 | total=1 ``` **PD1.** In the model a step is a number that shows how heavy a phase is; it is neither a second nor a frame. The steps of the phases are a lesson decision: dependency initialization is set up as the heaviest phase in a cold start. In a real app a measurement tells you which phase is heavy; the model only shows how the rule works. The cold start's total is more than three times the warm start's, and the reason is visible: the process, app and dependency phases do not exist at all in a warm start. That is why someone looking for the source of startup work looks at the cold start first. On the platform, the following pieces are used to trace these phases and to report that the first interaction has finished. ```kotlin // Platform code: tracing startup work and reporting the end of the first interaction import android.app.Application import android.os.Bundle import android.os.Trace import androidx.appcompat.app.AppCompatActivity class SampleApp : Application() { override fun onCreate() { super.onCreate() Trace.beginSection("app-startup") try { prepareCore() } finally { Trace.endSection() } } } class HomeActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.home) } // called when the first data load completes private fun showFirstContent(items: List) { render(items) reportFullyDrawn() } } ``` This piece is platform code, and here we claim no output, duration or file path. The `Trace` calls show a section of the startup code as a named slice in the IDE's profiler; `reportFullyDrawn` tells the platform the moment the app is ready for use. The model sets up the rule of this measurement; the model is not the tool itself. In addition, part of the cold start, namely getting the compiled code ready, can be brought forward with a file called a baseline profile. The gain from a baseline profile depends on the app and the device and is not quoted as a number here. #### Eight Launches Now we set a budget. Let the first-frame budget be six steps. The kinds of the eight launches come from a fixed list: two cold, two warm, four hot. The work of each launch is computed with `startupWork`, so the samples are the return values of a model function. ```kotlin // Main.kt — eight launches are measured against the first-frame budget fun main() { val launches = listOf("cold", "hot", "warm", "hot", "hot", "warm", "cold", "hot") val samples = launches.map { startupWork(it) } println("work: $samples") val overrun = Overrun("launch", Budget("first-frame", 6), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text work: [10, 1, 3, 1, 1, 3, 10, 1] launch: budget=first-frame<=6 | over=2/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` Two of the eight launches, the cold ones, exceeded six steps: 2/8, so **minority**. The six warm and hot launches stayed well below the budget. This result does not say that an average launch is good; the average is not computed here. What it says is that the samples exceeding the budget are in the minority. If most of an app's launches are warm or hot this number stays small, and yet the user's first encounter, the cold start, may be exceeding the budget every time. **PD2.** The list having two cold, two warm and four hot launches is an assumption of the model. In real use the ratio of the kinds varies with the user, the device's memory pressure and how often the app is closed, and that ratio is not presented here as measured data. The rule does not know the ratio of the kinds; it only counts the samples it is given. There are two things the rule does not see. The first is the size of the overrun: whether the cold start exceeds the budget by one step or by four, `over` stays the same. The second is the cause of the overrun: the cold start's total is ten, but the rule does not know whether this comes from the dependency phase or from process setup. The cause lies in the output of `startupPhases`, not in the line of the rule. The rule counts in how many samples the budget is exceeded, not where the cost is paid. #### Same Work, Two Budgets We apply three different budgets to the same samples. A budget is not a property of the function but a question put to it; when the strictness of the question changes, the same function yields a different overrun. ```kotlin // Main.kt — the same eight launches are measured with three budgets fun main() { val launches = listOf("cold", "hot", "warm", "hot", "hot", "warm", "cold", "hot") val samples = launches.map { startupWork(it) } val overruns = listOf( Overrun("wide", Budget("first-frame", 10), samples), Overrun("measured", Budget("first-frame", 6), samples), Overrun("tight", Budget("first-frame", 2), samples), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text wide: budget=first-frame<=10 | over=0/8 -> within measured: budget=first-frame<=6 | over=2/8 -> minority tight: budget=first-frame<=2 | over=4/8 -> majority within: 1 minority: 1 majority: 1 all: 0 ``` The budget of ten did not exceed any sample, because the heaviest launch was exactly ten steps and a sample equal to the limit is not an overrun. The budget of six exceeded two samples, and the budget of two exceeded four. Under the budget of two, the warm launches also became overruns: the three-step warm start passed the two-step limit. Four overrunning samples are exactly half of eight, and half counts as **majority**; with one sample fewer the result would have been minority. **PD3.** The lesson does not say which of the three budgets is right. Setting the budget is a product decision, and the rule counts only according to that decision. What matters here is this: the launch code did not change at all, only the question changed, and the overrun moved from within to majority. That is why an overrun alone does not show a good or a bad app; it is read together with the budget and the samples it was counted with. #### How to Choose a Budget In this lesson we tried three budgets and declared none of them right. The reason is that a budget is not a measurement result but a product decision. The person who sets the budget assumes the longest the user is willing to wait after opening the app; the rule applies that assumption to the samples. If the budget is too wide, every launch looks within and a regression goes unnoticed. If the budget is too tight, every launch exceeds it and the report stops saying anything, because everything is red. A good budget draws on two sources. The first is the app's own history: the measurement of the released version is a ceiling for later versions. The second is the start kind. Because cold, warm and hot starts do not do the same work, judging all of them with a single budget misleads. Setting a separate budget per kind, or pinning the measurement to a single kind, makes it more readable where an overrun comes from. A budget also concerns the moment being measured. In this lesson the first-frame budget and the first-interaction budget are separate questions: one looks at the moment the user sees the screen, the other at the moment the screen can respond to a touch. Merging the two into one number hides where the deferred work is paid. Another trap in startup measurement is the point where the measurement starts. The user starts the clock the moment they touch the icon; the app's own code runs after the process has been set up. A measurement taken from inside the app therefore does not see the platform's share of process setup. In this lesson the model's `process` phase keeps that share as a separate line; in a real measurement, numbers are not compared until it is written down which point they started from. Someone improving startup often focuses on the cold start, because most of the work is done there. But the warm start may be the case the user experiences most often during the day. Tracking the two kinds with separate budgets makes it possible to notice when one improves while the other goes backward. For the same reason a startup budget carries in its name which kind is targeted. "First-frame budget" is a vague name; "cold start, first frame, six steps" says who counts what against what. A budget whose name is not specific turns, years later, into a number nobody can interpret. #### Common Misconceptions **"If the hot start is measured, startup has been measured."** In a hot start the process and the app object are already in memory; the work done is mainly bringing the screen to the front. Measuring this kind leaves most of the startup work out. In the model too the hot launch is a single step and exceeded no budget; this does not show that the app's startup code is good. **"Deferring work past the first frame removes the work."** The deferred work is still done; the user just does not have to wait for it until they see the first frame. That is why the first-interaction budget is a separate measurement. Deferral moves the work from one budget to another; a gain cannot be declared without reading the two budgets together. **"A single measurement is enough."** The same launch can do different work under different conditions. That is why the lesson set up a series of eight launches and ran the rule over the number of samples. With a single sample the overrun can only be within or all; the intermediate categories appear only with several samples. **"The number of overruns tells the quality of the app."** The number carries meaning only together with the chosen budget and the chosen samples. The same app produced three different overruns under three budgets; the app did not change at all. #### The Bounding Measurement The heaviest phase of the cold start was dependency initialization. A common fix is to defer this work past the first frame: the first frame is drawn, the user sees the screen, and the dependency is prepared in the background. In the model we set this up with two functions. One returns the work remaining until the first frame, the other the deferred work. ```kotlin // Main.kt — the heavy dependency is deferred past the first frame; the work moves to the first interaction fun beforeFirstFrame(kind: String): Int = startupPhases(kind).filter { it.name != "dependency" }.sumOf { it.steps } fun afterFirstFrame(kind: String): Int = startupPhases(kind).filter { it.name == "dependency" }.sumOf { it.steps } fun main() { val launches = listOf("cold", "hot", "warm", "hot", "hot", "warm", "cold", "hot") val first = launches.map { beforeFirstFrame(it) } val later = launches.map { afterFirstFrame(it) } println("first frame: $first") println("first interaction: $later") val overruns = listOf( Overrun("frame-work", Budget("first-frame", 6), first), Overrun("interaction-work", Budget("first-interaction", 3), later), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text first frame: [6, 1, 3, 1, 1, 3, 6, 1] first interaction: [4, 0, 0, 0, 0, 0, 4, 0] frame-work: budget=first-frame<=6 | over=0/8 -> within interaction-work: budget=first-interaction<=3 | over=2/8 -> minority within: 1 minority: 1 majority: 0 all: 0 ``` The first-frame budget is now **within**: the cold start dropped to six steps, and because six equals the limit it was not counted as an overrun. But the work did not disappear. The four-step dependency initialization was left to the first interaction, and that separate budget, with its limit of three steps, was again exceeded in two samples: **minority**. The deferral took the cost off the first-frame line and wrote it onto the interaction line. **PD4.** The first-interaction budget being three steps is a lesson decision. How much work the user's first touch can bear depends on what the app does. It is also not claimed that deferral is always bad: the user may look at the screen for much longer before the first interaction, and the deferred work finishes in that time. The model does not set up this parallelism. The only thing it sets up is that the deferred work is counted again in another budget. Here too the rule sees how many samples overrun, not by how much or why. The improvement on the first-frame line is real, but that line alone does not give the whole picture. Moving the work elsewhere to reduce startup work and actually removing the work are different things; the rule separates them only when the two budgets are put side by side. #### Summary - This lesson's overrun: minority. Two of the eight launches, the cold ones, exceeded the six-step first-frame budget. - A cold start carries the process, app, dependency, screen and draw phases; a warm start does the screen and the draw, and a hot start only the draw. - The same eight samples came out within under the ten-step budget, minority under six steps, and majority under two steps. An overrun is a property not of the function but of the function together with the budget. - A sample equal to the limit is not an overrun: a six-step first-frame workload does not exceed a six-step budget. - Deferring the dependency initialization past the first frame made the first frame within; the work did not disappear, it reappeared as minority in a separate first-interaction budget. #### Next Step The first frame is drawn, but the app's work is not over. When the user scrolls a list, every frame is redrawn and every frame has its own budget. The next lesson, `Rendering Smoothness`, measures in how many frames the frame budget is exceeded when every frame recomposes all the items of a scrolled list. ### Rendering Smoothness Source: https://academia.sh/en/courses/testing-performance-and-release/rendering-smoothness Smooth scrolling depends on every frame staying within its budget. The lesson measures a scrolled list in which every frame recreates all the items, shows how a stable key narrows the recreated work, and shows why the rule does not see the size of an overrun. In the previous lesson the first frame was drawn, and the heavy work of the cold start moved to another budget. The user is now looking at the screen and scrolling a list. As the scrolling continues, every frame is redrawn; every frame has to be ready within a time that depends on the screen's refresh rate. We call this time the **frame budget**. A frame that cannot make the budget becomes a **dropped frame**; the user sees it as a stutter. In this lesson we do not measure time, because the time of a measurement environment is not a reliable number. Instead we count the work done per frame: how many items were recreated in a frame. The budget is in the same unit. The measure is the course's measure, that is, in how many frames this budget was exceeded. The measure is built with the same two classes in every lesson of the course. `Budget` carries the name and limit of a measurement, `Overrun` counts the samples that exceed the limit and gives one of four overruns, and `countOverruns` counts the four overruns in a fixed order. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` #### The Scrolled List Let a list have twenty items, and let five of them fit on the screen at the same time. Scrolling moves the list's first visible item in every frame. The helper file below defines the item, the visible window and the crudest frame work. ```kotlin // Frames.kt — computes the items recreated in one frame of a scrolled list class Item(val key: Int) fun buildItems(count: Int): List = List(count) { Item(it) } fun visible(items: List, first: Int, window: Int): List = items.drop(first).take(window) fun frameWork(items: List): Int = items.size ``` `Item` carries an item and the key given to it. `visible` returns the items that fit on the screen, starting from the first visible item. `frameWork` sets up the crudest rule: the frame recreates all the items of the list. We first look at the scrolling itself. Over eight frames, the position of the first visible item comes from a fixed sequence: a slow scroll, a pause and then a fast scroll. ```kotlin // Main.kt — the items entering the screen over eight frames fun main() { val items = buildItems(20) val positions = listOf(0, 1, 2, 2, 3, 8, 9, 9) for ((index, first) in positions.withIndex()) { val keys = visible(items, first, 5).map { it.key } println("frame-$index: $keys") } } ``` ```text frame-0: [0, 1, 2, 3, 4] frame-1: [1, 2, 3, 4, 5] frame-2: [2, 3, 4, 5, 6] frame-3: [2, 3, 4, 5, 6] frame-4: [3, 4, 5, 6, 7] frame-5: [8, 9, 10, 11, 12] frame-6: [9, 10, 11, 12, 13] frame-7: [9, 10, 11, 12, 13] ``` **PD1.** The scrolling sequence is the input of the model. Frame-3 is a pause, because the first visible item did not change. Frame-5 is a scroll where the finger drags fast: the visible window jumped five items from one frame to the next, so none of the items in the previous frame stayed on the screen. It is not claimed that a real scroll looks like this; the sequence was chosen to show the rule. #### Everything in Every Frame We measure the crude rule against the budget. Let the frame budget be four items. In a list of twenty items the work of each frame is twenty, because the frame recreates not the five items visible on the screen but all of the list. We take the measurement with three list lengths. The samples are the return values of a model function: `frameWork` is called for each frame. ```kotlin // Main.kt — all items are recreated in every frame fun main() { val positions = listOf(0, 1, 2, 2, 3, 8, 9, 9) val budget = Budget("frame", 4) val overruns = listOf(20, 12, 4).map { count -> val items = buildItems(count) val samples = positions.map { frameWork(items) } Overrun("list-$count", budget, samples) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text list-20: budget=frame<=4 | over=8/8 -> all list-12: budget=frame<=4 | over=8/8 -> all list-4: budget=frame<=4 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 2 ``` In the lists of twenty and twelve items, eight of the eight frames exceeded the budget: **all**. In the list of four items none exceeded it: **within**. The code is the same, and only the length of the list changed. The reason is the rule the model sets up: the frame work depends not on the number of items visible on the screen but on the number of items in the list. While the list is short the rule goes unnoticed; as it grows, every frame gets heavier in the same proportion. **PD2.** The budget being four items is a lesson decision. The frame budget is in reality a time, and how many items fit in that time depends on how complex an item is, on the device and on the screen's refresh rate. The four items in the model are a choice of measurement unit, not a claim about a fitting limit. That all items have the same weight is also an assumption. Here, all is not the name of bad code but of a situation the rule can see: every frame exceeded the budget. The rule does not say by how much the frames exceeded it. Every frame of the twenty-item list is five times the budget, and that of the twelve-item list three times; both fall on the same line, **all**. To see the size you either look at the output of `frameWork` or tighten the budget step by step. On the platform, a frame measurement library is used to track dropped frames in production. The piece below listens to each frame of the window and records the dropped frames. ```kotlin // Platform code: tracking dropped frames import android.os.Bundle import androidx.activity.ComponentActivity import androidx.metrics.performance.JankStats class ListActivity : ComponentActivity() { private lateinit var jankStats: JankStats override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) jankStats = JankStats.createAndTrack(window) { frameData -> if (frameData.isJank) { recordJank(frameData.states) } } } override fun onResume() { super.onResume() jankStats.isTrackingEnabled = true } override fun onPause() { super.onPause() jankStats.isTrackingEnabled = false } } ``` This piece is platform code, and here we claim no output, duration or file path. `JankStats` gives a data object for each frame, and this object says whether that frame was dropped. The model sets up the rule of this measurement; the model is not the tool itself. The real library decides a dropped frame by time, not by the model's step count. #### Reading the Frame Budget The frame budget is the upper limit of the amount of work set aside for drawing a frame. The screen asks for a new frame at regular intervals; a frame that is not ready within that interval is dropped, and the user sees it as a stutter. In the model we use the number of work steps instead of the interval, because we do not measure time in this repository. The budget is the largest number of work steps in one frame, and the rule counts how many frames pass this limit. This framing has two consequences. The first is that even a single expensive frame counts as an overrun; and the user often notices even a single stutter. The second is that the rule does not know the order of the frames. A scroll in which three frames in a row exceed the budget and three overruns scattered among other frames are reported with the same number. For the user, though, the first is a noticeable freeze and the second a slight roughness. Recomposition is the source of cost in this framework. If a frame builds every visible item from scratch, the work grows with the number of items. If only the changed items are recreated in the list, the work stays bounded by the number of changed items. A stable key is the identity that makes it possible to tell which item changed. #### Why Scroll Speed Matters In the lesson's scrolling sequence the window advanced one item in most frames, did not advance at all in one frame, and jumped five items at once in one frame. The costs of these three situations are not the same. In the paused frame there is no new item; in a slow scroll every frame brings one item; in a fast scroll the whole window is renewed. A smart list does almost no work in the first case, little in the second, and a window's worth of work in the third. This difference affects the design of the measurement. A list tested only with slow scrolling comes out clean, and the same list tested with fast scrolling exceeds the budget. For this reason frame samples should be chosen not from a single way of using the list but from the mix of movements the user really makes. In the model we wrote the mix ourselves; in a real app the mix comes from usage records, and only those records, not a model, know it. #### Work Outside the Window A lazy list can prepare slightly more than what fits on the screen in advance; when the user scrolls, the item is ready. This preparation does not remove the work, it only pulls its time earlier than the scroll. In terms of the frame budget the gain is real, because the work leaves the scrolling frames; but the preparation is also paid somewhere, and that place is another budget. The same principle held for the deferred work in the startup lesson: work can be moved from one budget to another, and an improvement is not complete unless the two budgets are read together. The limit of the model is clear here. `frameWork` counts only the items recreated in one frame and writes the pre-prepared items nowhere. Someone who wants to measure the effect of preparing ahead must add a second counter to the model; it was not added in this lesson, because the lesson answers a single question: in how many frames recreation exceeds the frame budget. Tying the measurement to the frame budget has another benefit: the goal is clear. "The list should be smooth" is a wish; "in the twenty-item list, none of the eight frames should exceed four items" is a sentence that can be tested. A testable goal shows, when a change is made, whether the result improved, without leaving room for debate. Keeping the budget and the sample sequence together with the code ensures that this sentence is not forgotten over time. Finding which item the frames that exceed the frame budget come from is the second step. The first step is knowing how many frames exceed it; the second is looking for what those frames have in common. In this lesson the common point was the first frame and the five-item jump in the scroll sequence, and both could be read from the samples. In a real app, the information is provided by the output in which the profiling tool gives a per-frame breakdown. #### Common Misconceptions **"If the list is short, there is no rendering problem."** A short list stays within the budget; the lesson showed this with the four-item list. But this does not mean the same code will keep the budget in a long list. The work grows with the number of items, and a measurement is valid only for the dimension it tested. **"A stable key makes rendering faster."** The key does not reduce the work itself; it determines which work will not be repeated. If the key is chosen wrongly, for example if the position number is used as the identity, then when items change places the same item appears with a different identity and the work starts over. **"A stutter can be summarized with a single number."** The number of exceeding frames gives the frequency of the stutter; it does not give its size or its consecutiveness. That is why the lesson's bounding measurement teaches reading an improvement not by the overrun category alone but together with the per-frame work. **"Rendering is only about rendering code."** The work of a frame also includes the preparations done before drawing: data transformation, layout computation, determining the visible items. In the model all of these were gathered inside `frameWork`; in a real app the profiling tool says which part is expensive. #### The Bounding Measurement The remedy for the crude rule is to give every item a stable key and to recreate only the item whose key changed or which has newly become visible. In a scrolled list an item is recognized by the same key unless it leaves and enters the screen; the platform does not have to rebuild its content. The piece below is its counterpart on the platform. ```kotlin // Platform code: scrolled list with a stable key import androidx.compose.foundation.lazy.LazyColumn import androidx.compose.foundation.lazy.items import androidx.compose.runtime.Composable @Composable fun MessageList(messages: List) { LazyColumn { items(messages, key = { it.id }) { message -> MessageRow(message) } } } ``` A lazy list creates only the items that fit on the screen, and the key gives each item an identity that does not change. In the model we set it up as follows: the work of a frame is the number of items that were not visible in the previous frame. In the first frame there is no previous frame, so the whole window is recreated. ```kotlin // Main.kt — with a stable key only the items entering the screen are recreated fun keyedWork(previous: List, current: List): Int { val seen = previous.map { it.key }.toSet() return current.count { it.key !in seen } } fun main() { val items = buildItems(20) val positions = listOf(0, 1, 2, 2, 3, 8, 9, 9) val windows = positions.map { visible(items, it, 5) } val samples = windows.mapIndexed { index, now -> keyedWork(if (index == 0) emptyList() else windows[index - 1], now) } println("work: $samples") val overrun = Overrun("keyed", Budget("frame", 4), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text work: [5, 1, 1, 0, 1, 5, 1, 0] keyed: budget=frame<=4 | over=2/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The same scroll, the same budget and the same twenty items: the overrun dropped from **all** to **minority**. The recreated work fell from twenty to one, sometimes to zero. But two frames still exceed the budget. The first is the first frame; when the list is built for the first time, all five items are new. The second is the fifth frame of the fast scroll; because the window jumped five items at once, all five count as new. The key narrows recreation to the items entering the screen, but if the number of entering items is larger than the budget it does not save the frame. **PD3.** The "newly visible item" in the model and recreation in a real list are not exactly the same. In reality, if an item's content changes, it is recreated even when its key stays the same; in the model the content does not change. The key prevents unnecessary recreation only by preserving an item's identity. The rule does not see this difference either. We say plainly what the rule does not see. The difference between all and minority is a difference of class in the rule's eyes, but in terms of the work it is much bigger: the total work fell from twenty items times eight frames, that is one hundred and sixty items, to fourteen. The rule does not give this ratio, only the number of exceeding frames. If we want to see the size of an overrun, we look at the samples themselves, that is, the `work:` line. The rule does not say the cause either: the overrun in the first frame comes from the list setup and the overrun in the fifth frame from the fast scroll, and the two are counted as the same number of overruns. #### Summary - This lesson's overrun: all. In the twenty-item list every frame recreated all the items, and eight of the eight frames exceeded the four-item budget. - The frame budget is in reality a time; in this lesson it was measured as a work count in the same unit, as the items recreated per frame, and no time was written. - The same code came out within in the four-item list and all in the twelve- and twenty-item lists: in the crude rule the frame work depends not on the visible items but on the list length. - A stable key limited recomposition to the items entering the screen and reduced the overrun from the all class to the minority class. The remaining two overruns were the first frame and the five-item jump. - The rule sees how many frames overrun; it does not see the size or the cause of the overrun. #### Next Step Even when scrolling is smooth, every frame leaves new objects in memory. Some of them are not freed after use and stay in memory even when the screen is closed. The next lesson, `Memory Management`, measures in how many cycles the number of retained objects exceeds the budget when a listener that is never unregistered holds a screen on every open-and-close cycle. ### Memory Management Source: https://academia.sh/en/courses/testing-performance-and-release/memory-management A screen that stays in memory after it closes is a memory leak. The lesson measures a listener that is never unregistered and holds one screen per cycle, in how many cycles the retained objects exceed the budget, and how the same leak changes category with the budget. In the previous lesson scrolling fit the frame budget, but every frame leaves objects in memory, and some of these objects are not freed when their work is done. A memory problem looks different from a rendering problem. A stutter is noticed at the moment it happens; memory fills silently, and the app is noticed only when the platform terminates it or when memory runs out. That is why we tie the measurement to a cycle instead of a frame: opening and closing a screen is a cycle, and at the end of each cycle we look at how many objects remain in memory. The name of the problem is the **memory leak**. Garbage collection frees an object that nobody can reach. If there is still a path that reaches an object, that is, if another object holds a reference to it, that object is not freed even when its work is done. We call an object that stays in memory this way a **retained object**. A leak is the situation in which retained objects accumulate cycle by cycle. The measure is again the course's measure: at the end of how many cycles the budget for the number of retained objects was exceeded. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` This file is the same in every lesson of the course. `Budget` carries the name and limit of a measurement, and `Overrun` counts the samples that exceed the limit and gives one of four overruns. #### A Listener That Is Never Unregistered The most common form of a leak is this: a long-lived object holds a reference to a short-lived object. Let there be a data source that lives for the whole app. When a screen opens, it registers itself with this source as a listener, because it wants to receive updates. If the screen does not unregister when it closes, the source keeps the screen in its listener list. The screen's work is done, but the path to it continues; garbage collection cannot free it. ```kotlin // Retention.kt — a listener that is never unregistered keeps the closed screen in memory class Screen(val id: Int) { var closed: Boolean = false } class Source { private val strong = mutableListOf() private val weak = mutableListOf() fun register(screen: Screen) { strong.add(screen) } fun registerWeak(screen: Screen) { weak.add(screen) } fun unregister(screen: Screen) { strong.remove(screen) weak.remove(screen) } // a weak reference does not hold a closed screen; a strong reference does fun retainedObjects(): Int = strong.size + weak.count { !it.closed } } fun cycle(source: Source, id: Int, how: String): Int { val screen = Screen(id) when (how) { "strong" -> source.register(screen) "weak" -> source.registerWeak(screen) "lifecycle" -> { source.register(screen) source.unregister(screen) } } screen.closed = true return source.retainedObjects() } ``` `Source` is the source that lives for the whole app. `register` keeps a record with a strong reference, and `registerWeak` with a weak reference. `cycle` is one cycle: a screen opens, registers with the source, closes, and the number of screens the source is holding at that moment is returned. `how` says how the registration was made. We first run, for eight cycles, the form that never removes the registration, that is, `strong`. Let the budget be three objects. ```kotlin // Main.kt — a listener that is never unregistered accumulates screens over eight cycles fun main() { val source = Source() val samples = (1..8).map { cycle(source, it, "strong") } println("retained: $samples") val overrun = Overrun("leak", Budget("retained", 3), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text retained: [1, 2, 3, 4, 5, 6, 7, 8] leak: budget=retained<=3 | over=5/8 -> majority within: 0 minority: 0 majority: 1 all: 0 ``` Every cycle left a screen behind, and the number of retained objects grew by one each time. The first three cycles stayed within the budget, and five cycles, from the fourth to the eighth, exceeded it: 5/8, that is, **majority**. The nature of a leak shows here: it is not noticed in one cycle, but the accumulating number sooner or later passes every budget. That is why a leak is measured across cycles, not by looking at a single cycle. **PD1.** In the model each screen is one object and each leaking cycle leaves exactly one object behind. A real screen also holds its views, images and data structures beside itself; the size of the retained memory varies a great deal from screen to screen. The model counts screens; it does not set up the size of the memory they hold. #### The Budget Hides the Leak We apply four different budgets to the same leak. The leak is measured once and the samples are the same; only the limit changes. ```kotlin // Main.kt — the same leak is measured with four budgets fun main() { val source = Source() val samples = (1..8).map { cycle(source, it, "strong") } val overruns = listOf(0, 3, 6, 8).map { limit -> Overrun("limit-$limit", Budget("retained", limit), samples) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text limit-0: budget=retained<=0 | over=8/8 -> all limit-3: budget=retained<=3 | over=5/8 -> majority limit-6: budget=retained<=6 | over=2/8 -> minority limit-8: budget=retained<=8 | over=0/8 -> within within: 1 minority: 1 majority: 1 all: 1 ``` Four lines, four overruns and a single leak. Under the budget of zero objects every cycle exceeded it. Under the budget of eight objects no cycle exceeded it, because in the eighth cycle the retained number was eight and a sample equal to the limit is not an overrun. The leak was there at eight objects too; the budget made it invisible. This is the weak side of the rule: if the budget is chosen wider than the leak, the leak is reported as **within**. That is why a leak budget is set not by the number of cycles but by the number the app should never retain. The budget that catches a leak is often based on zero or on a number close to zero. On the platform, a virtual machine check and a heap dump are used to catch retained screens. The piece below sets up a check that writes leaking screens to the log and shows a function that takes a heap dump. ```kotlin // Platform code: leak detection and heap dump import android.app.Application import android.os.Debug import android.os.StrictMode class SampleApp : Application() { override fun onCreate() { super.onCreate() StrictMode.setVmPolicy( StrictMode.VmPolicy.Builder() .detectActivityLeaks() .detectLeakedRegistrationObjects() .penaltyLog() .build() ) } } fun dumpHeap(path: String) { Debug.dumpHprofData(path) } ``` This piece is platform code, and here we claim no output, duration or file path. `StrictMode` writes leaking screens, and broadcast receivers and service connections whose registration was never removed, to the log; the app's own listeners do not fall under this check; the heap dump shows, in the IDE's profiler, which object is held by which path. The model sets up the rule of this measurement; the model is not the tool itself. What the dump will show, that is, which object the retaining path passes through, is information the model does not show. #### The Weak Reference One way to solve the leak is for the source to hold the screen with a weak reference instead of a strong one. A **weak reference** does not prevent garbage collection: if no other strong path to the object remains, the object is freed and the weak reference becomes empty. In the model we set this up as a closed screen no longer being counted. ```kotlin // Main.kt — a screen registered with a weak reference is not retained once it closes fun main() { val source = Source() val samples = (1..8).map { cycle(source, it, "weak") } println("retained: $samples") val overrun = Overrun("weak", Budget("retained", 3), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text retained: [0, 0, 0, 0, 0, 0, 0, 0] weak: budget=retained<=3 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` **PD2.** In the model a weak reference lets go of a closed screen instantly. In reality garbage collection runs when its time comes; after a screen closes it stays in memory for a while, until the collection takes place, and this period varies from app to app. The instantaneousness of the model is a simplification. Also, a weak reference does not replace removing the registration: the source can go on carrying emptied records in its list, and that is an accumulation, though a small one. #### Why We Measure Across Cycles Memory problems are not instantaneous but cumulative. Opening and closing a single screen raises the retained object count by one or not at all; this difference stays below the budget in a single observation. A leak becomes visible by repeating the same operation again and again. That is why the samples of the measurement are the retained object counts taken at the end of consecutive cycles, and each sample contains the one before it. This structure also affects how the rule is read. In a cumulative sequence the overrun usually concentrates toward the end of the sequence: the first cycles stay within, the later ones exceed. In the lesson's first measurement the exceeding cycles were the fourth to the eighth. That is why increasing the number of cycles makes a leak more visible, and decreasing it hides the leak. One more detail matters in measurement design: every cycle must start under the same conditions. If the state left over from a previous cycle affects the next one, the overrun may come from the measurement setup and not from the leak. That is why in the model every cycle runs with the same `cycle` function and on the same source; the only thing that differs is how the registration is made. What a retained object is should also be said openly. A screen is an object, but everything that holds it stays with it: the screen's views, its data, the helpers it uses. Finding a leak therefore requires looking not at the retained object itself but at the path that holds it. A heap dump shows exactly this path. Three questions are asked in order when looking for a leak. Which object is being retained, which path holds it, and when should this path have been cut. The counter answers the first question, the heap dump the second, and the app's lifecycle the third. In the model the path is a single link: the source's list. In reality the path passes through several objects and each link belongs to a different owner; choosing the right link to cut the leak takes more thought than removing a registration. A memory budget is also not a search for a lower bound. Some retained objects while the app is open are legitimate: the open screen, cached data, the user session. A leak budget takes this legitimate share into account and limits only the excess left after the screen closes. That is why the measurement looks not at the moment the screen is open but at the moment it closes, and the model reads the counter at the end of each cycle. Comparing the leak budget between versions is also useful. If a screen that retained zero objects in eight cycles in the previous version retains one object in eight cycles in the next version, that single object is the first sign of a new leak. When the budget is zero this sign shows up immediately as an overrun; when the budget is left wide, a new leak accumulates unnoticed for several versions. #### Common Misconceptions **"Garbage collection cleans up the leak."** Garbage collection frees only objects that cannot be reached. If the source still holds a reference to the screen, the screen is reachable and is not collected. A leak is exactly the existence of a path that still reaches an object whose work is done. **"A weak reference solves every leak."** A weak reference does not prevent an object from being collected, but it does not replace removing the registration either. The source can go on carrying emptied records, and this accumulation, though not as large as the object, is real. The right solution is to tie the registration to the object's lifecycle. **"If the screen has closed, it is gone from memory."** Closing means the screen is not visible to the user; it does not mean it was deleted from memory. In the model the screen whose registration was never removed stayed in the source's list after closing, and the counter never fell again. **"The fix undoes the old leak."** The lesson's bounding measurement showed this plainly: the four screens that leaked before the fix stayed in memory after the fix too. The fix only stops the new leak. **"The budget defines the leak."** The budget defines not the leak but an acceptance limit. The same leak gave four different overruns under four budgets. The budget that catches a leak is set to a value close to the number the app should never retain. #### The Bounding Measurement The arrangement that solves the leak at its root is to tie the listener to the screen's lifecycle: the screen registers when it becomes visible and removes its registration when it stops being visible. Its form on the platform is as follows. ```kotlin // Platform code: tying the listener to the lifecycle import androidx.appcompat.app.AppCompatActivity class OrdersActivity : AppCompatActivity() { private val listener = OrderListener { orders -> render(orders) } override fun onStart() { super.onStart() OrderSource.register(listener) } override fun onStop() { OrderSource.unregister(listener) super.onStop() } } ``` The registration is made in `onStart` and removed in `onStop`, so the source's path to the screen is cut when the screen stops being visible. In the model this is the `lifecycle` form: the screen registers and removes its registration as it closes. We take two measurements. In the first, the arrangement is correct from the very beginning. In the second, the first four cycles run with the old, leaking arrangement and then the fix arrives. ```kotlin // Main.kt — tying to the lifecycle does not increase the retained objects of later cycles fun main() { val fixed = Source() val fromStart = (1..8).map { cycle(fixed, it, "lifecycle") } val patched = Source() val afterPatch = (1..8).map { cycle(patched, it, if (it <= 4) "strong" else "lifecycle") } println("from-start: $fromStart") println("patched: $afterPatch") val overruns = listOf( Overrun("from-start", Budget("retained", 3), fromStart), Overrun("patched", Budget("retained", 3), afterPatch), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text from-start: [0, 0, 0, 0, 0, 0, 0, 0] patched: [1, 2, 3, 4, 4, 4, 4, 4] from-start: budget=retained<=3 | over=0/8 -> within patched: budget=retained<=3 | over=5/8 -> majority within: 1 minority: 0 majority: 1 all: 0 ``` The arrangement set up correctly from the beginning is **within**: no retained objects in any cycle. The fix made afterward came out **majority** under the same budget. After the first four, the fix leaked no new screens and the number stopped at four; but all four stayed in memory. The retained object count did not increase in later cycles, and the earlier leak did not go away by itself. In the rule's eyes the difference between the two lines is between staying under the budget and staying over it; what the rule does not see is that in the second line the growth stopped. To see that growth stopped, we look at the samples themselves, that is, the `patched:` line. **PD3.** That the first four screens are not freed in the second measurement is a rule the model sets up. In a real app a leaked screen stays in memory until the source holding it disappears, often until the process terminates. For this reason, releasing the fix does not rescue the sessions that leaked before it. They are cleaned up only when the app is restarted. #### Summary - This lesson's overrun: majority. The listener that was never unregistered held a screen in every cycle, and five of the eight cycles exceeded the budget of three objects. - A memory leak is the accumulation of retained objects across cycles; it is not seen by looking at a single cycle. - The same leak came out all under a budget of zero objects, majority under three, minority under six and within under eight. If the budget is chosen wider than the leak, the leak is not visible in the report. - A screen registered with a weak reference was not retained once it closed in the model. In reality this depends on when garbage collection runs. - Tying the listener to the lifecycle did not increase the number of retained objects in later cycles; but it did not free the four screens that had leaked earlier. #### Next Step Memory is what is held inside the app. The app also talks to the outside: it sends requests to the network, wakes the device, spends battery. The next lesson, `Network and Battery Profiling`, measures in how many cycles the number of requests sent in a synchronization cycle exceeds the budget, and how sending the requests in batches lowers this number. ### Network and Battery Profiling Source: https://academia.sh/en/courses/testing-performance-and-release/network-and-battery-profiling The number of requests sent in a synchronization cycle determines battery and network use. The lesson measures sending requests one by one, in batches and by polling on the same stream of changes, and shows that polling exceeds the budget even in cycles without changes. Memory was what the app held inside itself. Network and battery are what the app spends toward the outside. A leak fills memory silently; small requests sent often also drain the battery silently. The user does not see this on a single screen; they notice it at the end of the day, by how quickly the battery ran out. So the question of this lesson is this: when synchronization runs, how many requests go out in one cycle, and in how many cycles does this number exceed the budget. The battery cost of a request comes less from the data it sends than from the device waking up to send it. While the device is idle the network hardware waits at low power; a request puts the hardware into a **wakeup** state, and it stays awake for a while after the request ends. Sending three small requests one after another is usually more expensive than sending the same three changes in a single request, because the wakeup is paid once and not three times. Collecting changes and sending them together is called **batching**. Asking the server at regular intervals "is there anything new" is called **polling**. The measure is the course's measure. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Budget` carries the name and limit of a measurement, and `Overrun` counts the samples that exceed the limit and gives one of four overruns. This file is the same in every lesson of the course. #### The Synchronization Cycle Model In the model a cycle is one round of synchronization. In every round a certain number of local changes arrive in the app: a note the user added, a record that was deleted, a setting that changed. We call these numbers the **arrivals**; let the stream of eight rounds be `3, 0, 5, 1, 0, 4, 2, 0`. The synchronization mode determines how many requests these changes turn into. ```kotlin // Sync.kt — counts the requests sent in a synchronization cycle fun requestCount(arrivals: List, mode: String): List = arrivals.map { count -> when (mode) { "single" -> count "batched" -> if (count > 0) 1 else 0 "polling" -> 1 else -> 0 } } ``` `requestCount` returns, for each round, the number of requests sent in that round. In the `single` mode every change is a separate request. In the `batched` mode the changes accumulated during the round go in a single request, and if there is no change no request goes. In the `polling` mode one request goes in every round, whether a change arrives or not. We first measure sending one by one, with a request budget of two. ```kotlin // Main.kt — every change is sent as a separate request fun main() { val arrivals = listOf(3, 0, 5, 1, 0, 4, 2, 0) val samples = requestCount(arrivals, "single") println("requests: $samples") val overrun = Overrun("single", Budget("request", 2), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text requests: [3, 0, 5, 1, 0, 4, 2, 0] single: budget=request<=2 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` Three rounds exceeded: the rounds in which three, five and four changes arrived. The round with two changes was not counted as an overrun, because it is equal to the limit. Since three of the eight rounds exceeded, the result is **minority**. **PD1.** The model maps one change to one request and counts every request with the same weight. In reality requests differ greatly from one another in body size, connection setup and retries. The model counts the requests sent; it does not set up how long a request stays on the network, how much data it carries or how much battery it takes. #### Batching Accumulating the same stream of arrivals and sending it in a single request separates the number of requests from the number of changes. ```kotlin // Main.kt — the same changes are sent in a single request per round fun main() { val arrivals = listOf(3, 0, 5, 1, 0, 4, 2, 0) val samples = requestCount(arrivals, "batched") println("requests: $samples") val overrun = Overrun("batched", Budget("request", 2), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text requests: [1, 0, 1, 1, 0, 1, 1, 0] batched: budget=request<=2 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` No round went past two requests: **within**. The same fifteen changes went with five requests. The difference between the earlier measurement and this one is the number of rounds that exceeded the budget, and batching lowered this number from three to zero. On the platform, batching is done by handing the work over to the system's job scheduler: the app registers the work with conditions that say when it should run, and the system can run the work of many apps in the same wakeup window. ```kotlin // Platform code: conditional periodic synchronization import android.content.Context import androidx.work.Constraints import androidx.work.ExistingPeriodicWorkPolicy import androidx.work.NetworkType import androidx.work.PeriodicWorkRequestBuilder import androidx.work.WorkManager import java.util.concurrent.TimeUnit fun scheduleSync(context: Context) { val constraints = Constraints.Builder() .setRequiredNetworkType(NetworkType.UNMETERED) .setRequiresBatteryNotLow(true) .build() val request = PeriodicWorkRequestBuilder(1, TimeUnit.HOURS) .setConstraints(constraints) .build() WorkManager.getInstance(context) .enqueueUniquePeriodicWork("sync", ExistingPeriodicWorkPolicy.KEEP, request) } ``` This piece is platform code, and here we claim no output, duration or battery figure. The conditions ask for the work to run only on a suitable network and when the battery is not low; the system starts the work at a moment when the conditions are met. The model sets up the rule of this measurement; the model is not the tool itself. How the system really combines jobs is information the model does not show. #### Polling Polling is asking the server whether or not there is a change. If changes arrive often, polling may look reasonable, but it wakes the device even at times when no change arrives. To measure this, we set up a stream in which all the rounds pass quietly: no round brings a change. ```kotlin // Main.kt — on a stream with no changes, three modes are measured with the same budget fun main() { val quiet = List(8) { 0 } val overruns = listOf("single", "batched", "polling").map { mode -> Overrun(mode, Budget("request", 0), requestCount(quiet, mode)) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text single: budget=request<=0 | over=0/8 -> within batched: budget=request<=0 | over=0/8 -> within polling: budget=request<=0 | over=8/8 -> all within: 2 minority: 0 majority: 0 all: 1 ``` With no changes, the `single` and `batched` modes sent no requests at all, while `polling` sent a request in eight of the eight rounds: **all**. The cost of polling does not depend on the number of changes; it depends on time. Even if the user does not use the app, polling makes you pay for a wakeup in every round. **PD2.** In the model polling is exactly one request per round. In reality the polling interval can be lengthened, the server can give the "no change" answer with a very small response, and the system can run the request in the same window as other work. These mitigations are not in the model, so polling coming out all is close to the worst case of a real app. #### Why We Count Requests Battery consumption is not measured directly, because there is no reliable way to measure it in this repository. Instead we count one of the behaviors that affect the battery: the number of requests sent. This choice has a cost. The request count does not stand in for the battery; it is only a proxy related to it. Two requests sometimes come cheaper than one large request, sometimes more expensive. For this reason the lesson gives no number about the battery and mentions it only through the concept of wakeup. The reason the request count is a good measure is that it ties directly to a design decision. Batching or polling is a decision the developer makes, and the request count is the result of this decision. The measurement makes it possible to read the effect of the decision from inside the code. The quiet-stream measurement is kept separate for the same reason. A real app is not used heavily all the time; for most of the night the device is idle and no change reaches the app. What drains the battery in those hours is not the work the user does but the work the app does by itself. Polling is exactly this kind of work. That is why a synchronization design is tested not only on a busy stream but also on a quiet one. Work handed over to the job scheduler also raises a design question: how long the work can be delayed. Saving a note can be delayed a few minutes; a message cannot be delayed. Loosening the conditions raises the system's chance of combining the work, tightening them lowers it. The model does not set up this bargain; it gives only the request count and does not measure delay. Three questions are asked in choosing the synchronization mode. How often do changes arrive, how much delay does the other side accept, and is synchronization needed while the app is closed. With frequent changes the gain from batching grows, because each batched request carries more changes. In a stream that does not accept delay, batching gives way to immediate sending, and the budget is loosened in that case. If synchronization is wanted while the app is closed too, the work is registered as work that can run without the app process. The three modes in the model are the plainest choices matching the answers to these three questions. When interpreting the measurement in production, the state of the device also enters the account. The same request comes cheap when the network hardware is already awake and expensive when it is idle. This difference does not enter the model: the model counts every request as independent. That is why the request count is a sound measure for comparing two designs but insufficient to state the real battery cost of a design. In batching, the length of the window is also a setting. As the window lengthens, each batched request carries more changes and the request count falls; but the delay before the changes reach the other side also grows. In the model the window is one round long and is not changed. In a real app, choosing the window means striking a balance between request count and delay, and only the product's requirement determines this balance. Failure cases also affect the request count. If a failed request is retried, a second wakeup is paid for the same change. A design whose retry interval lengthens limits this cost. The model counts every request as successful and does not include retries; so the request count of a real stream may be larger than what the model gives. Finally, the mode in which the measurement was taken is written in the report. The same stream gives three different numbers under three modes, and someone reading the number alone cannot know which design was measured. Printing the mode name, the budget and the total request count together makes it possible to interpret a line afterward; in the lesson's outputs these three appear in every line. #### Common Misconceptions **"Polling is harmless in apps where changes are frequent."** If changes arrive often, the wasted rounds of polling decrease, but they do not become zero. In the lesson's quiet-stream measurement, polling sent a request in every round even when there was no change. **"Batching is always better."** Batching delays the sending of the changes. In a stream where a change must reach the other side instantly, this delay may be unacceptable. Because the lesson measures only the request count, it does not show the delay; this does not mean batching is free. **"The same overrun means the same saving."** Under a budget of zero, single sending and batched sending gave the same overrun, yet one sent fifteen requests and the other five. The rule does not see this difference; it becomes visible when the total is printed separately. **"The platform combines the work, so there is no need to measure."** The system can bring work together in a suitable window, but how many jobs the app registers and how often they are asked to run is still the app's decision. The system's share of combining does not reduce the registered work itself. #### How to Choose a Budget The request budget is the number of requests allowed per round. Setting the budget to zero demands that no round send a request; this makes every round in which a change arrives an overrun and leaves only the quiet rounds clean. Setting the budget to one accepts a single request per round; this keeps batching within and leaves sending one by one, in this stream, in the majority. In the lesson's first two measurements the budget was two. The value chosen somewhere between them depends on how much delay the app accepts. #### The Bounding Measurement So far we have seen two things: batching lowers the request count, and polling sends requests even on a quiet stream. To see what the rule cannot see, we measure the same stream of arrivals with the three modes, this time with a budget of zero, and print the total request count as well. ```kotlin // Main.kt — the same stream is measured with three modes at a budget of zero and the total requests are printed fun main() { val arrivals = listOf(3, 0, 5, 1, 0, 4, 2, 0) val overruns = listOf("single", "batched", "polling").map { mode -> val samples = requestCount(arrivals, mode) println("$mode total: ${samples.sum()}") Overrun(mode, Budget("request", 0), samples) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text single total: 15 batched total: 5 polling total: 8 single: budget=request<=0 | over=5/8 -> majority batched: budget=request<=0 | over=5/8 -> majority polling: budget=request<=0 | over=8/8 -> all within: 0 minority: 0 majority: 2 all: 1 ``` Under a budget of zero, `single` and `batched` gave the same overrun: both **majority**. Yet one sent fifteen requests and the other five. The rule looks at whether a round exceeded the budget, not at how many requests it sent in excess; so it cannot see, under this budget, the saving of batching, measured as ten requests. The same function gives two overruns under two budgets: with a budget of two `single` was minority, with zero it was majority. To see the saving you either look at the total count or pull the budget to a limit that batching can reach, that is, one request per round. The lesson's first two measurements therefore drew a picture with two budgets: with a budget of two the difference is visible, with zero it disappears. Polling came out `all` on its own in this measurement too, because it sent requests in the three rounds in which no change arrived. Eight requests are more than the five of batching and fewer than the fifteen of sending one by one; the rule saying all does not mean polling sent the most requests, it means it sent a request in every round. **PD3.** The arrival stream in the model is fixed and has eight rounds. In real use changes arrive in clusters, interrupted by long quiet periods, and vary greatly by user. A result gained on one stream must be measured again on another stream before it is carried over. #### Summary - This lesson's overrun: within. With batching, the same changes turned into at most one request per round, and no round exceeded the two-request budget. - A request spends battery because it wakes the device; batching separates the number of wakeups from the number of changes. - Sending one by one exceeded the budget in three of the eight rounds on the same stream: minority. - Polling sent a request in every round even on a stream with no changes, and came out all at a budget of zero. - The rule counts whether a round exceeded the budget. At a budget of zero, fifteen and five requests gave the same overrun; to see the saving you must also look at the total count. #### Next Step Every number we have measured so far was measured once, and that one number was accepted. Yet measurement is noisy: running the same work twice can give different numbers. The next lesson, `Benchmarking`, covers how a regression is told apart in repeated measurements and why warm-up iterations do not take part in the measurement. ### Benchmarking Source: https://academia.sh/en/courses/testing-performance-and-release/benchmarking Comparing two versions of a piece of work requires repeating the measurement. The lesson measures how warm-up iterations distort the measurement, how a regression looks once they are dropped, and how the rule misses it when the budget is chosen wider than the regression. In the previous lesson we measured the request count once on a stream and accepted the result as certain. A real measurement is not a single number. When you run the same work back to back, the first runs take a different time from the later ones: classes are loaded, code is compiled, caches fill. We call these first iterations **warm-up**. When a measurement includes warm-up, it measures not the steady state of the work but the road travelled until the steady state is reached. The question of this lesson is this: how do we tell whether a change slowed the work down, that is, whether there is a **regression**, while warm-up distorts the measurement. The answer has two parts: separating warm-up from the measurement and choosing the budget narrow enough to catch the regression. The measure is the course's measure again: in how many of its iterations the work exceeds the budget. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Budget` carries the name and limit of a measurement, and `Overrun` counts the samples that exceed the limit and gives one of four overruns. This file is the same in every lesson of the course. #### The Iterative Measurement Model In the model the cost of a piece of work is its number of work steps. The work has two versions: `old` and `new`. After a change, the new version does three more steps in every run; this is our regression. Warm-up is the extra steps added to the first iterations, and it shrinks as the iterations progress. ```kotlin // Bench.kt — iterative measurement of a piece of work; the first iterations carry a warm-up share fun workOf(version: String, size: Int): Int = when (version) { "old" -> size * 2 "new" -> size * 2 + 3 else -> size } fun iterations(version: String, size: Int, runs: Int, warmup: Int): List = (0 until runs).map { run -> workOf(version, size) + if (run < warmup) (warmup - run) * 4 else 0 } fun dropWarmup(samples: List, warmup: Int): List = samples.drop(warmup) ``` `workOf` gives the steady-state cost of a version. `iterations` repeats this cost `runs` times and adds a decreasing warm-up share to the first `warmup` iterations. `dropWarmup` removes the warm-up iterations from the measurement. For a piece of work of five units, the steady cost of the `old` version is ten and that of the `new` version is thirteen. We first measure the old version with eight iterations, warm-up included. Let the budget be twelve steps. ```kotlin // Main.kt — the old version is measured with the warm-up iterations included fun main() { val samples = iterations("old", 5, 8, 3) println("old: $samples") val overrun = Overrun("old", Budget("step", 12), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text old: [22, 18, 14, 10, 10, 10, 10, 10] old: budget=step<=12 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The steady cost of the version is below the twelve-step budget, but the first three iterations exceeded it because of warm-up: 3/8, **minority**. The old version does work that fits the budget; the measurement wrote three overruns against it. This is warm-up distorting the measurement. **PD1.** In the model the warm-up share decreases linearly and the number of iterations is known exactly. In reality, when warm-up ends depends on the work, the device and the state of the system at that moment; it is not possible to say after which iteration the steady state is reached without measuring. The model counts how many work steps each iteration does; it does not set up how long an iteration takes. #### Dropping Warm-Up and Seeing the Regression To compare, we measure both versions with warm-up dropped and hold them to the same budget. ```kotlin // Main.kt — the two versions are compared with warm-up dropped, under the same budget fun main() { val warmup = 3 val oldSamples = dropWarmup(iterations("old", 5, 8, warmup), warmup) val newSamples = dropWarmup(iterations("new", 5, 8, warmup), warmup) println("old: $oldSamples") println("new: $newSamples") val overruns = listOf( Overrun("old", Budget("step", 12), oldSamples), Overrun("new", Budget("step", 12), newSamples), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text old: [10, 10, 10, 10, 10] new: [13, 13, 13, 13, 13] old: budget=step<=12 | over=0/5 -> within new: budget=step<=12 | over=5/5 -> all within: 1 minority: 0 majority: 0 all: 1 ``` With warm-up dropped the measurement became clean: the old version is **within**, the new version is **all**. The three overruns written against the old version in the previous measurement were warm-up's share; in this measurement the new version exceeding in five of five shows a real regression. The difference in the steady state is three steps, and the budget was chosen at twelve steps so that it passes right through this difference. On the platform these two distinctions are set up with a rule object that repeats the measurement. In the piece below, the number of warm-up iterations and the number of iterations to measure are given explicitly. ```kotlin // Platform code: repeated benchmark import androidx.benchmark.macro.CompilationMode import androidx.benchmark.macro.StartupMode import androidx.benchmark.macro.StartupTimingMetric import androidx.benchmark.macro.junit4.MacrobenchmarkRule import androidx.test.ext.junit.runners.AndroidJUnit4 import org.junit.Rule import org.junit.Test import org.junit.runner.RunWith @RunWith(AndroidJUnit4::class) class StartupBenchmark { @get:Rule val benchmarkRule = MacrobenchmarkRule() @Test fun coldStartup() = benchmarkRule.measureRepeated( packageName = "com.example.app", metrics = listOf(StartupTimingMetric()), iterations = 5, startupMode = StartupMode.COLD, compilationMode = CompilationMode.Partial(warmupIterations = 3), ) { pressHome() startActivityAndWait() } } ``` This piece is platform code, and here we claim no output, duration or ratio. The rule object repeats the measurement five times with `measureRepeated` and runs the warm-up iterations separately, in the number specified; which durations come out depends on the device and the app. The model sets up the rule of this measurement; the model is not the tool itself. When the tool counts warm-up as finished is information the model does not show. #### The Iteration Count Changes the Measurement If warm-up is not dropped, the overrun count also depends on the number of iterations, because the warm-up share is fixed and its weight shrinks as the iteration count grows. We measure the same old version, warm-up included, with four, eight and sixteen iterations. ```kotlin // Main.kt — the same version is measured, warm-up included, with three different iteration counts fun main() { val overruns = listOf(4, 8, 16).map { runs -> Overrun("runs-$runs", Budget("step", 12), iterations("old", 5, runs, 3)) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text runs-4: budget=step<=12 | over=3/4 -> majority runs-8: budget=step<=12 | over=3/8 -> minority runs-16: budget=step<=12 | over=3/16 -> minority within: 0 minority: 2 majority: 1 all: 0 ``` The version is the same, the warm-up is the same, the number of overrunning iterations is three. Only the iteration count changed, and the overrun moved from majority to minority. For this reason, in a measurement that includes warm-up the iterations decide the result; in a measurement that drops warm-up the steady state decides it. For a benchmark to be reliable, both are fixed together: how many iterations will be measured and how many will be counted as warm-up and dropped. #### Micro and Macro Benchmarks Benchmarking is done at two levels. At the micro level a single function or a small piece of code is run repeatedly in an isolated environment; the aim is to separate the cost of that piece from other work. At the macro level a flow of the app, for example startup or the opening of a screen, is measured end to end; the aim is to see the result the user experiences. The two do not replace each other. The micro measurement may look improved while the macro measurement stays the same, because the measured piece is a small part of the flow. The macro measurement may show a regression while the micro measurement comes out clean, because the problem lies in the interaction of the pieces. In this lesson the model measures a single unit of work; a real project uses both levels. #### The Baseline A reference point is needed to catch a regression. This point is the measurement of the accepted version of the work, and it is kept as the **baseline**. The new measurement is compared with this value; if the difference exceeds a certain limit, it counts as a regression. In the lesson's model the baseline is the steady-state cost of the old version. The baseline is itself a measurement and is affected by warm-up and noise. Taking the baseline with warm-up included spoils all later comparisons. For this reason the baseline is computed only from steady iterations and recorded together with the conditions in which it was measured. #### Why Warm-Up Exists Warm-up means the first runs of the work are more expensive than the later ones. In the first run the code may not be compiled yet, the caches are empty, the resources used are opened for the first time. After a few iterations these costs are paid and the work reaches its steady state. In the model, warm-up is a decreasing share added to the first iterations; the shape of real warm-up varies with the work. Dropping warm-up from the measurement means choosing to measure the steady state. This choice is right, because to catch a regression the work itself must be measured, not the luck of the first run. But a measurement that drops warm-up does not see the cost the user meets first. For this reason a measurement of startup and a measurement of regression are kept separate: in the first, warm-up is the subject of the measurement; in the second, it is noise. #### The Setup Must Stay Fixed In a benchmark the two versions are measured not side by side but one after the other in the same setup. If the setup changes, for example if a measurement is taken on another device or in another system state, the difference may come not from the regression but from the setup. In this lesson the setup is the model itself and is the same in every measurement; in a real project the same assurance is provided only by fixing the device, the system state and the iteration count. This fixedness also determines how measurement results are stored. When a baseline is recorded, which version it was taken on, with how many iterations, and whether warm-up was dropped or not is written beside it. Without this information two numbers cannot be compared, because the lesson showed that the same version can give two different overruns depending on the iteration count. Where the budget comes from must also be asked. In this lesson twelve steps was placed between ten, the steady cost of the old version, and thirteen, the cost of the new version, and decided in advance which version would stay within it. In a real project the budget is set by adding a small margin to the measurement of the accepted version; the margin must be larger than the width of the noise and smaller than the size of the regression. If the margin is too large the rule cannot see the regression; if it is too small it raises a false alarm in every measurement. When reporting results, the distribution of the numbers is given instead of a single number: the best iteration, the worst iteration and the middle one. In this lesson the number of work steps per iteration is the samples themselves and is printed under every measurement. The single line the rule writes summarizes the overrun count; when the samples are read, one sees where warm-up ended and where the regression began. The decision made as a result of a benchmark must also be written down. If a change introduced a regression there are three ways: revert the change, accept the regression and raise the baseline, or compensate for the regression somewhere else. If the choice and its reason are not recorded, a later measurement cannot explain why the new baseline is what it is. In this lesson the three-step difference was clearly a regression; the product decision lies between accepting it or not, and the rule does not give that. The order of the iterations also carries information. Because the warm-up share is concentrated in the first iterations, the beginning of the sample series measures something different from its end. Shuffling the series and taking its average mixes warm-up into the steady state and combines two separate phenomena in a single number. Samples printed in order show, without eyeballing, the iteration where warm-up ended, and are the basis for choosing how many iterations to drop. #### Common Misconceptions **"Measuring once is enough."** A single measurement cannot separate warm-up from noise. The lesson showed this by the same version giving different overruns depending on the iteration count. **"Dropping warm-up makes the measurement look nicer."** When warm-up is dropped, what the measurement targets must be clear. The user pays the warm-up cost when opening the app for the first time; a measurement that wants to measure this cost does not drop warm-up, it takes it as a separate measurement. **"A small difference is a regression."** While noise exists, a small difference may come from something other than a regression. In this lesson the model is noiseless, so the three-step difference is certain. In a real measurement, iterations are multiplied to be sure the difference is larger than the noise. **"The rule finds the regression."** The rule only counts the samples that exceed the budget. If the budget is above the regression, the rule cannot see it; the lesson's bounding measurement showed this. #### The Bounding Measurement The most important limit of the rule is this: if the budget is chosen wider than the regression, the rule cannot see the regression. We show it with two trials. In the first, warm-up is included and the budget is twenty steps. In the second, warm-up is dropped and the budget is thirteen steps. ```kotlin // Main.kt — under a wide budget the regression escapes the rule fun main() { val warmup = 3 val oldAll = iterations("old", 5, 8, warmup) val newAll = iterations("new", 5, 8, warmup) val oldSteady = dropWarmup(oldAll, warmup) val newSteady = dropWarmup(newAll, warmup) val overruns = listOf( Overrun("old-warm", Budget("step", 20), oldAll), Overrun("new-warm", Budget("step", 20), newAll), Overrun("old-steady", Budget("step", 13), oldSteady), Overrun("new-steady", Budget("step", 13), newSteady), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text old-warm: budget=step<=20 | over=1/8 -> minority new-warm: budget=step<=20 | over=2/8 -> minority old-steady: budget=step<=13 | over=0/5 -> within new-steady: budget=step<=13 | over=0/5 -> within within: 2 minority: 2 majority: 0 all: 0 ``` Under the twenty-step budget both versions are **minority**; under the thirteen-step budget both are **within**. Although the new version has slowed by three steps, the rule wrote the same overrun to both versions in both cases. In the measurement with warm-up, the new version's overrunning iteration count is two and the old version's is one: the difference is one iteration, and from the sample count alone it cannot be told whether it came from the regression or from warm-up. In the measurement with warm-up dropped, thirteen steps is exactly the new version's limit, and a sample equal to the limit does not count as an overrun. The rule sees the number of overruns; it does not see their size, that is, the three-step difference. The same function can give two overruns under two budgets; unless the budget is set at a value below the regression, the regression stays inside the rule. **PD2.** The regression in the model is the same three steps in every iteration. Real regressions do not arrive the same in every iteration; some appear only on certain inputs or certain devices. It cannot be concluded that a regression not visible in one measurement setup will not be visible in another. #### Summary - This lesson's overrun: minority. The steady cost of the old version did not exceed the budget, but the first three warm-up iterations did, and three of the eight iterations went past the budget. - Warm-up iterations measure not the steady state of the work but the road to reaching it. For comparison, warm-up is dropped from the measurement. - Once warm-up was dropped, the old version came out within and the new version all; the three-step regression became visible. - In a measurement that includes warm-up, the same version came out majority or minority depending on the iteration count. - If the budget is chosen wider than the regression, the rule writes the same overrun to both versions: it sees the number of overruns, not their size. #### Next Step This lesson fixed what a measurement measures. What remains is how the measured thing enters the package to be published. The next lesson, `Code Shrinking and Obfuscation`, measures which classes keep rules hold alive while unused classes are stripped from the app package, and how a broad rule defeats shrinking. ### Code Shrinking and Obfuscation Source: https://academia.sh/en/courses/testing-performance-and-release/code-shrinking-and-obfuscation Unused classes are stripped from the published app package and the names of the remaining classes are shortened. The lesson measures in how many modules the remaining class count exceeds the budget after shrinking, shows that obfuscation does not change this count, and shows that a broad keep rule defeats shrinking. In the previous lesson we built the way to measure whether a change slowed the work down. This lesson looks at how the measured thing enters the package that reaches the user. A published app does not have to carry every class written during development. If the app calls only a few of a library's hundreds of classes, the classes that are never called only take up space when they enter the package; they enlarge the download and increase the loading work. Reducing the code that enters the package is two separate operations. **Shrinking** keeps classes and members that cannot be reached from the app's entry points out of the package. **Obfuscation** replaces the names of the remaining classes and members with short, meaningless names. Both run at build time, in the step that produces the release package; they do not touch the source code written during development. Before a shrinker removes code, it analyzes whether that code can be reached, and the analysis cannot see code reached by reflection. For this reason the developer declares the classes that must not be discarded with a **keep rule**. The measure is the course's measure again: in how many modules the number of classes remaining in a module after shrinking exceeds the budget. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Budget` carries the name and limit of a measurement, and `Overrun` counts the samples that exceed the limit and gives one of four overruns. This file is the same in every lesson of the course. #### The Reachability Analysis Model In the model a module consists of class numbers and the call edges between classes. The app's entry point is class number zero; starting from that point the shrinker follows the call edges and keeps the classes it reaches. It discards the classes it cannot reach. A class reached by reflection carries no call edge; the shrinker does not see it, and it stays only thanks to a keep rule. ```kotlin // Shrink.kt — classes reachable from the root stay, the others are discarded class Module(val name: String, val edges: Map>, val total: Int) fun buildModule(name: String, total: Int, used: Int): Module = Module(name, (0 until used - 1).associateWith { listOf(it + 1) }, total) fun sampleModules(): List { val sizes = listOf(12 to 4, 20 to 6, 14 to 3, 30 to 9, 16 to 5, 11 to 4, 24 to 7, 14 to 5) return sizes.mapIndexed { index, (total, used) -> buildModule("module-${index + 1}", total, used) } } fun reachableFrom(module: Module, root: Int): Set { val seen = mutableSetOf(root) val queue = ArrayDeque(listOf(root)) while (queue.isNotEmpty()) { val current = queue.removeFirst() for (next in module.edges[current].orEmpty()) { if (seen.add(next)) queue.addLast(next) } } return seen } fun keptClasses(module: Module, rule: String): Int = when (rule) { "shrink", "obfuscate" -> reachableFrom(module, 0).size "narrow" -> (reachableFrom(module, 0) + (module.total - 1)).size else -> module.total } fun mappingLines(module: Module): List = reachableFrom(module, 0).sorted().map { "C$it -> ${'a' + it}" } ``` `sampleModules` builds eight modules; each module carries `total` classes and `used` of them are reachable from the entry point. `reachableFrom` finds the reachable classes. `keptClasses` gives the number of classes remaining in a module according to the rule: `shrink` keeps only the reachable ones, `obfuscate` gives the same count, `narrow` adds to the reachable ones the single class reached by reflection, and the other rules keep all the classes of the module. `mappingLines` returns the correspondences of the names obfuscation writes. We first leave shrinking on and measure the remaining class count with a budget of ten classes. ```kotlin // Main.kt — the class count remaining after shrinking is measured per module fun main() { val modules = sampleModules() val samples = modules.map { keptClasses(it, "shrink") } println("kept: $samples") val overrun = Overrun("shrink", Budget("class", 10), samples) overrun.line() countOverruns(listOf(overrun)) } ``` ```text kept: [4, 6, 3, 9, 5, 4, 7, 5] shrink: budget=class<=10 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` In no module did the remaining class count reach ten: **within**. The rule counts not whether this code goes to production but in how many samples the work remaining after shrinking exceeds the budget. **PD1.** In the model the class count stands in for a package size and every class has the same weight. The size of a real class varies from a few lines to thousands of lines; the model counts classes, does not set up how much space each takes, and does not state by what ratio the package shrank. Whether the number remaining after shrinking is right against the budget is decided by the product, not by the model. #### With Shrinking Off To see the gain from shrinking, we measure the same eight modules without shrinking: in that case every module carries all its classes. ```kotlin // Main.kt — the same budget with shrinking off and on fun main() { val modules = sampleModules() val closed = modules.map { keptClasses(it, "closed") } val shrunk = modules.map { keptClasses(it, "shrink") } println("closed: $closed") println("shrink: $shrunk") val overruns = listOf( Overrun("closed", Budget("class", 10), closed), Overrun("shrink", Budget("class", 10), shrunk), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text closed: [12, 20, 14, 30, 16, 11, 24, 14] shrink: [4, 6, 3, 9, 5, 4, 7, 5] closed: budget=class<=10 | over=8/8 -> all shrink: budget=class<=10 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 1 ``` With shrinking off, every module carried a class count near ten or past it: **all**. With shrinking on, all eight stayed within the budget. The difference lies not in the code itself changing but in the code that enters the package decreasing. On the platform this decision is made in the build script. The piece below turns shrinking on for the release build and attaches the rule file. ```kotlin // Platform code: turning on shrinking in the release build android { buildTypes { release { isMinifyEnabled = true proguardFiles( getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro", ) } } } ``` This piece is platform code, and here we claim no output, size or duration. The model sets up the rule of this measurement; the model is not the tool itself. Which classes the shrinker counts as reachable in a real app is information the model does not show, and it is read from the reports produced at the end of the build. #### Obfuscation Does Not Change the Count Obfuscation shortens the names of classes. It does not change the remaining class count, because it discards no class; it only changes its name. In addition, the correspondences between the original names and the short names are written to a **mapping file**. Without this file the short names in crash traces cannot be translated back to the original names. ```kotlin // Main.kt — obfuscation changes the names, not the class count fun main() { val modules = sampleModules() println("mapping:") mappingLines(modules[0]).forEach { println(" $it") } val overruns = listOf("shrink", "obfuscate").map { rule -> Overrun(rule, Budget("class", 10), modules.map { keptClasses(it, rule) }) } overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text mapping: C0 -> a C1 -> b C2 -> c C3 -> d shrink: budget=class<=10 | over=0/8 -> within obfuscate: budget=class<=10 | over=0/8 -> within within: 2 minority: 0 majority: 0 all: 0 ``` The four classes of the first module were shortened from `a` to `d`, and the two lines gave the same overrun. Obfuscation alone does not affect the budget; it serves a separate purpose, that is, making the names unreadable. Losing the mapping file, on the other hand, leads to the short names being impossible to resolve. Because this lesson's measurement looks only at the class count, it does not see the presence or absence of the mapping file. **PD2.** In the model obfuscation replaces a name with a single letter and no collision occurs. A real obfuscator shortens many classes in nested namespaces, keeps some names (for example names looked up by reflection) and regenerates the mapping file in every build. The mapping file of an earlier build does not resolve the names of a later build. #### What Shrinking Solves The aim of shrinking is to zero the cost of code that does not enter the package. If code does not enter the package, it is not downloaded, not loaded, and not scanned during loading. But this gain holds only for unreachable code. A class reachable from an entry point stays in the package even if the user never calls it, because the shrinker analyzes not the use at run time but the structural reachability of the code. For this reason the number remaining after shrinking is not a lower bound but an upper estimate. In the model reachability is a chain: each class calls the next. In real apps the call graph is far more branched, and the number of classes the shrinker leaves varies with the shape of the graph. A keep rule has a cost, and this cost escapes the eye of the person writing it. Every rule tells the shrinker "keep this" instead of "discard this". Broadening a rule is easy, because the error shows only at run time: a missing rule crashes the app, an excess rule only enlarges the package. Because of this asymmetry rules widen over time and the gain from shrinking quietly erodes. The numeric measurement in this lesson makes exactly this erosion visible with a budget. Writing a narrow rule requires knowing which class is reached by reflection and why. This information is most often in a library's documentation, or in your own code on the line where a string of characters is turned into a class at run time. Writing the reason next to the rule leaves a trace with which the next person safely narrows or removes the rule. Testing the release build is a separate job as well. Since shrinking is usually turned on only in the release build, an app that runs during development can crash after shrinking. For this reason the shrunk build is run end to end at least once, and the paths reached by reflection are walked in particular. Catching the crash beforehand comes cheaper than writing the keep rule along with its reason. The result of shrinking changes as the app is updated. Adding a new library increases the reachable class count, removing an unused feature decreases it. For this reason the remaining class count is not measured once and forgotten; in every release build it is counted again with the same budget, and a module that exceeds the budget is treated as a regression that raises the question of which change enlarged it. The eight modules of this lesson are a snapshot of a single moment of this monitoring. Keeping the mapping file is a process question. The file is produced together with the build and must match the published package; the file of the wrong build translates the short names to wrong original names and misleads the crash investigation. For this reason the file is archived together with the version number at the moment the package is published. #### Common Misconceptions **"Obfuscation shrinks the package."** Obfuscation shortens names; this may save space when names are long, but it does not reduce the class count. In the lesson's obfuscation measurement the two lines gave the same overrun. **"Obfuscation makes code secure."** Obfuscation makes code harder to read, but it provides no protection. Short names do not prevent the code from being decoded, they only slow it down. Information that must be kept secret should never enter the package. **"Writing a keep rule solves the problem."** A keep rule saves the class reached by reflection, but the breadth of the rule determines the cost. A broad rule keeps the whole package and erases the gain from shrinking. The lesson's bounding measurement showed this. **"The mapping file is optional."** Without the mapping file, the short names in crash traces cannot be translated to the original names. The mapping file of every release build is stored together with that build. #### The Budget and What the Rule Sees The class-count budget states the most classes a module may carry after shrinking. There is no single right way to compute the budget: some projects use the average of the previous version, some use a distribution working backward from the package's total limit. The rule does not know where the budget comes from; it only applies it to the samples. #### The Bounding Measurement The shrinker cannot see code reached by reflection. If there is a single class reached by reflection, that class must be kept. We can do this in two ways. A narrow rule keeps only that class. A broad rule keeps everything in that class's package. On the platform their counterparts in the rule file are as follows. ```proguard # narrow: only the class reached by reflection is kept -keep class com.example.plugin.PluginEntry { *; } # broad: every class in the package is kept -keep class com.example.plugin.** { *; } ``` This rule file is not run and has no output; the two lines give two different answers to the same reflection question. We measure the same two answers in the model. ```kotlin // Main.kt — a narrow and a broad keep rule are measured under the same budget fun main() { val modules = sampleModules() val narrow = modules.map { keptClasses(it, "narrow") } val broad = modules.map { keptClasses(it, "broad") } println("narrow: $narrow") println("broad: $broad") val overruns = listOf( Overrun("narrow", Budget("class", 10), narrow), Overrun("broad", Budget("class", 10), broad), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text narrow: [5, 7, 4, 10, 6, 5, 8, 6] broad: [12, 20, 14, 30, 16, 11, 24, 14] narrow: budget=class<=10 | over=0/8 -> within broad: budget=class<=10 | over=8/8 -> all within: 1 minority: 0 majority: 0 all: 1 ``` The narrow rule added to the reachable classes only the class reached by reflection and stayed **within**; the fourth module is exactly at ten and, being equal to the limit, was not counted as an overrun. The broad rule, on the other hand, also kept the unused classes in every module and gave the same values as the numbers measured with shrinking off: **all**. How the broad rule defeats shrinking is read from these two lines: writing the rule did not turn the shrinker off, but it produced the same count as the state in which it is off. The rule sees how many samples overrun, not why. From a line that came out all, one cannot read why the remaining classes remained; for that, which keep rule holds which class is looked up in the reports the shrinker writes at the end of the build. Obfuscation fixes nothing here either: even if names are shortened, the count stays the same. **PD3.** In the model the broad rule keeps every class in the package, and the package is the same as the module. A real keep rule may keep only certain members or only certain name patterns; the two rules written in this lesson show the narrowest and the broadest ends. The result of rules that fall between them cannot be derived from the model without measuring. #### Summary - This lesson's overrun: within. With shrinking on, the remaining class count did not exceed the ten-class budget in any of the eight modules. - Shrinking discards classes that cannot be reached from the entry point. Obfuscation only shortens names; the remaining class count does not change, and the mapping file is kept to translate the short names back. - With shrinking off, the same eight modules exceeded the budget in all. The difference came not from the code changing but from the code entering the package decreasing. - The narrow keep rule stayed within. The broad keep rule covering the package reached by reflection also kept the unused classes in every module, and the result came out all. - The rule sees how many samples overrun, not why. Which rule keeps the remaining classes is read from the shrinker's reports. #### Next Step After the code is shrunk and obfuscated, the app turns into a package, and this package has more than one format: the format that carries everything in a single file and the format that produces device-specific parts. The next lesson, `App Package Formats`, measures what these formats distribute and how and in how many device types the number of parts downloaded per device exceeds the budget. ## Distribution App package formats optimized per device, the version code and name scheme, automated building and signing, internal, closed and open testing tracks, staged rollout and halting, store review, and post-release monitoring. ### App Package Formats Source: https://academia.sh/en/courses/testing-performance-and-release/app-package-formats The app bundle, the APK and the split APK decide which files reach a device. The lesson compares the number of files that land on a configuration with a budget and shows that the universal package overruns it on every configuration. The previous lesson measured code shrinking and obfuscation: how many classes remain in a module after a keep rule. That work happened inside the build. This lesson looks at the build's output: when the finished package reaches a device, how many files does it carry? The course's question has not changed: **in how many samples does this measurement exceed its budget?** The measurement is a number, the budget is the limit on that number, and a sample is the same measurement taken on a different input. The samples that exceed the limit strictly are counted, and the count gives one of four answers: **within**, **minority**, **majority**, **all**. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that exceed the budget. If none exceed it, the answer is within; if all exceed it, the answer is all; if fewer than half exceed it, the answer is minority; every other case is majority, and exactly half is majority. The rule sees how many samples overrun, not by how much or why. The course's opening lesson classified four measurements with this rule and saw each answer once; this lesson only refers to that distribution. #### Three Formats, One Release In Android, three different things are called a package, and they are often confused. The APK is the file that is installed on a device. The app bundle is the publishing format uploaded to the app store; it is not installed on a device itself. From this bundle the store produces a split APK for each device configuration: a small set of files containing the native libraries that match the device's processor architecture (ABI), the resources that match its screen density, and the shared code. The universal APK is the format in which this selection is never made: a single set of files carries the libraries of every ABI and the resources of every density, and the same set lands on every device. Splitting by configuration has three axes: language, screen density and ABI. In the build system, the module build script turns them on or off. The script fragment looks like this. ```kotlin // Platform code: module build script android { bundle { language { enableSplit = true } density { enableSplit = true } abi { enableSplit = true } } } ``` This fragment is platform code; we claim no output, duration or size here. The model sets up the rule of this measurement; the model is not the tool itself. In the model, a package is a list of files and a device configuration is a pair of an ABI and a density. Each of the three lines in the script answers a separate question. With the `abi` axis on, native libraries are split into separate files by processor architecture; the device takes the library that matches its architecture and leaves the others. With the `density` axis on, the resources that depend on screen density are separated in the same way. With the `language` axis on, language resources are separated and the device asks only for its own languages. If one of the axes is off, that axis's files stay in the shared part without being separated and land on every device. The lesson does not run this script; in the model it turns the distinction the script expresses into a `wants` policy and counts the result of the policy. #### File List and Configuration The helper file below builds the package. Each `PackageFile` is tied either to one ABI, to one density, or to neither. `filesForConfig` counts how many files land on a configuration; the `wants` policy decides which files land. ```kotlin // PackageModel.kt — models the package files and the files that land on a configuration class PackageFile(val name: String, val abi: String? = null, val density: String? = null) class DeviceConfig(val label: String, val abi: String, val density: String) fun buildPackage(abis: List, densities: List): List = listOf(PackageFile("code.dex"), PackageFile("manifest.xml")) + abis.map { PackageFile("library-$it.so", abi = it) } + densities.map { PackageFile("resource-$it.bin", density = it) } fun samplePackage(): List = buildPackage(listOf("arm64", "arm32", "x86"), listOf("mdpi", "hdpi", "xhdpi", "xxhdpi")) fun sampleConfigs(): List = listOf( DeviceConfig("narrow-phone", "arm32", "mdpi"), DeviceConfig("mid-phone", "arm64", "hdpi"), DeviceConfig("wide-phone", "arm64", "xhdpi"), DeviceConfig("tablet", "arm64", "xxhdpi"), DeviceConfig("old-device", "arm32", "hdpi"), DeviceConfig("emulator", "x86", "xhdpi"), ) fun filesForConfig( files: List, config: DeviceConfig, wants: (PackageFile, DeviceConfig) -> Boolean, ): Int = files.count { wants(it, config) } val everything: (PackageFile, DeviceConfig) -> Boolean = { _, _ -> true } val matching: (PackageFile, DeviceConfig) -> Boolean = { file, config -> (file.abi == null || file.abi == config.abi) && (file.density == null || file.density == config.density) } ``` The package is nine files: two shared files, three ABI libraries and four density resources. The configuration list gives six samples. The `everything` policy ignores the configuration; this is the definition of the universal package. The `matching` policy asks only for the files that match the configuration's ABI and density. #### The Universal Package We set the budget at five files. This number is an input to the model: it is not a platform threshold or a store rule, but the limit the lesson chose to ask the question. We measure how many files the universal package delivers to the six configurations. ```kotlin // Main.kt — the universal package delivers every file to every configuration fun main() { val files = samplePackage() val samples = sampleConfigs().map { filesForConfig(files, it, everything) } println("package files: ${files.size}") println("delivered files: $samples") Overrun("universal-package", Budget("file", 5), samples).line() } ``` ```text package files: 9 delivered files: [9, 9, 9, 9, 9, 9] universal-package: budget=file<=5 | over=6/6 -> all ``` All six samples exceeded the budget: the answer is **all**. We take this result not from the rule being generous but from the policy ignoring the configuration. An old phone with a narrow screen and a wide tablet both received nine files; this is why the `samples` list contains no number other than nine. Most of the density resources will never be used on that device. **DI1.** The model counts a file as one unit of work and weighs every file equally. In reality a density resource file can be of a very different size than a shared code file; the model does not set up size, compression or download conditions. The file count is a unit of work standing in for size and is not a byte measurement. #### The Budget Is an Input to the Model The same function giving two different overruns under two budgets is not a contradiction; it is the nature of the measurement. We raise the budget to nine: the universal package delivers at most nine files, so no sample strictly exceeds this limit. For the split package, we lower the budget to three: every configuration that receives four files exceeds it this time. ```kotlin // Main.kt — the same measurement gives two overruns under two budgets fun main() { val files = samplePackage() val configs = sampleConfigs() val loose = Overrun("universal-package", Budget("file", 9), configs.map { filesForConfig(files, it, everything) }) val strict = Overrun("split-package", Budget("file", 3), configs.map { filesForConfig(files, it, matching) }) loose.line() strict.line() countOverruns(listOf(loose, strict)) } ``` ```text universal-package: budget=file<=9 | over=0/6 -> within split-package: budget=file<=3 | over=6/6 -> all within: 1 minority: 0 majority: 0 all: 1 ``` A sample equal to the limit is not counted as an overrun: the universal package, which delivers nine files, stayed within the budget of nine. The split package delivered four files and came out as all against the budget of three. These two results do not say that a package is good or bad; they only show that the number was compared with the budget. This is why we set the budget at the start of the lesson and state its reason: the file count represents the download work of a configuration, not a package format. #### Half Splitting It is not mandatory to split all three axes together. A package that splits only by ABI still delivers all of the density resources to every device. In the model, this is a policy that matches the ABI but ignores the density. ```kotlin // Main.kt — a package split only by ABI still delivers the density files fun main() { val files = samplePackage() val abiOnly: (PackageFile, DeviceConfig) -> Boolean = { file, config -> file.abi == null || file.abi == config.abi } val samples = sampleConfigs().map { filesForConfig(files, it, abiOnly) } println("delivered files: $samples") Overrun("abi-only", Budget("file", 5), samples).line() } ``` ```text delivered files: [7, 7, 7, 7, 7, 7] abi-only: budget=file<=5 | over=6/6 -> all ``` Every configuration received seven files: two shared files, a single ABI library and four density resources. Partial splitting reduced nine to seven, but the overrun did not change. The rule does not see that nine is worse than seven; it only sees that both are greater than five. #### The Language Axis The third axis is language, and as we will note in DI2, it is absent from the main model. We imitate it in a separate block, by file name: we add three language files to the package and give each device one or two languages. With the language axis on, a device receives only the files of its own languages; the ABI and density rule stays the same in the `matching` policy. ```kotlin // Main.kt — with the language axis on, a device with more than one language receives one more file fun main() { val languages = listOf("tr", "en", "de") val files = samplePackage() + languages.map { PackageFile("lang-$it.bin") } val deviceLanguages = listOf( listOf("tr"), listOf("tr", "en"), listOf("en"), listOf("de", "en"), listOf("tr"), listOf("en"), ) val samples = sampleConfigs().mapIndexed { index, config -> files.count { file -> if (file.name.startsWith("lang-")) file.name.removePrefix("lang-").removeSuffix(".bin") in deviceLanguages[index] else matching(file, config) } } println("delivered files: $samples") Overrun("language-axis", Budget("file", 5), samples).line() } ``` ```text delivered files: [5, 6, 5, 6, 5, 5] language-axis: budget=file<=5 | over=2/6 -> minority ``` Four files come from the shared part, the ABI library and the density resource; one language adds one file, two languages add two files. The four single-language devices stayed at five files and did not exceed the budget. The two devices with two languages rose to six and exceeded it. Because two of the six samples overrun, the answer is **minority**: the count stayed below half. This result shows something interesting. Splitting the package fully does not reduce the file count to a single number independent of the device; the device's own properties, here how many languages it uses, move the number. Because the limit is set at five, even a single extra language changes the answer. **DI3.** The language files are parsed from their names (such as `lang-tr.bin`) and each language is a single file. In reality a language's resources can be spread over more than one file; in the model, a language is a file. #### Why the Universal Package Exists The universal package is not a wrong choice; it is a convenience that comes with a different cost. Because it is a single file, it can be handed as is to a teammate, a test device or a distribution channel outside the store, and the question of which configuration will be installed never arises. The store producing split APKs, on the other hand, depends on the store having this configuration information. That is why the two formats live together within the same release: one is for easy hand-to-hand transfer, the other for device-specific download. The question that decides the choice in practice is this: who will pick the package, and for which configuration? If the answer is "the store, per device," the split package is the natural one. If the answer is "no one, it will be installed as it is," the universal package is the only way. The measurement puts the cost of these two cases side by side in the same unit, the file count. #### Common Misconceptions **"The app bundle is installed on the device."** What is installed is the APK. The app bundle is uploaded to the store and the store converts it into split APKs according to the configuration. They are not the same file and they are not produced in the same step. **"A split package always delivers fewer files."** It depends on which of the split axes are on. In the half-splitting example seven files landed, in the full split four. If only the ABI axis is left on, all of the density resources still land. **"A split package delivers the same number of files to every device."** For the ABI and density resources in a fully split package this is true, because every device has one ABI and one density. It stops being true when the language axis is turned on: a device with two languages received six files, a device with one language five. As the number of values a device has grows, the number of delivered files grows too. **"The universal package is just a wrong choice."** The universal package is still required for distribution outside the store; the section above explained this with its reason. What is wrong is treating the universal package as the only way in a store release that has a budget, without ever weighing which files should land on which device. **"If the file count shrank, the package shrank."** These two statements are not the same thing. The count shows the effect of the configuration selection; size needs a separate measurement, and it was not made in this lesson. #### The Bounding Measurement This measurement is called bounding because it looks for the best answer the lesson's question can receive: the budget is the same, the package is the same, the samples are the same, and only the policy that decides which files land changes. The universal package coming out all should not be read as a defect of the package; when the policy ignores the configuration, no other result is possible. The reason we place the bounding measurement separately is to prevent a reader who sees the answer all from concluding that "this budget can never be met." Showing that it can be met under the same budget also proves that the budget is realistic. A fully split package for the configuration delivers only the matching files to each configuration. The budget is again five files. ```kotlin // Main.kt — the split package delivers only the matching files to each configuration fun main() { val files = samplePackage() val samples = sampleConfigs().map { filesForConfig(files, it, matching) } println("delivered files: $samples") Overrun("split-package", Budget("file", 5), samples).line() } ``` ```text delivered files: [4, 4, 4, 4, 4, 4] split-package: budget=file<=5 | over=0/6 -> within ``` The same package, the same six configurations, the same budget: the only difference is the policy, and the answer went from all to within. The universal package was this lesson's all example; the split package is its within counterpart. The format of the package did not decide the result on its own: the same app bundle gave two answers depending on the policy that decides which files land. There is something the rule cannot see in this result: the size of the files. Four small files and four large files count the same, and nine small files may be less download work than four large files. The model counts the file count and does not set up file size. The file count only makes the effect of the configuration-dependent selection visible. **DI2.** The model sets up the configuration as a pair of an ABI and a density. In reality language is also a split axis and a device can have more than one language; the main model has no language axis, and the `The Language Axis` section above sets it up separately in a single block. #### Summary - This lesson's overrun: all. The universal package exceeded the budget on six of six configurations, because the policy ignored the configuration and delivered all nine of nine files. - The app bundle is the publishing format uploaded to the store; the APK is the file installed on a device; the split APK is the set of files that matches a configuration. - The same measurement changed with the budget: under the budget of nine the universal package came out within, under the budget of three the split package came out all. The budget is an input to the model. - A package split only by ABI delivered seven files to every configuration and stayed at all. - With the language axis on, single-language devices received five files and two-language devices six, and the answer was minority; a device's property moves the number. - The package split by configuration delivered four files to every configuration and stayed within under the budget of five; the rule sees the file count, not the size. #### Next Step In this lesson we counted which files a build delivered to a device. But not all devices use the latest build: some downloaded the new version, some stayed on the old one. The next lesson, `Version Management`, measures how many versions behind the latest release the installed version codes of eight devices are. ### Version Management Source: https://academia.sh/en/courses/testing-performance-and-release/version-management The version code is the integer the platform compares, while the version name is the string shown to the user. The lesson compares how far the installed version codes of eight devices are from the latest release with a budget and shows that changing the version name does not change the distance. The previous lesson looked at the output of the build: how many files land on a configuration. This lesson deals with giving that output an identity. Devices do not all receive every build at the same time; some stay on the latest version and some on an old one. For this reason every build needs a number that says which build is newer than which. The question is again the course's question: **in how many samples does this measurement exceed its budget?** The measurement is how many versions a device has fallen behind the latest version, the budget is the number of versions back that are supported, and each device is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and gives how widespread the overrun is with four names: within, minority, majority, all. The rule sees the number; whether a device is one version behind or ten versions behind, it counts the same for every sample that overruns. #### Version Code and Version Name Android gives every app version two separate values. The version code is an integer; the platform and the app store look at this number to understand which of two versions is newer. The version name is the string shown to the user, and the platform does not compare it. `2.2.1` is a version name; for the platform it carries no meaning about whether it is newer than the version before it. ```kotlin // Platform code: module build script android { defaultConfig { versionCode = 12 versionName = "2.2.1" } } ``` This fragment is platform code; we claim no output here. The model sets up the rule of this measurement; the model is not the tool itself. In the model a release is a `Release` carrying two fields: a code and a name. Only the code does the comparison. The two values stand side by side in the script and both are written by hand, but they have different readers. `versionCode` tells the machine: this build comes after the previous one. `versionName` tells the human: this is which version the build is. A team can increase both at the same time, can increase only one, or can compute the code from the name; which one is done directly affects the distance we will measure throughout the lesson. This is why the model puts the code and the name in separate fields: when one changes, the other does not have to. Two rules govern the version code. The first is monotonic increase: the code of every new release must be greater than that of the one before it. The second is that the code cannot be lowered: the platform does not accept an update with a package whose code is lower than the installed one. Therefore, rolling back a faulty release does not mean republishing the old code; it means publishing a new build that carries the old behavior but a higher code. #### The Number of Versions Left Behind The model gathers the release history, the installed devices and a distance function in a helper file. `isMonotonic` checks that the codes strictly increase. `versionsBehind` counts how many distinct codes are greater than a device's installed code; two releases that carry the same code count as one, because the platform treats them as the same version. ```kotlin // VersionModel.kt — models the version code and name, computes the number of versions left behind class Release(val code: Int, val name: String) class InstalledDevice(val label: String, val installedCode: Int) fun isMonotonic(releases: List): Boolean = releases.zipWithNext().all { (previous, next) -> next.code > previous.code } fun latestCode(releases: List): Int = releases.maxOf { it.code } fun versionsBehind(releases: List, device: InstalledDevice): Int = releases.map { it.code }.distinct().count { it > device.installedCode } fun releaseHistory(): List = listOf( Release(5, "1.8.0"), Release(6, "1.8.1"), Release(7, "1.9.0"), Release(8, "2.0.0"), Release(9, "2.0.1"), Release(10, "2.1.0"), Release(11, "2.2.0"), Release(12, "2.2.1"), ) fun installedDevices(): List = listOf( InstalledDevice("device-a", 12), InstalledDevice("device-b", 11), InstalledDevice("device-c", 10), InstalledDevice("device-d", 9), InstalledDevice("device-e", 8), InstalledDevice("device-f", 6), InstalledDevice("device-g", 5), InstalledDevice("device-h", 12), ) ``` The eight releases carry eight different codes and two of the eight devices are on the latest version. We derive the budget as the result of a decision: the team decided not to support devices whose installed code is less than 10. If the lowest supported version code is 10 and the latest code is 12, the number of supported versions back is two. ```kotlin // Main.kt — the budget is derived from the lowest supported version, eight devices are measured fun main() { val releases = releaseHistory() println("codes monotonic: ${isMonotonic(releases)}") val minSupported = 10 val budget = Budget("version", latestCode(releases) - minSupported) val samples = installedDevices().map { versionsBehind(releases, it) } println("versions behind: $samples") Overrun("version-distance", budget, samples).line() } ``` ```text codes monotonic: true versions behind: [0, 1, 2, 3, 4, 6, 7, 0] version-distance: budget=version<=2 | over=4/8 -> majority ``` Four devices are more than two versions behind: three, four, six and seven. Because half of the eight exceed the limit, the answer is **majority**; exactly half counts as majority, not minority. This is the lesson's overrun. The rule does not see that device-g is seven versions behind and device-d is three; both are samples that overrun. **DI3.** The model measures the distance as a number of versions and weighs every version equally. In reality one version can carry a small fix and another a large change; the model does not set up this difference. The model also takes the devices' installed codes as an input; in reality, when updates are downloaded depends on the user's settings and the network. #### The Budget Is a Decision If the lowest supported version changes, the budget changes, and the same eight devices give a different overrun. We set three separate budgets: the team supports seven versions back, four versions back, two versions back. ```kotlin // Main.kt — the same eight devices give three different overruns under three budgets fun main() { val releases = releaseHistory() val samples = installedDevices().map { versionsBehind(releases, it) } val wide = Overrun("seven-versions", Budget("version", 7), samples) val middle = Overrun("four-versions", Budget("version", 4), samples) val narrow = Overrun("two-versions", Budget("version", 2), samples) listOf(wide, middle, narrow).forEach { it.line() } countOverruns(listOf(wide, middle, narrow)) } ``` ```text seven-versions: budget=version<=7 | over=0/8 -> within four-versions: budget=version<=4 | over=2/8 -> minority two-versions: budget=version<=2 | over=4/8 -> majority within: 1 minority: 1 majority: 1 all: 0 ``` With the budget at seven, no device strictly exceeds the limit; a device seven versions behind is equal to the limit, not an overrun. When the budget drops to four, only those six and seven versions behind overrun. When it drops to two, four devices overrun. Three of the four answers came out of the same measurement; **all** never appeared, because two devices are on the latest version and zero exceeds no valid budget. These three results are not a contradiction; they are the nature of the measurement. The number of supported versions back is a product decision. The longer the team supports old versions, the heavier the support load becomes; the more it supports, the wider the budget gets and the smaller the overrun. The lesson does not impose a budget; it only shows how the budget determines the overrun. #### The Version Name Does Not Affect the Rule The version name belongs to the user: calling a version `2.3.0` is a marketing or communication decision. The platform reads only the code. We imagine the team changes the version name to `2.3.0` and leaves the code at 12; in a second scenario the same version name is given but the code is increased to 13. #### The Bounding Measurement The bounding measurement asks, before the lesson's question is tied to one of the four answers of the overrun, which change moves the category. We measure three scenarios with the same eight devices and the same budget: the baseline, only the name changed, and both the name and the code changed. In the first scenario a release with code 12 but the name `2.3.0` is added to the release history. ```kotlin // Main.kt — if the version name changes and the code does not increase, the distance stays the same fun main() { val devices = installedDevices() val budget = Budget("version", 2) val before = releaseHistory() val renamed = before + Release(12, "2.3.0") val bumped = before + Release(13, "2.3.0") val baseline = Overrun("baseline", budget, devices.map { versionsBehind(before, it) }) val nameOnly = Overrun("name-only", budget, devices.map { versionsBehind(renamed, it) }) val nameAndCode = Overrun("name-and-code", budget, devices.map { versionsBehind(bumped, it) }) listOf(baseline, nameOnly, nameAndCode).forEach { it.line() } println("codes monotonic: ${isMonotonic(renamed)} ${isMonotonic(bumped)}") } ``` ```text baseline: budget=version<=2 | over=4/8 -> majority name-only: budget=version<=2 | over=4/8 -> majority name-and-code: budget=version<=2 | over=5/8 -> majority codes monotonic: false true ``` The scenario in which only the name changed gave exactly the same line as the baseline: four samples overrun, the answer is majority. The rule does not see the version name; `versionsBehind` counts only the codes and the codes did not change. When the code was increased to 13, the number of samples that overrun rose to five and the answer still stayed **majority**. Here the rule's second limit appears: the number of overruns increased but the category stayed constant, because the rule reduces the number of samples that exceed the limit to one of four answers. The result is read in two directions. A name change tells the user that there is a new version but does not make a device up to date; the update decision is made by the code. A code increase, on the other hand, leaves the same eight devices one version further behind: when the latest version is republished, everyone drops back a step. Every release pushes the budget in one direction. **DI4.** The model accepts a second release with the same code and counts it as the existing code. In reality the publishing console does not accept a second upload with the same version code; the model does not set up the rejection, because the lesson wants to show that the rule does not look at the version name. In this scenario `isMonotonic` returns false in the first case: this return is the model's counterpart of the rejection. #### Deriving the Code from the Name Many teams do not increase the code by hand; they compute it from the version name. A common form is to add the `major`, `minor` and `patch` values with weights. This rule has a silent trap: if the weights are chosen narrow, different names fall onto the same code. We measure the trap. The input is nine version names; this time the list also contains `1.10.0`. The budget is zero: every release that shares a code is an overrun. For each release, the number of other releases carrying the same code is the sample. ```kotlin // Main.kt — in deriving the code from the name, narrow multipliers collapse different names onto the same code fun foldCode(name: String, minorWeight: Int, majorWeight: Int): Int { val (major, minor, patch) = name.split(".").map { it.toInt() } return major * majorWeight + minor * minorWeight + patch } fun collisions(names: List, minorWeight: Int, majorWeight: Int): List { val codes = names.map { foldCode(it, minorWeight, majorWeight) } return codes.map { code -> codes.count { it == code } - 1 } } fun main() { val names = listOf("1.8.0", "1.8.1", "1.9.0", "1.10.0", "2.0.0", "2.0.1", "2.1.0", "2.2.0", "2.2.1") val budget = Budget("collision", 0) val narrow = collisions(names, 10, 100) val wide = collisions(names, 100, 10000) println("narrow multiplier: $narrow") println("wide multiplier: $wide") Overrun("narrow-multiplier", budget, narrow).line() Overrun("wide-multiplier", budget, wide).line() } ``` ```text narrow multiplier: [0, 0, 0, 1, 1, 0, 0, 0, 0] wide multiplier: [0, 0, 0, 0, 0, 0, 0, 0, 0] narrow-multiplier: budget=collision<=0 | over=2/9 -> minority wide-multiplier: budget=collision<=0 | over=0/9 -> within ``` With the narrow multiplier, `1.10.0` and `2.0.0` both produce 200. Because two of the nine releases collide, the answer is **minority**; with the wide multiplier there is no collision and the answer is within. A collision is not only a table defect: when two releases carry the same code, the platform cannot tell them apart, that is, it cannot see the second as an update. The rule does not see the cause of the collision; it only counts how many releases are affected. **DI5.** The model folds the code with a three-field name and two weights. Real teams also use other schemes such as a build number or a date; the model does not set these up and does not say which scheme should be chosen. The only thing measured is the number of collisions. #### Common Misconceptions **"If the version name grew, the version is newer."** For the platform, the newer one is the one whose code is greater. A package with a large name and a small code is counted as older than the installed one. **"The version code has a form like `2.2.1`."** The version code is an integer. A dotted notation is the job of the version name; teams that split the code into several fields do it by their own rule, folding it into a single integer. **"Deriving the code from the name is always safe."** The derivation seems to preserve monotonicity, but if the weights are chosen narrow it collides. In the measurement above, `1.10.0` and `2.0.0` gave the same code. The derivation rule should be tried against all the names in the release history before it changes. **"Changing only the version name publishes an update."** For the platform, an update is a package whose code increased. A package whose name alone changed is counted as the same version; in this lesson the number of samples that overrun stayed at four. **"A rolled-back version's code is rolled back."** The code cannot be lowered. A rollback is done with a new build with a higher code; by this lesson's measurement, the budget is pushed one more version. **"Skipping a version code is harmless."** The code must only increase; skipped numbers do not break the ordering but they go to waste. In a derived code, skipping also appears as a side effect: if the weights are chosen wide, the difference between two consecutive releases grows. The measurement looks only at the ordering between two releases; it does not see the gap in between, so whether the gap has a cost is something the team decides by its own rule. **"A collision only produces the same code."** The real harm of a collision is that two separate releases are counted as the same release for the platform. The second package cannot be installed over the first, because its code is not greater. This is why the collision budget is zero and even a single collision is an overrun. #### Summary - This lesson's overrun: majority. Four of eight devices fell more than two versions behind: because half exceeded the limit, the answer is majority, not minority. - The version code is the integer the platform compares and it increases monotonically; the version name is the string shown to the user and does not enter the comparison. - The budget was derived from the lowest supported version: the difference between 10 and 12 was two. The same eight devices came out within under the budget of seven, minority under the budget of four, majority under the budget of two. - When only the version name changed, the number of samples that overrun stayed at four; the rule does not see the version name. - A rule that derives the code from the name collapsed the names `1.10.0` and `2.0.0` onto the same code with a narrow weight: two overruns out of nine releases, the answer minority; with a wide weight, within. - When the version code was increased to 13, the number of samples that overrun rose to five but the answer stayed majority; the rule counts the number and does not see the size. #### Next Step In this lesson the version's identity was written by hand: `versionCode = 12` in a script fragment. In a real team, someone has to increase this number on every change and someone has to produce every build. The next lesson, `Continuous Integration and Delivery`, measures how many jobs run when a change arrives. ### Continuous Integration and Delivery Source: https://academia.sh/en/courses/testing-performance-and-release/continuous-integration-and-delivery Measures how many jobs the pipeline runs when a change arrives. The change filter runs only the jobs of the affected modules; changes that touch the shared module exceed the budget, and an unfiltered pipeline runs every job on every change. The previous lesson gave the version identity by hand: an integer in a build script. Someone has to increase this number on every change, and someone has to produce and test every build. If the work is done by hand it gets skipped, and the skipped work is noticed only when a user runs into a problem. Continuous integration ties this work to a pipeline: every time a change arrives, the build and the tests run on their own. Continuous delivery extends the end of the same pipeline: the build that passes the tests waits as an output ready for distribution. When the pipeline works well, a change comes back green or red and nobody asks what ran. The question appears when the pipeline grows and every change starts waiting a long time. So in this lesson we adapt the course's question to the pipeline: **in how many samples does this measurement exceed its budget?** The measurement is the number of jobs that run for a change, the budget is the number of jobs the team accepts per change, and each change is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and gives how widespread the overrun is with four names: within, minority, majority, all. The rule sees how many changes exceed the limit; whether a change ran six jobs or seven is the same for every sample that overruns. #### Pipeline, Job and Filter A pipeline (pipeline) is built from jobs. A job (job) is a step with a single purpose: static analysis, the unit tests of a module, building the app, a test that runs on a device. Each job says which modules it depends on. The change filter (change filter) uses this link: if a change touched only the `data` module, a job unrelated to `data` does not need to run. ```yaml # Platform code: workflow definition on: pull_request: paths: - "feature/**" - "core/**" jobs: unit-feature: steps: - run: ./build.sh :feature:test ``` This fragment is a workflow definition; it is not run and we claim no output here. The model sets up the rule of this measurement; the model is not the tool itself. In the model the filter is a set intersection: a job runs if one of the modules it depends on has changed. A job that depends on no module, that is, one whose dependency list is empty, runs on every change. In the fragment above, the `paths` field is the filter: only a change that touches files under `feature` or `core` starts this workflow. The filter has two faces. Its benefit is that it does not run the job at all for unrelated changes. Its risk is that the paths are written by hand: if a new module is added and not written into the paths, a change in that module triggers no job. In the model, this risk is carried by the `modules` set in the `Job` class; if the set is written incompletely the job does not run and the number drops. ```kotlin // PipelineModel.kt — models the jobs, the changes and the change filter class Job(val name: String, val modules: Set) class Change(val label: String, val modules: Set) fun runs(job: Job, change: Change): Boolean = job.modules.isEmpty() || job.modules.any { it in change.modules } fun jobsForChange(jobs: List, change: Change): List = jobs.filter { runs(it, change) }.map { it.name } fun pipelineJobs(): List = listOf( Job("checks", emptySet()), Job("unit-core", setOf("core")), Job("unit-data", setOf("data")), Job("unit-feature", setOf("feature", "core")), Job("unit-app", setOf("app", "core")), Job("device-test", setOf("app", "feature")), Job("build", emptySet()), ) fun sampleChanges(): List = listOf( Change("comment", emptySet()), Change("data", setOf("data")), Change("feature", setOf("feature")), Change("app", setOf("app")), Change("data-feature", setOf("data", "feature")), Change("app-data", setOf("app", "data")), Change("core", setOf("core")), Change("typo-fix", emptySet()), ) ``` There are seven jobs and eight changes. `checks` and `build` run on every change because their module list is empty; the other five jobs run only when the module they depend on changes. Two of the eight changes are only a comment and a typo fix, and touch no module. #### Jobs per Change We set the budget at four jobs: the team does not want a change to wait for more than four jobs. The measurement is the size of the `jobsForChange` result for each change. ```kotlin // Main.kt — the number of jobs that run for each change is measured, the budget is four jobs fun main() { val jobs = pipelineJobs() val changes = sampleChanges() for (change in changes) { println("${change.label}: ${jobsForChange(jobs, change)}") } val samples = changes.map { jobsForChange(jobs, it).size } println("running jobs: $samples") Overrun("jobs-per-change", Budget("job", 4), samples).line() } ``` ```text comment: [checks, build] data: [checks, unit-data, build] feature: [checks, unit-feature, device-test, build] app: [checks, unit-app, device-test, build] data-feature: [checks, unit-data, unit-feature, device-test, build] app-data: [checks, unit-data, unit-app, device-test, build] core: [checks, unit-core, unit-feature, unit-app, build] typo-fix: [checks, build] running jobs: [2, 3, 4, 4, 5, 5, 5, 2] jobs-per-change: budget=job<=4 | over=3/8 -> minority ``` Three changes exceed four jobs: two changes that touch two modules and one change that touches the shared module. Because fewer than half of the eight exceeded, the answer is **minority**. This is the lesson's overrun. The filter keeps most changes within the budget; the samples that overrun are the weak point of the pipeline. The `core` line shows one more thing: only `core` changed, but `core` is in the dependencies of three jobs, so this change triggered the widest run even though it touched a single module. The shared module behaves like a junction inside the pipeline: the more modules depend on it, the more jobs a change in that module runs. **DI3.** The model counts the number of jobs and weighs every job equally. In a real pipeline a unit test can take very little time and a device test very long; the model does not set up this difference. The module links are also an input: in reality the links come from the dependencies in the build script, and here they are written by hand. #### Transitive Dependency So far we have written by hand the modules each job depends on. In reality, when a module changes, the modules that depend on it are affected as well: when `core` changes, `data` and `feature`, which rely on `core`, and `app`, which relies on them, count as changed too. The links sit in a graph, and the set a change affects is derived from the graph. The measurement below counts the same eight changes in two ways. The first uses only the modules that changed directly. The second spreads the change to the dependent modules. This time the jobs are a separate unit test for each module; `checks` and `build` still run on every change. ```kotlin // Main.kt — when the transitive dependency is taken into account, the same changes run more jobs val dependsOn = mapOf( "data" to setOf("core"), "feature" to setOf("core"), "app" to setOf("feature", "data"), ) fun affected(changed: Set): Set { var result = changed while (true) { val next = result + dependsOn.filter { (_, deps) -> deps.any { it in result } }.keys if (next == result) return result result = next } } fun main() { val jobs = listOf( Job("checks", emptySet()), Job("unit-core", setOf("core")), Job("unit-data", setOf("data")), Job("unit-feature", setOf("feature")), Job("unit-app", setOf("app")), Job("build", emptySet()), ) val changes = sampleChanges() val direct = changes.map { jobsForChange(jobs, it).size } val spread = changes.map { jobsForChange(jobs, Change(it.label, affected(it.modules))).size } println("direct jobs: $direct") println("transitive jobs: $spread") val budget = Budget("job", 4) Overrun("direct", budget, direct).line() Overrun("transitive", budget, spread).line() } ``` ```text direct jobs: [2, 3, 3, 3, 4, 4, 3, 2] transitive jobs: [2, 4, 4, 3, 5, 4, 6, 2] direct: budget=job<=4 | over=0/8 -> within transitive: budget=job<=4 | over=2/8 -> minority ``` The same changes under the same budget came out **within** with the direct count and **minority** with the transitive count. The difference is largest in the `core` line: three jobs in the direct count, six jobs in the transitive count, because when core changed, all four modules were affected. We said the shared module is a junction; this measurement derives its cost from the graph. The direct count makes the pipeline look cheaper than it really is. **DI5.** The model takes the dependency graph as a table written by hand and assumes it contains no cycle. A real build system derives the graph from the scripts; there is no derivation here. The model also runs the tests of every affected module; a real pipeline may choose to run only in the modules whose interface changed, and the model does not set this up. #### The Budget Is a Decision We measure the same eight changes with four separate budgets. When the budget changes, the overrun changes; this is not a contradiction, it is the nature of the measurement. ```kotlin // Main.kt — the same eight changes give four different overruns under four budgets fun main() { val jobs = pipelineJobs() val samples = sampleChanges().map { jobsForChange(jobs, it).size } val roomy = Overrun("five-jobs", Budget("job", 5), samples) val normal = Overrun("four-jobs", Budget("job", 4), samples) val tight = Overrun("three-jobs", Budget("job", 3), samples) val minimal = Overrun("one-job", Budget("job", 1), samples) listOf(roomy, normal, tight, minimal).forEach { it.line() } countOverruns(listOf(roomy, normal, tight, minimal)) } ``` ```text five-jobs: budget=job<=5 | over=0/8 -> within four-jobs: budget=job<=4 | over=3/8 -> minority three-jobs: budget=job<=3 | over=5/8 -> majority one-job: budget=job<=1 | over=8/8 -> all within: 1 minority: 1 majority: 1 all: 1 ``` The four budgets gave four different categories. Under the budget of five jobs no change exceeded, because the widest run was five jobs. Under the budget of one job, all eight changes exceeded, because `checks` and `build` run on every change. A lesson follows from this: the number of jobs that run on every change is a floor the filter can never go below. If the budget is set below that floor, the answer is all no matter which filter is built. #### The Delivery End The delivery end of the pipeline asks a different question from integration: is this build ready for distribution? Here signing and packaging come into play. A signed build is an output signed with the release key; the signing key is a secret input of the pipeline and is kept apart from the jobs that run on ordinary changes. For this reason the delivery step runs not on every change but only when a release tag arrives. ```kotlin // Platform code: signed build android { signingConfigs { create("release") { storeFile = file(providers.gradleProperty("keystorePath").get()) keyAlias = providers.gradleProperty("keyAlias").get() } } } ``` The key path and alias are not embedded in the build script; the pipeline supplies them from outside. This fragment has no output; it only shows that the signature is a release input. The job at the delivery end does not enter the budget: a model that counts it works per release, not per change, and the lesson does not measure it. #### The Bounding Measurement This section measures the value of the filter by removing the filter altogether, so that we can see whether the budget is reachable. We remove the filter. All the jobs of the pipeline are written so that they cover all modules: every job runs on every change. This is the easiest pipeline to set up and the first form of most teams. ```kotlin // Main.kt — an unfiltered pipeline runs every job on every change fun main() { val everyModule = setOf("app", "core", "data", "feature") val unfiltered = pipelineJobs().map { Job(it.name, emptySet()) } val filtered = pipelineJobs() val changes = sampleChanges() val budget = Budget("job", 4) val withFilter = Overrun("filtered", budget, changes.map { jobsForChange(filtered, it).size }) val noFilter = Overrun("unfiltered", budget, changes.map { jobsForChange(unfiltered, it).size }) listOf(withFilter, noFilter).forEach { it.line() } println("module count: ${everyModule.size}, job count: ${filtered.size}") } ``` ```text filtered: budget=job<=4 | over=3/8 -> minority unfiltered: budget=job<=4 | over=8/8 -> all module count: 4, job count: 7 ``` The filtered pipeline came out **minority** under the same budget and the unfiltered pipeline **all**. In the unfiltered pipeline all the jobs are written with an empty dependency list, so seven jobs run on every change, and eight of eight changes exceed four jobs. Even a comment fix runs seven jobs. Here the limit of the rule appears. The rule does not know that the filter prevented a job from running; it sees only the number. If the filter is set up wrongly, that is, if a needed job does not run because a dependency was written incompletely, the number of jobs falls and the pipeline looks better, although a change that should have been tested was not tested. The number of overruns does not measure the correctness of the pipeline; it measures only its cost. **DI4.** The model assumes the filter is set up correctly. The filter being set up wrongly, that is, a dependency being forgotten, is not in the model; to see it, it is necessary to verify separately, for each job, on which changes it should run. #### Common Misconceptions **"Continuous integration and continuous delivery are the same thing."** Integration builds and tests every change. Delivery keeps the build that passes the tests ready for distribution. In this lesson the second was not measured, because the delivery step runs per release, not per change. **"If a filter is set up, the pipeline has become cheaper."** The filter only reduces the number of jobs that run. The difference between the direct count and the transitive count showed that a wrongly set up filter can make the pipeline look cheaper than it really is. The correctness of the filter is checked separately. **"Fewer jobs is always a better pipeline."** The rule counts the number of jobs; it does not see which job was skipped. If a needed job was skipped, the number drops but the pipeline gets worse. **"The faster the pipeline, the better."** This lesson did not measure time; it counted only the number of jobs that run on a change. The number of jobs does not give the time, because jobs can run in parallel and one job's time can differ greatly from another's. A claim about time needs a separate measurement. **"The dependency graph is derived once and does not change."** The dependencies between modules are born again with every change. If the graph is kept by hand it becomes stale and the transitive count is silently wrong; this lesson's model took the graph as an input written by hand, and in a real pipeline the graph must be read from the configuration. **"Every change affects every module."** It does not. An edit that changes only one module does not force the modules that do not depend on it to run; the job of the filter is to make this distinction. #### Summary - This lesson's overrun: minority. Three of eight changes exceeded the budget of four jobs; because fewer than half exceeded, the answer is minority. - Continuous integration runs the build and the tests on every change, and continuous delivery extends the same pipeline to an output ready for distribution. - The filter runs only the jobs whose module has changed; a change that touched the shared module triggered three jobs at once and gave the widest run. - The same changes under the budgets of five, four, three and one job came out within, minority, majority and all in turn; the two jobs that run on every change set the floor. - When the dependency graph was taken into account, the same changes ran more jobs: the direct count came out within and the transitive count minority; the direct count makes the pipeline look cheap. - The unfiltered pipeline exceeded the budget of four jobs on eight of eight changes: all. The rule counts the cost; it cannot count the correctness of the filter. #### Next Step The pipeline tested a build and left it ready for distribution. Before a build reaches real users, there are paths it passes through: internal testing, closed testing, open testing. The next lesson, `Testing Tracks`, measures in how many steps a build passes through these paths. ### Testing Tracks Source: https://academia.sh/en/courses/testing-performance-and-release/testing-tracks Measures how many steps a build takes to pass through the internal, closed and open testing tracks and reach production. Shows how the overrun changes when a rejected build is rebuilt and its steps are counted again from the start. The previous lesson measured how many jobs a change runs in the pipeline. At the end of the pipeline a build ready for distribution waits, but being ready does not mean reaching users. The build first lands on the team's own devices, then on a selected group of testers, then on a wider test audience; only at the end does it reach production. The track at each stop is a test of that build with a different audience. The course's question is the same here: **in how many samples does this measurement exceed its budget?** The measurement is the number of promotion steps a build takes before it reaches production, the budget is the number of steps the team accepts, and each build is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and reports how widespread they are with four names: within, minority, majority, all. The rule sees how many builds exceed the budget; whether a build took one step too many or three is the same for every sample that overruns. #### Four Tracks The release path is divided into four stops. The internal testing track belongs to the team itself: the build is uploaded and can be tried at once on the team's devices. The closed testing track is opened to a group of testers the team has chosen. The open testing track is opened to a test that is visible in the app store and that anyone can join. The production track is the users themselves. The order of the four stops gives the release path: internal, closed, open, production. A build is either promoted to the next track or rejected on a track. A rejection is the discovery of a bug: the build is fixed and tried again. The model builds the path as a list. Entering a track counts as one step. A build rejected on a track uses up the rejection and enters again, which means one more step. ```kotlin // ReleaseTrackModel.kt — models the track path and the number of promotion steps class Build(val label: String, val rejectedAt: List) fun trackPath(): List = listOf("internal", "closed", "open", "production") fun promotionSteps(path: List, build: Build, restartOnReject: Boolean): Int { var steps = 0 var index = 0 val pending = build.rejectedAt.toMutableList() while (index < path.size) { steps++ val track = path[index] if (track in pending) { pending.remove(track) index = if (restartOnReject) 0 else index } else { index++ } } return steps } fun sampleBuilds(): List = listOf( Build("build-a", emptyList()), Build("build-b", emptyList()), Build("build-c", emptyList()), Build("build-d", emptyList()), Build("build-e", listOf("closed")), Build("build-f", listOf("open")), Build("build-g", listOf("internal")), Build("build-h", emptyList()), ) ``` Four of the eight builds were never rejected and three were rejected once: one on the internal track, one on the closed track, one on the open track. The last build also passed clean. #### Measuring the Path The team says a build accepts four tracks plus one retry: the budget is five steps. We first measure the case where a rejected build retries only that track. ```kotlin // Main.kt — a rejected build is retried on the same track, budget five steps fun main() { val path = trackPath() val builds = sampleBuilds() val samples = builds.map { promotionSteps(path, it, false) } for (build in builds) { println("${build.label}: rejected=${build.rejectedAt}") } println("steps: $samples") Overrun("promotion-steps", Budget("step", 5), samples).line() } ``` ```text build-a: rejected=[] build-b: rejected=[] build-c: rejected=[] build-d: rejected=[] build-e: rejected=[closed] build-f: rejected=[open] build-g: rejected=[internal] build-h: rejected=[] steps: [4, 4, 4, 4, 5, 5, 5, 4] promotion-steps: budget=step<=5 | over=0/8 -> within ``` No build exceeded five steps: the three rejected builds finished in exactly five steps, the rest in four. The answer is **within**. This is the lesson's overrun. Five steps equals the limit, and equality does not count as an overrun. This result hides something: the budget was set wide enough to cover the share of a build rejected once. Lowering the budget to four makes even a single rejection an overrun. **DI3.** The model counts each track as one step and does not set up the time spent on a track. In reality a group of testers using the build takes waiting; the model sees neither the length of the wait nor that one track can take longer than another. The rejections are also an input: which build was rejected on which track was written by hand. #### A Budget Is a Decision We measure the same eight builds with three budgets: five steps, four steps, three steps. ```kotlin // Main.kt — the same eight builds give three different overruns with three budgets fun main() { val path = trackPath() val samples = sampleBuilds().map { promotionSteps(path, it, false) } val roomy = Overrun("five-steps", Budget("step", 5), samples) val normal = Overrun("four-steps", Budget("step", 4), samples) val tight = Overrun("three-steps", Budget("step", 3), samples) listOf(roomy, normal, tight).forEach { it.line() } countOverruns(listOf(roomy, normal, tight)) } ``` ```text five-steps: budget=step<=5 | over=0/8 -> within four-steps: budget=step<=4 | over=3/8 -> minority three-steps: budget=step<=3 | over=8/8 -> all within: 1 minority: 1 majority: 0 all: 1 ``` With the four-step budget only the three rejected builds exceeded it, and the answer is minority. With the three-step budget the four tracks themselves exceed the budget, so all eight builds overran: even a build that passes clean takes four steps. Note the floor here: as long as the path consists of four tracks, no build can fall below four steps. If a three-step budget were written down as a team decision, it would require skipping one of the tracks. #### Rebuilding A rejected build is fixed and rebuilt. A rebuild is a new package, and its version code, as we saw in an earlier lesson, must be greater than the previous one. ```kotlin // Platform code: rebuild android { defaultConfig { versionCode = 13 } } ``` This fragment is platform code; we do not claim output for it. The model sets up the rule of this measurement; the model is not the tool itself. What matters is the consequence: because the rebuilt package is not the same as the previous build, the result obtained on the earlier tracks may not carry over to the new package automatically. In this case the team either retries only the track it was rejected on, or walks the path from the beginning. This choice is a policy; the model sets up both with a flag. The `versionCode = 13` line in the script says two things at once. The first is that a rebuild carries a new version code; per the previous lesson's rule, the code can only increase. The second is that the result obtained on the tracks is tied not to the package itself but to the identity that the package's code carries. When the identity changes, the earlier track's approval does not carry over to the new package unless the team's policy says otherwise. In the model the `restartOnReject` flag sets up this policy: if false, only the rejected track is repeated; if true, the path returns to the start. #### The Bounding Measurement The lesson's overrun is within; the bounding measurement asks what this comfortable result rests on. If a rebuild starts from the beginning, every rejection also recounts the earlier tracks. We measure the same eight builds with two separate rules and add a ninth that is rejected twice on the same track. ```kotlin // Main.kt — a rebuild walks the path from the start, the steps are counted again fun main() { val path = trackPath() val builds = sampleBuilds() val budget = Budget("step", 5) val retry = builds.map { promotionSteps(path, it, false) } val restart = builds.map { promotionSteps(path, it, true) } println("retry on same track: $retry") println("restart from start: $restart") Overrun("same-track", budget, retry).line() Overrun("restart", budget, restart).line() val twice = Build("build-i", listOf("closed", "closed")) val withTwice = builds + twice Overrun("same-track-two-rejections", budget, withTwice.map { promotionSteps(path, it, false) }).line() Overrun("restart-two-rejections", budget, withTwice.map { promotionSteps(path, it, true) }).line() } ``` ```text retry on same track: [4, 4, 4, 4, 5, 5, 5, 4] restart from start: [4, 4, 4, 4, 6, 7, 5, 4] same-track: budget=step<=5 | over=0/8 -> within restart: budget=step<=5 | over=2/8 -> minority same-track-two-rejections: budget=step<=5 | over=1/9 -> minority restart-two-rejections: budget=step<=5 | over=3/9 -> minority ``` On the path retried on the same track the overrun came out **within**; under the rule where the path is walked from the beginning it came out **minority**. On the path that restarts, the build rejected on the closed track took six steps and the one rejected on the open track took seven; the build rejected on the internal track stayed at five, because at the moment it was rejected the path was already at its beginning. The further along the rejection happens, the more steps are counted from the start. The ninth build changes the table too. A build rejected twice on the same track takes six steps with the retry and eight steps under the rule that restarts. The category stayed **minority** in both cases, but the number of overrunning samples went from one to three. The rule counts the number and does not see the magnitude: one build exceeding by one step and another by three give the same category. **DI4.** The model takes a rejection as an input and assumes the fix will succeed. In reality a fix can bring a new bug; the model does not set this up. The number of rejections is also a hand-written list, not a rule. #### Skipping a Track The shortest way to reduce the number of steps is to remove a track from the path. A path that skips the closed track consists of three stops, and a clean build finishes in three steps. But a track's job is not to count steps; it is to catch a bug that would be caught there. The measurement below sets up the path as internal, open and production and lowers the budget to three steps. It also lists which builds would be rejected on the closed track, that is, which would have been caught had that track been on the path. ```kotlin // Main.kt — a path that skips the closed track reduces steps but misses the rejection that track would catch fun main() { val builds = sampleBuilds() val skipped = listOf("internal", "open", "production") val samples = builds.map { promotionSteps(skipped, it, false) } println("steps: $samples") val missed = builds.filter { build -> build.rejectedAt.any { it !in skipped } } println("missed rejections: ${missed.map { it.label }}") Overrun("closed-skipped", Budget("step", 3), samples).line() } ``` ```text steps: [3, 3, 3, 3, 3, 4, 4, 3] missed rejections: [build-e] closed-skipped: budget=step<=3 | over=2/8 -> minority ``` The four-track path had come out all at the three-step budget; the three-track path came out **minority** at the same budget. But the gain came at a cost: `build-e` would have been rejected on the closed track, and because the path did not include that track the rejection was never seen, and the build moved on to the open track without being fixed. The number of steps dropped; the stop where the bug is caught disappeared. **DI5.** The model does not assume that a rejection on a skipped track will be caught on another track, and it shows this with the `missed rejections` list. But it does not set up whether the bug will appear later on the open track or in production, or what it would cost if it did. #### Common Misconceptions **"The number of tracks is a measure of quality."** The number of tracks is the number of stops a build passes through. A build passing through many tracks does not show it was well tested; the quality of testing depends on the audience on the track and on the feedback. **"A rejected build is a bad build."** A rejection is a track doing its job. Tracks exist to find bugs before they reach production. **"Removing a track is enough to meet the budget."** Removing a track lowers the number of steps and can bring the measurement inside the budget or near it. But the bug that the track would have caught goes uncaught. The measurement above showed this on one build: the overrun decreased, and the missed rejection was one. **"A rebuild carries over the result of the earlier tracks."** It does not, because it is a new package and carries a new version code. Whether it carries over is the team's policy; in this lesson both policies were measured separately. **"The step count gives the duration."** It does not. A step can take a day or a few hours; the model does not measure duration. **"A build that passes the internal track also passes the open track."** It may not. As the audience grows, the variety of devices and usage grows; a bug that never shows up on the internal track appears only in a wider audience. The order of the tracks follows a growing audience for this reason. **"Skipping tracks only saves time."** It does save time, but the bug the track would have caught also goes to production. In this lesson the path that skips the closed track met the step budget and the number of missed rejections rose to one; the gain and the loss were seen in the same table. **"The order of the tracks is a matter of the team's taste."** The order determines whose hands a build passes through before it reaches whom. Changing the order means showing a build earlier to a less closely watched audience. This is a design decision, and the measurement itself does not recommend an order. **"The amount of feedback is the success of a track."** A lot of feedback can mean a lot of bugs, or it can mean the audience is engaged. The model does not measure the amount of feedback; it measures the number of steps. **"If the overrun decreased, the process has improved."** The overrun is counted only against a budget. If the budget is loosened or a track is removed, the overrun decreases by itself; whether the process has improved is understood by looking at which bugs are still caught before production. For this reason the overrun count and the number of missed rejections should be read side by side. Deciding by looking at a single number is not hearing what the other number would say. **"Every build must pass through all four tracks."** This is a team policy, not a rule. A small fix and a large feature need not follow the same path; but every decision that shortens the path brings with it a list of bugs that will go uncaught. Writing this list down and knowing who accepted it is the precondition for shortening the path. #### Summary - This lesson's overrun: within. None of the eight builds exceeded the five-step budget; the three builds rejected once stayed at exactly five. - The release path consists of the internal, closed, open and production tracks, and entering each track counted as one step; for this reason no build can fall below four steps. - The same eight builds came out within, minority and all at budgets of five, four and three steps. - When a rejected build walks the path from the beginning, the steps rose to six and seven, and the overrun went from within to minority. - Removing the closed track took the three-step budget from all down to minority, but a rejection that would have been caught on the closed track was missed. - The ninth build, rejected twice, raised the number of overrunning samples from one to three; the category stayed minority because the rule does not see the magnitude. #### Next Step The build has passed through the tracks and is ready to go to production, but going to production does not have to mean going to all users at once. The next lesson, `Staged Rollout`, widens the release stage by stage and compares the crash count at each stage with a threshold. ### Staged Rollout Source: https://academia.sh/en/courses/testing-performance-and-release/staged-rollout Widens a release to the user base stage by stage and compares the crash count at each stage with an error threshold. Shows that halting at the stage where the overrun appears stops collecting samples for the later stages, and why the sample count needs to be at least four. The previous lesson measured how a build passes through the tracks. A build that reaches the production track lands on users' own devices; however well tested it is, some bugs show up only in real use. A staged rollout limits the cost of such a bug: the build is first opened to a small group of users, and if no problem is seen it widens to a larger group in the next stage. The team watches the release at each stage. The monitoring window is the period during which a stage stays open and crashes are counted. If the crash count exceeds the error threshold, the team has two decisions: to halt the release, that is, not to move on to the next stage and to stop offering the version to new users, though devices that already received the version do not go back to the old one; or to roll back, that is, to republish the previous version with a higher version code. The stages are mentioned in order in this lesson; every team has its own stage layout. The course's question is the same here: **in how many samples does this measurement exceed its budget?** The measurement is the crash count (`crashCount`) in a stage, from a user group of the same size; the budget is the error threshold; each stage is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and reports how widespread they are with four names: within, minority, majority, all. The rule sees how many stages exceed the threshold; whether a stage carries one crash too many or ten is the same for every sample that overruns. #### The Code That Collects Crashes When the app crashes, the system arrives at a default handler; teams route this handler to their own code to record the crash. ```kotlin // Platform code: CrashRecorder Thread.setDefaultUncaughtExceptionHandler { thread, error -> recordCrash(thread.name, error) } ``` This fragment is a platform call; we do not claim output for it. The model sets up the rule of this measurement; the model is not the tool itself. In the model a crash is the outcome of a session: a session either ends sound or crashes. The handler does not change the system's crash path; it only ties the recording of the crash to the team's code. After the handler takes its record, the app still terminates. This distinction matters for the measurement: the crash count is the number of crashes that could be recorded. A crash that could not be recorded at all, or a session that never crashed but ran in a broken way, does not enter the count. Since in the model a session is either `crash` or `sound`, both of these gaps stay outside the model. #### Modeling the Stages The helper file builds each stage as a group made of a fixed number of sessions. Because the stages are the same size, the crash counts can be compared directly; had the stages differed in size, the count would mislead. ```kotlin // RolloutModel.kt — models the staged rollout and the crash count per stage class Stage(val label: String, val outcomes: List) fun crashCount(stage: Stage): Int = stage.outcomes.count { it == "crash" } fun stageNames(): List = listOf( "first-stage", "second-stage", "third-stage", "fourth-stage", "fifth-stage", "final-stage", ) fun rolloutStages(crashes: List): List = stageNames().zip(crashes).map { (name, crashed) -> Stage(name, List(10) { if (it < crashed) "crash" else "sound" }) } fun releaseA(): List = listOf(1, 1, 2, 5, 3, 4) fun decide(count: Int, limit: Int): String = if (count > limit) "halt" else "continue" fun firstOverIndex(counts: List, limit: Int): Int { val index = counts.indexOfFirst { it > limit } return if (index == -1) counts.lastIndex else index } ``` Each stage is built from ten sessions; `releaseA` gives the crash count per stage. Six stages means six samples. #### Measuring the Stages We set the error threshold at three crashes: the team does not accept more than three crashes in a stage. ```kotlin // Main.kt — the crash count in each stage is measured, error threshold three crashes fun main() { val stages = rolloutStages(releaseA()) val samples = stages.map { crashCount(it) } for (stage in stages) { println("${stage.label}: crashes=${crashCount(stage)}/${stage.outcomes.size}") } println("crashes: $samples") Overrun("stage-crashes", Budget("crash", 3), samples).line() } ``` ```text first-stage: crashes=1/10 second-stage: crashes=1/10 third-stage: crashes=2/10 fourth-stage: crashes=5/10 fifth-stage: crashes=3/10 final-stage: crashes=4/10 crashes: [1, 1, 2, 5, 3, 4] stage-crashes: budget=crash<=3 | over=2/6 -> minority ``` Two stages exceeded three crashes: the fourth stage carried five crashes, the final stage four. The fifth stage stopped at three and, being equal to the limit, did not count as an overrun. Since fewer than half of the six stages exceeded, the answer is **minority**. This is the lesson's overrun. **DI3.** The model builds each stage from ten sessions of the same size and counts a crash as one outcome per session. In reality stages differ in size, and one user's many sessions can repeat a single crash; the model sees neither. The crash counts are hand-written inputs, not a measurement. #### The Threshold Is a Decision We measure the same six stages with four separate thresholds. When the threshold changes, the overrun changes; this is not a contradiction, it is what the measure is. ```kotlin // Main.kt — the same six stages give four different overruns with four thresholds fun main() { val samples = rolloutStages(releaseA()).map { crashCount(it) } val loose = Overrun("five-crashes", Budget("crash", 5), samples) val normal = Overrun("three-crashes", Budget("crash", 3), samples) val strict = Overrun("one-crash", Budget("crash", 1), samples) val zero = Overrun("zero-crashes", Budget("crash", 0), samples) listOf(loose, normal, strict, zero).forEach { it.line() } countOverruns(listOf(loose, normal, strict, zero)) } ``` ```text five-crashes: budget=crash<=5 | over=0/6 -> within three-crashes: budget=crash<=3 | over=2/6 -> minority one-crash: budget=crash<=1 | over=4/6 -> majority zero-crashes: budget=crash<=0 | over=6/6 -> all within: 1 minority: 1 majority: 1 all: 1 ``` The four thresholds gave four categories. The threshold that allows five crashes counted no stage as an overrun. The threshold that allows zero crashes, on the other hand, was exceeded by all six stages, because every stage had at least one crash. What value the threshold should have is a product decision; the model does not decide, it shows how the decision changes the overrun. #### An Overrun Is Not a Decision A distinction is needed here. A halt is not one of the four categories of the overrun; it is a decision made as a result of the overrun. The rule sees only the number and says within, minority, majority or all. Halting the release is the decision the team makes after seeing the overrun: when the threshold is exceeded in a stage, the release does not move on to the next stage. For this reason the decision lives in a separate function. `Overrun` never says "halt"; `decide` makes a separate decision for each stage. ```kotlin // Main.kt — the category and the decision are separate functions fun main() { val limit = 3 val stages = rolloutStages(releaseA()) val samples = stages.map { crashCount(it) } val overrun = Overrun("stage-crashes", Budget("crash", limit), samples) overrun.line() for (stage in stages) { println("${stage.label}: decision=${decide(crashCount(stage), limit)}") } println("category: ${overrun.verdict}") } ``` ```text stage-crashes: budget=crash<=3 | over=2/6 -> minority first-stage: decision=continue second-stage: decision=continue third-stage: decision=continue fourth-stage: decision=halt fifth-stage: decision=continue final-stage: decision=halt category: minority ``` The category is **minority**; the decision is to halt at the fourth stage. The two answer different questions: the category answers "how widespread is the overrun?", the decision answers "what will we do?". If the release is going to be halted the moment the first overrun is seen, the fifth and final stages are never opened and their counts are never measured. #### The Bounding Measurement This section asks when a category stops being readable. A halt cuts off the sampling of the later stages. The sample count shrinks and the category can change. We take three separate releases: in the first the overrun appears at the fourth stage, in the second at the second stage, in the third at the first stage. Each is measured with all six stages and in its form halted at the first overrun. ```kotlin // Main.kt — halting at the first overrun reduces the sample count, the category can change fun main() { val limit = 3 val releases = listOf( "release-a" to releaseA(), "release-b" to listOf(1, 4, 1, 1, 2, 1), "release-c" to listOf(4, 1, 1, 1, 2, 1), ) for ((name, crashes) in releases) { val full = rolloutStages(crashes).map { crashCount(it) } val halted = full.take(firstOverIndex(full, limit) + 1) Overrun("$name-full", Budget("crash", limit), full).line() Overrun("$name-halted", Budget("crash", limit), halted).line() } } ``` ```text release-a-full: budget=crash<=3 | over=2/6 -> minority release-a-halted: budget=crash<=3 | over=1/4 -> minority release-b-full: budget=crash<=3 | over=1/6 -> minority release-b-halted: budget=crash<=3 | over=1/2 -> majority release-c-full: budget=crash<=3 | over=1/6 -> minority release-c-halted: budget=crash<=3 | over=1/1 -> all ``` In the first release four stages were sampled and the category stayed **minority**. In the second release only two samples remained, and the same release that was minority in the full measurement appeared as **majority** in the halted one. In the third release a single sample remained and the answer became **all**. The full measurement of all three releases is minority; what changed is not the behavior of the release but the number of samples measured. From this comes the sample count rule: at least four samples are needed to draw a category from a measurement. Below four samples the category only tells that sample collection was stopped, not about the release itself. With two samples, one overrun is half of the samples. For this reason the category of a halted release is not read; the decision itself is read. **DI4.** The model assumes the halt happens right at the first overrun. In reality teams may wait for a few overruns, or a person may assess the overrun; the model does not set up this delay. The model also does not know what will happen to the later stages of a halted release: if the halted stages are opened later, the measurement starts again. #### Stage Size In the third assumption we said the stages are the same size. A real release does not proceed this way: the first stage is small and the last stage covers most of the users. When the size changes, the raw crash count becomes misleading, because in a large stage the same crash rate means more crashes. The measurement below builds six stages with growing sizes and puts two measurements side by side: the raw crash count and the number of crashes per thousand sessions. The second is computed with integer division, so the rounding is downward. The budgets of the two measurements are in different units and are not derived from each other; we set three crashes for the raw count and fifty crashes per thousand sessions for the rate. ```kotlin // Main.kt — in growing stages the raw crash count and the rate per thousand sessions give different answers fun main() { val sizes = listOf(10, 20, 50, 100, 200, 500) val crashes = listOf(1, 1, 2, 5, 6, 40) val stages = stageNames().indices.map { index -> Stage(stageNames()[index], List(sizes[index]) { if (it < crashes[index]) "crash" else "sound" }) } val counts = stages.map { crashCount(it) } val rates = stages.map { crashCount(it) * 1000 / it.outcomes.size } println("crashes: $counts") println("crashes per thousand sessions: $rates") Overrun("raw-count", Budget("crash", 3), counts).line() Overrun("rate", Budget("crash-per-thousand", 50), rates).line() } ``` ```text crashes: [1, 1, 2, 5, 6, 40] crashes per thousand sessions: [100, 50, 40, 50, 30, 80] raw-count: budget=crash<=3 | over=3/6 -> majority rate: budget=crash-per-thousand<=50 | over=2/6 -> minority ``` In the raw count three stages exceeded the budget and the answer was **majority**; in the rate only two stages exceeded and the answer stayed **minority**. The difference shows in the first and the last stage. In the first stage a single crash means a hundred crashes per thousand sessions and does not show up in the raw count at all. In the last stage forty crashes look like a lot in the raw count, but divided by five hundred sessions they come to eighty crashes per thousand sessions. The fourth stage's five crashes correspond to fifty crashes per thousand sessions and are not above the budget in either measurement. The result is that the choice of measurement moves the category. The rule does not decide which measurement is right; the team sets the budget in a unit and counts in that unit. **DI5.** The model counts a session as one crash outcome and computes the rate with integer division. In reality rates are fractional, and on small stages the fact that a single crash makes the rate so volatile carries a statistical uncertainty; the model does not measure this uncertainty. The sizes and the crashes are inputs. #### Common Misconceptions **"A staged rollout prevents bugs."** It does not; it limits how many users a bug affects. **"If the crash count is low, the release is healthy."** The count only counts sessions that crashed. A version that does not crash but runs in a broken way does not show up in the count. **"A single crash in a small stage is unimportant."** If the stage is small, a single crash is a high rate; the raw count hides this. The first stage above corresponded to a hundred crashes per thousand sessions. **"Many crashes in a large stage is a bad sign."** Having many can be a result of being large. In the final stage forty crashes corresponded to eighty per thousand sessions, and it was only one of the two rates that exceeded the budget. **"A halt is an outcome."** A halt is not one of the categories, it is a decision. The same category can end in two different decisions on two teams. **"More stages make a release safer."** Each extra stage adds a window in which a crash can show up, but it also delays the completion of the release. The number of stages is a balancing decision; this lesson only measured the given stages and did not recommend an ideal number. **"The rate is always more correct than the raw count."** The rate takes size into account, but on a small stage even a single crash makes the rate jump. This was seen in the first stage above. For the rate to be reliable, enough sessions are needed; how many sessions are enough was not given in this lesson, because it was not measured. **"Crash monitoring ends when the release ends."** Crashes keep coming after the release has reached all users. Stages only raise the chance of catching a bug early; the closing of the monitoring windows does not say the version is free of problems. **"Setting the threshold once is enough."** The threshold is reconsidered according to the size of the release and the number of users. Putting the same number on every stage carries a different meaning as the stages grow; a raw-count threshold is exceeded sooner in a large stage, while a rate threshold is independent of size. For this reason which measure the threshold is tied to matters as much as the threshold itself. When the threshold changes, the results of the earlier stages are interpreted again too. #### Summary - This lesson's overrun: minority. Two of the six stages exceeded the three-crash threshold; since fewer than half exceeded, the answer is minority. - A staged rollout limits the cost of a bug; when an overrun is seen, the team halts the release or rolls it back. - The same six stages came out within, minority, majority and all at thresholds of five, three, one and zero crashes. - A halt is not a category but a decision made as a result of the overrun; the category is minority, the decision is to halt at the fourth stage. - When the stage sizes grow, the raw crash count came out majority and the rate per thousand sessions came out minority; the choice of measurement moves the category. - When fewer than four samples remain after a halt, the category tells that sample collection stopped, not about the release: it came out majority with two samples and all with one. #### Next Step The release widened stage by stage and each stage was compared with a threshold. But before it goes out there is one more obstacle: the app store's review rules. The next lesson, `Store Review and Policies`, counts the permissions the app declares but does not use. ### Store Review and Policies Source: https://academia.sh/en/courses/testing-performance-and-release/store-review-and-policies Counts, per module, the permissions the app declares but does not use, and compares the count with a budget. Shows how the overrun closes once the unused permissions are removed, and that the rule does not see the weight of a finding. The previous lesson widened the release stage by stage and looked at the crash count at every stage. But before a build reaches users, it also passes the app store's own inspection. The store reviews every uploaded release against its policies (store review): are the permissions the app declares really used, how is user data handled, does the description on the store page match what the app does. The details of the review are set by policies that change from one store to another and from one time to another; this lesson neither lists the content of the policies nor counts how many there are. Instead it measures something the team itself holds: the permissions the app declares but does not use. The course's question is the same here: **in how many samples does this measurement exceed its budget?** The measurement is the number of permissions in a module that are declared but unused (`policyFindings`), the budget is the number the team accepts, and each module is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and gives the prevalence with four names: within, minority, majority, all. The rule sees how many modules overrun; whether a module carries one or three unused permissions is the same for every sample that overruns. #### Permission Declaration When an app wants to reach the user's data or a piece of the device's hardware, it declares this request in advance in a manifest. The declaration is the contract that tells the user and the store what the app can do. ```xml ``` This snippet is a configuration file; it is not run, and we claim no output. The model sets up the rule for this measurement; the model is not the tool itself. In the model a permission is a name; a module carries both the set of permissions it declares and the set it actually uses in its code. The finding is the difference between the two: a permission that is declared and not used. It is easy to see why this finding matters. An unused permission looks to the user like an unnecessary request and calls for an explanation in review. This is why teams try to declare permissions only as far as they actually use them. The declaration is addressed to two separate readers, and they look for different things. The user sees the declaration as a request: the app wants to reach the camera, should that be allowed? The store reads the declaration as a question of consistency: does what is declared match the app's description and behavior? The two-line snippet above is therefore not only a configuration; it is a promise the team makes to the user and to the store. If the promise is made wider than needed, the user sees an unnecessary permission and the store asks for an explanation. If it is made too narrow, the feature does not work. The way between the two is to make the declared set equal to the used set, and this lesson's measurement counts that inequality. #### Setting Up the Model The helper file sets up eight modules, the permissions each declares and uses, and the finding count. The permission names are plain words; they are the short labels used in the lesson, not the real names on the platform. ```kotlin // PolicyModel.kt — models, per module, the permissions that are declared but unused class AppModule(val label: String, val declared: Set, val used: Set) fun unusedPermissions(module: AppModule): List = (module.declared - module.used).sorted() fun policyFindings(module: AppModule): Int = unusedPermissions(module).size fun trimmed(module: AppModule): AppModule = AppModule(module.label, module.declared.intersect(module.used), module.used) fun appModules(): List = listOf( AppModule("main", setOf("location", "camera"), setOf("location")), AppModule("login", setOf("contacts"), setOf("contacts")), AppModule("payment", setOf("camera", "location", "contacts"), setOf("camera")), AppModule("map", setOf("location", "storage"), setOf("location")), AppModule("profile", setOf("camera", "storage", "contacts"), setOf("camera", "storage")), AppModule("search", setOf("microphone"), setOf("microphone")), AppModule("help", setOf("location", "calendar"), emptySet()), AppModule("settings", setOf("storage", "calendar"), setOf("calendar")), ) ``` #### Measuring the Findings The team's rule is simple: a module must not declare any permission it does not use. The budget is zero: even a single unused permission is an overrun. ```kotlin // Main.kt — the unused permissions of each module are counted, budget zero fun main() { val modules = appModules() for (module in modules) { println("${module.label}: unused=${unusedPermissions(module)}") } val samples = modules.map { policyFindings(it) } println("findings: $samples") Overrun("unused-permission", Budget("finding", 0), samples).line() } ``` ```text main: unused=[camera] login: unused=[] payment: unused=[contacts, location] map: unused=[storage] profile: unused=[contacts] search: unused=[] help: unused=[calendar, location] settings: unused=[storage] findings: [1, 0, 2, 1, 1, 0, 2, 1] unused-permission: budget=finding<=0 | over=6/8 -> majority ``` Six modules exceeded while the budget was zero; only `login` and `search` came out clean. Because more than half of the eight exceeded, the answer is **majority**. This is the lesson's overrun. **DI3.** The model counts a permission as a name. In reality a permission may be used not by one module but by several, a permission may come together with another permission, or a library may add a permission without the app knowing. The model does not set up these three cases. The declared and used sets are also hand-written inputs; in a real app the usage comes from scanning the code. #### The Budget Is a Decision Measure the same eight modules with three separate budgets: two, one and zero unused permissions. ```kotlin // Main.kt — the same eight modules give three different overruns under three different budgets fun main() { val samples = appModules().map { policyFindings(it) } val roomy = Overrun("two-permissions", Budget("finding", 2), samples) val normal = Overrun("one-permission", Budget("finding", 1), samples) val strict = Overrun("zero-permissions", Budget("finding", 0), samples) listOf(roomy, normal, strict).forEach { it.line() } countOverruns(listOf(roomy, normal, strict)) } ``` ```text two-permissions: budget=finding<=2 | over=0/8 -> within one-permission: budget=finding<=1 | over=2/8 -> minority zero-permissions: budget=finding<=0 | over=6/8 -> majority within: 1 minority: 1 majority: 1 all: 0 ``` With a budget of two, no module exceeded. Dropping to one, only `payment` and `help`, which have two unused permissions, exceeded. At zero, six modules exceeded. The three results came from the same measurement; they do not refute one another, they only show what the budget accepts. Setting the budget to zero is a strict team rule, and as long as the rule is not loosened the answer cannot come out as all, because two modules carry no findings at all. #### The Count Does Not See Weight The count treats two findings as equal, yet not all findings carry the same weight. The camera, location, contacts and microphone permissions reach more sensitive user data; the storage and calendar permissions will be assumed less sensitive in the lesson. This is a model assumption, not the platform's classification. ```kotlin // Main.kt — the count does not see weight, the number of sensitive permissions is measured separately fun main() { val sensitive = setOf("camera", "location", "contacts", "microphone") val modules = appModules() for (module in modules) { val unused = unusedPermissions(module) val heavy = unused.count { it in sensitive } println("${module.label}: findings=${unused.size} sensitive=$heavy") } val samples = modules.map { policyFindings(it) } Overrun("unused-permission", Budget("finding", 0), samples).line() } ``` ```text main: findings=1 sensitive=1 login: findings=0 sensitive=0 payment: findings=2 sensitive=2 map: findings=1 sensitive=0 profile: findings=1 sensitive=1 search: findings=0 sensitive=0 help: findings=2 sensitive=1 settings: findings=1 sensitive=0 unused-permission: budget=finding<=0 | over=6/8 -> majority ``` The `map` and `settings` modules carry one finding but have no sensitive findings; `main` and `profile` also carry one finding, and that finding turns out to be sensitive. The rule counts the four the same: all four are samples that overrun. The category is **majority**, and this category does not say which module is riskier in review. The verdict of the review does not come out of this count. The store may reject a release or ask for a fix; this decision depends not on the category but on the review's own assessment. The model does not set up the rejection decision, and here we claim neither a rejection nor an acceptance. **DI4.** The sensitivity distinction in this lesson is an assumed set; which permission counts as sensitive is determined separately in a real store policy and changes over time. The model takes this set as an input. #### Difference Between Releases The declaration is not written once and left; a new permission may be added in every release. A permission added in a new release is a new declaration whose justification may be asked for in review. So for the team the real question is not only today's surplus but what was added in this release. The measurement below takes the previous release's declaration as an input for each module and gives two counts: the number of added permissions and the number of added permissions that are unused. The budget is zero in both. ```kotlin // Main.kt — how many of the permissions added in the new release are unused fun main() { val previous = mapOf( "main" to setOf("location"), "login" to setOf("contacts"), "payment" to setOf("camera", "location"), "map" to setOf("location", "storage"), "profile" to setOf("camera", "storage", "contacts"), "search" to emptySet(), "help" to setOf("location"), "settings" to setOf("storage", "calendar"), ) val modules = appModules() val added = modules.map { it.declared - previous.getValue(it.label) } val addedUnused = modules.mapIndexed { index, module -> (added[index] - module.used).size } println("added: ${added.map { it.size }}") println("added and unused: $addedUnused") val budget = Budget("permission", 0) Overrun("added", budget, added.map { it.size }).line() Overrun("added-unused", budget, addedUnused).line() } ``` ```text added: [1, 0, 1, 0, 0, 1, 1, 0] added and unused: [1, 0, 1, 0, 0, 0, 1, 0] added: budget=permission<=0 | over=4/8 -> majority added-unused: budget=permission<=0 | over=3/8 -> minority ``` DI5. The previous release's declaration is a hand-written input, and the module names are assumed not to change between releases. In reality a module may be split, merged or renamed; the model does not set up these cases. It also does not tell whether an added permission came for a new feature or as a side effect of a library. #### The Bounding Measurement Remove the unused permissions: each module's declaration comes down to only the permissions it uses. This is exactly the case in which the finding closes. The bounding measurement therefore shows the best answer that can be given to the lesson's question, and tells whether the budget is reachable. ```kotlin // Main.kt — once the unused permissions are removed, all modules fit the budget fun main() { val before = appModules() val after = before.map { trimmed(it) } val budget = Budget("finding", 0) val beforeSamples = before.map { policyFindings(it) } val afterSamples = after.map { policyFindings(it) } Overrun("before", budget, beforeSamples).line() Overrun("after", budget, afterSamples).line() println("declared before: ${before.sumOf { it.declared.size }}") println("declared after: ${after.sumOf { it.declared.size }}") } ``` ```text before: budget=finding<=0 | over=6/8 -> majority after: budget=finding<=0 | over=0/8 -> within declared before: 16 declared after: 8 ``` Once the unused permissions were removed, the overrun dropped from the **majority** category to the **within** category; the number of declared permissions fell from sixteen to eight. This result is a count: eight permissions were never used in the app's code, and removing them did not change the functionality. If it turns out that one of the removed permissions was used in the code in some other way, the usage set was wrong. Then what needs fixing is not the declaration but how the usage was detected. This result shows that the budget is reachable, but it does not show the cost of reaching it. Removing eight permissions did not change the functionality, but saying that a permission is really unused depends on how usage is detected. The model took the usage set as an input; in a real app this set is derived from the code, and the derivation may miss indirect uses, for example calls a library makes on the app's behalf. The result of the bounding measurement is therefore read on the condition that usage is verified by an independent path before the removal. #### Where the Usage Set Comes From The lesson's measurement rests on the difference between two sets, but only one of the sets is written in a file. The declared set is read from the manifest and is explicit. The usage set is the result of which permission-requiring calls the code makes, and it needs a scan or a compiler output. The reliability of these two sources is not the same: the declaration is exact, the usage is an inference. The difference comes from this asymmetry. If the usage set is computed too small, the surplus looks larger than it is, and a permission that is removed breaks a feature. If the usage set is computed too large, the surplus looks smaller than it is, and an unnecessary permission stays. The first error shows up to the user, the second in review. For this reason the team tests both directions separately: for each removed permission it verifies that there really is no feature that uses it, and for each remaining permission that there is a feature that uses it. The model does not set up this verification; it takes the usage set as correct and only counts the difference. #### Common Misconceptions **"If there are no findings, the review passes."** It does not. The count measures only the difference between declaration and usage; the review checks other policies too. **"If the finding count is low, the risk is low."** The number does not show the weight of a finding. One sensitive finding can matter more than several non-sensitive ones. **"Adding a new permission is always a problem."** The problem is a permission that is added and not used. The `search` module added a new permission in this release, and the permission was used in the code; it appeared as an overrun in the first count but not in the second. **"Shrinking the declaration does not affect features."** It does not if the usage set is correct. If the usage set is incomplete, that is, if a permission is used in the code in some other way, shrinking the declaration breaks that feature. Before shrinking, it is necessary to verify that usage detection is reliable. **"Every permission should be removed."** Only the unused ones are removed. Removing a permission that is used makes that feature stop working. **"A non-sensitive permission is not a problem."** The sensitivity set was an assumption in this lesson. A permission not counted as sensitive also exceeds the budget if it is unused, and calls for an explanation in review. Sensitivity only serves to order which finding is handled first; it does not remove the existence of the finding. **"Keeping the declaration clean is a one-time job."** In every release a new feature may bring a new permission, and when a feature is removed its permission may stay in the declaration. This is why the count runs again at a release gate, before every release; an audit that runs once and closes accumulates surplus again in the next release. **"Keeping the budget at zero is always realistic."** Zero was a strict rule the team chose in this lesson. The three budgets showed that the same eight modules come out majority at zero, minority at one and within at two. A realistic budget depends on which findings the team can close before review. #### Summary - This lesson's overrun: majority. Six of eight modules carried an unused permission under a zero budget; because more than half exceeded, the answer is majority. - Store review checks against policies; this lesson did not measure the content of the policies but the finding count in the team's hands. - The same eight modules came out within, minority and majority under budgets of two, one and zero, in that order. - The rule counts four findings the same, but two of them were sensitive permissions; the count shows neither weight nor the rejection decision. - The fault of the permissions added relative to the previous release is not in their being added but in their being unused; two counts separate the two. - Once the unused permissions were removed, the declaration fell from sixteen to eight and the overrun became within. #### Next Step The app passed review, widened stage by stage and reached users. But the release is not over; the real data arrives once users begin to use the app under real conditions. The next lesson, `Post-Release Monitoring`, compares the crash counts of the days after release with a budget. ### Post-Release Monitoring Source: https://academia.sh/en/courses/testing-performance-and-release/post-release-monitoring Compares each day's crash count in the monitoring window after release with the error threshold. Measures how the jump in the first days leaves the overrun at minority, and how the same counts change category when summed weekly and the sample count drops; then gathers the course's 19 lessons in one table. The previous lesson took the app through review and delivered it to users; `Post-Release Monitoring` compares the crash counts of the days after release with a budget. A release is not passing through a gate but the opening of a window: while the build is in real users' hands, the team waits for data to arrive for a period called the **monitoring window**. This lesson treats each day of the window as a measurement. The course's question is the same here: **in how many samples does this measurement exceed its budget?** The measurement is a day's crash count (`crashCount`), the budget is the error threshold the team accepts in a day, and each day of the window is a sample. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `Overrun` counts the samples that strictly exceed the limit and gives the prevalence with four names: within, minority, majority, all. The rule sees how many days overrun; whether a day carries one or ten crashes more is the same for every day that overruns. #### Distinction: Report or Threshold In the earlier part of the course, the Analytics and Crash Reporting lesson counted in how many copies a crash report reaches its recipient. That lesson measured the delivery of the report. This lesson does not measure delivery: it assumes the reports arrive and counts how many times over the days the threshold was exceeded. One examines the path of an event, the other the relation of a number to a budget. #### The Platform Piece That Shows a Crash A crash report is the record produced at the moment the app stops with an unexpected error. The record may also carry an Application Not Responding (ANR) condition as a separate type. During monitoring, teams usually look at the crash-free session rate; because this lesson measures a count and not a rate, we use only the raw count. ```kotlin // Platform code: CrashReporting val reporter = crashReportingFramework.instance() reporter.setCustomKey("release_window_day", dayIndex) reporter.recordException(error) ``` This snippet is a platform call; we claim no output. The model sets up the rule for this measurement; the model is not the tool itself. In the model a session ends either as `crash` or as `sound`; the extra information the framework collects, the stack, or the ANR type remains outside the model. #### Modeling the Monitoring Window The helper file builds each day of the window from ten sessions. Because the days are the same size, the crash counts are directly comparable. ```kotlin // MonitoringModel.kt — models the monitoring window and the crash count per day class ReleaseDay(val label: String, val outcomes: List) fun crashCount(day: ReleaseDay): Int = day.outcomes.count { it == "crash" } fun releaseWindow(crashes: List): List = crashes.mapIndexed { index, crashed -> ReleaseDay("day-${index + 1}", List(10) { if (it < crashed) "crash" else "sound" }) } fun launchWindow(): List = listOf(6, 5, 4, 2, 2, 1, 1, 2, 1, 1, 2, 1, 1, 1) fun weeklyTotals(counts: List): List = counts.chunked(7).map { it.sum() } ``` `launchWindow` keeps the first days high and the rest low. Crashes cluster in the first days after release: in the first hours of a release come the most eager users, the sessions that try the most different paths. #### Daily Measurement Set the error threshold at three crashes a day. ```kotlin // Main.kt — each day of the monitoring window is measured, the error threshold is three crashes a day fun main() { val days = releaseWindow(launchWindow()) val samples = days.map { crashCount(it) } println("crashes: $samples") Overrun("daily-crash", Budget("crash", 3), samples).line() } ``` ```text crashes: [6, 5, 4, 2, 2, 1, 1, 2, 1, 1, 2, 1, 1, 1] daily-crash: budget=crash<=3 | over=3/14 -> minority ``` Three of the fourteen days exceeded the threshold: the first three days carried six, five and four crashes respectively. Because fewer than half exceeded, the answer is **minority**. This is the lesson's overrun. **DI6.** The model builds every day from the same ten sessions and counts a crash as one outcome per session. In reality the number of daily sessions grows as the release widens, and a user can repeat the same crash; the model sees neither. `launchWindow` is a hand-written input; the days' crash counts come out of the model through `days.map { crashCount(it) }`, but the numbers themselves do not come from a real monitoring. #### Separating the Jump The difference of the first three days from the others does not show in the rule; the rule only says that three days exceeded. If we split the window in two and measure each part separately, the jump becomes visible. ```kotlin // Main.kt — the window is split into the first three days and the later days, and measured fun main() { val samples = releaseWindow(launchWindow()).map { crashCount(it) } val spike = Overrun("first-three-days", Budget("crash", 3), samples.take(3)) val rest = Overrun("later-days", Budget("crash", 3), samples.drop(3)) spike.line() rest.line() countOverruns(listOf(spike, rest)) } ``` ```text first-three-days: budget=crash<=3 | over=3/3 -> all later-days: budget=crash<=3 | over=0/11 -> within within: 1 minority: 0 majority: 0 all: 1 ``` In the first three days every day exceeded, and in the following eleven days none did. The whole window came out minority because eleven calm days diluted the three jump days. This is not a problem but a property of the measure: as the window grows, the overrun value depends more on the length of the window than on the jump. **DI7.** The three-day split is the split of a team that knows from the start where the jump ends. In real monitoring the end of the jump is not known; choosing the split point is the team's decision, and the model does not make this decision. #### The Bounding Measurement What happens if the same fourteen days are summed weekly rather than daily? Teams often report with a single number, and that number is usually weekly. ```kotlin // Main.kt — the same crash counts are summed weekly and measured with two separate budgets fun main() { val samples = releaseWindow(launchWindow()).map { crashCount(it) } val weekly = weeklyTotals(samples) println("weekly: $weekly") val same = Overrun("weekly-same-budget", Budget("crash", 3), weekly) val scaled = Overrun("weekly-sevenfold", Budget("crash", 21), weekly) same.line() scaled.line() } ``` ```text weekly: [21, 9] weekly-same-budget: budget=crash<=3 | over=2/2 -> all weekly-sevenfold: budget=crash<=21 | over=0/2 -> within ``` The same fourteen days were summed into two weeks: 21 crashes in the first week, 9 in the second. When the daily budget was left as it is, both weeks exceeded and the answer was **all**. When the budget was raised to seven times, that is 21, the first week also did not count as an overrun because 21 equals the limit, and the answer was **within**. The daily cut's answer of **minority** did not appear in either weekly measurement. With two samples, minority cannot be built: the condition `over * 2 < total` holds with 2 samples only for zero overruns, and zero overruns means within. **DI8.** Taking the weekly total's budget as seven times the daily budget assumes the days are evenly distributed. Crashes after release are not evenly distributed; the sevenfold budget therefore hides the first week's jump. Pulling the budget to another number would have given another overrun. This measurement tests not a rule but a way of cutting: the lesson says how the sample is cut decides the overrun. In the daily cut there were fourteen samples and the overrun came out minority; in the weekly cut two samples remained and the overrun became tied to the sample count itself. This is why the course has the principle that the sample count should be at least four. #### The Budget Is a Decision Three crashes a day is a number the team chose. A stricter threshold counts more days as overruns; a looser threshold also takes in the jump of the first days. Choosing the number is deciding which release you will halt. The model does not make this decision; it only says how many days the chosen number makes exceed. #### Common Misconceptions **"The longer the window, the more accurate the data."** As the window grows, calm days multiply and the jump thins out; the overrun value changes, but this change is not the release improving, it is the way the sample is cut. **"The weekly total is a better summary than the daily one."** The weekly total keeps the count but divides the sample count by seven. With two samples, one of the four categories, minority, cannot be built at all. **"A release that is minority is healthy."** Minority says that fewer than half of the days exceeded; it does not say which days exceeded. In this lesson the days that exceeded were the first three, the moments when the release reached the most users. #### Closing Table The course asked the same question in 19 lessons. Each lesson's overrun is the "This lesson's overrun:" line in that lesson's `## Summary` section; the measured unit of work and the bounding measurement are in the source lesson itself. | Lesson | Measured unit of work | Overrun | Bounding measurement | |---|---|---|---| | Unit Tests | input validation step | within | at a budget of 5 steps, minority | | Mocking | network call in a transaction | all | with a stub cache, calls drop to zero, within | | On-Device Tests | work per test | all | with shared setup, minority | | UI Tests | step to find a node | majority | with a test tag, a single step, within | | Test Stability | wait for readiness | minority | with condition-based polling, within | | Static Analysis | violations per module | majority | with a baseline file, minority | | Startup Time | work until the first frame | minority | by deferring the dependency, within | | Rendering Smoothness | items recomposed per frame | all | with a stable key, minority | | Memory Management | retained object | majority | binding the listener to the lifecycle, within | | Network and Battery Profiling | requests per round | within | with one-by-one sending, minority | | Benchmarking | work per iteration | minority | with warm-up discarded, the new version is all | | Code Shrinking and Obfuscation | classes remaining per module | within | with a broad keep rule, all | | App Package Formats | files that go down to a configuration | all | in a split APK, within | | Version Management | device's distance from the release | majority | stays the same when only the version name changes | | Continuous Integration and Delivery | jobs that run per change | minority | with an unfiltered pipeline, all | | Testing Tracks | steps until production | within | when a rejected build is counted from the start, minority | | Staged Rollout | crashes per stage | minority | with a halt the sample count shrinks, all | | Store Review and Policies | unused permissions per module | majority | with the permissions removed, within | | Post-Release Monitoring | daily crash count | minority | with weekly summing the sample count drops | Of the 19 lessons, 4 close within, 6 minority, 5 majority and 4 all. The core in the opening lesson gave a different distribution: the four measurements came out 1 within, 1 minority, 1 majority, 1 all. The two distributions do not count the same thing; the core counts four measurements, the table counts nineteen lessons. #### Summary - This lesson's overrun: minority. Three of fourteen days, the first three days, exceeded the threshold of three crashes a day. - The jump of the first days thins out when spread over the whole window: the first three days came out all, the following eleven days within. - When the same counts are summed weekly, the sample count falls from fourteen to two and the category changes. How the sample is cut decides the overrun. - This lesson counts not the delivery of the crash report but how many times over the days the reports exceeded the threshold. - The crash reporting framework is platform code and is not measured in this lesson. #### Course Wrap-Up Across testing, performance and distribution, the course asked a single question: in how many samples does this measurement exceed its budget? The table shows that all four separate answers to the same question were really given. The next curriculum, M32, moves to the subject area of cross-platform mobile development.