--- title: 'Unit Tests' source: 'https://academia.sh/en/courses/testing-performance-and-release/unit-tests' course: 'Testing, Performance and Release' language: en updated: '2026-10-21T05:02:43+00:00' license: 'CC BY-SA 4.0' objectives: - 'Classifies in how many samples a measurement exceeds its budget as within, minority, majority, or all' - 'Explains that a unit test pins a number for a set of inputs, not for a single input' - 'Sets a budget from the worst case of a function and counts the samples that exceed it' - 'Measures that the same function yields a different overrun when the budget changes' --- # Unit Tests A unit test pins the number a function in the pure Kotlin layer produces for a set of inputs. The lesson classifies in how many samples a measurement exceeds its budget with four answers: within, minority, majority, all. The previous course counted how many copies of an event reached a recipient. That course's measurement axis does not continue here. In this course the app is taken to production, and every step toward production compares a measurement with a limit: a unit test compares the steps a function takes, an instrumented test compares the number of rounds of a wait loop, a startup measurement compares the work done before the first screen, and a staged rollout compares the crashes in each stage. A single measurement misleads, because on a single input everything can look fine. So the same measurement is taken on several inputs, and each input becomes a sample. The course's one question starts with this lesson: **in how many samples does this measurement exceed its budget?** A budget is the limit a measurement is expected not to cross. The answer has four forms, and their names do not change throughout the course. If no sample exceeds the budget, the measurement is **within**. If some samples exceed it but fewer than half of the samples do, it is **minority**. If at least half of the samples exceed it but not all, it is **majority**. If every sample exceeds it, it is **all**. The umbrella name for the four answers is overrun; when we speak of a measurement's overrun, we mean one of these four answers. ## The Rule of the Measurement The label does not give the overrun; the rule does. The core below defines a budget (`Budget`) and a measurement (`Overrun`) with a label, a budget, and a list of samples. It takes the number of samples and the number of samples that strictly exceed the limit, then asks the question in order. A sample equal to the limit does not count as exceeding. Exactly half also counts as majority: if four of eight samples exceed the budget, the measurement is majority, not minority. ```kotlin // Overrun.kt — classifies in how many samples a measurement exceeds its budget class Budget(val name: String, val limit: Int) class Overrun(val label: String, val budget: Budget, samples: List) { val total: Int = samples.size val over: Int = samples.count { it > budget.limit } val verdict: String = when { over == 0 -> "within" over == total -> "all" over * 2 < total -> "minority" else -> "majority" } fun line() { println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict") } } fun countOverruns(overruns: List) { for (name in listOf("within", "minority", "majority", "all")) { println("$name: ${overruns.count { it.verdict == name }}") } } ``` `countOverruns` counts the four overruns in a fixed order and prints an overrun even when its count is zero. The rule misses two things. The first is magnitude: a sample that exceeds the budget by one step counts the same as a sample that exceeds it by ten steps. The second is cause: a slow algorithm, an extra call, and a budget set too tight can all produce the same overrun. The prose of the lesson states the magnitude and the cause; the rule does not. Writing a list of samples by hand shows the rule, but it measures nothing; it is a demonstration, not a measurement. In this course, every measurement that carries an overrun takes its samples from the return values of a model function: the function produces a unit of work from an input, and the list of samples is the numbers the function returns for each input. Depending on the lesson, the unit of work is a step, a round, a request, or a crash; it is never a duration. The budget is an input of the model as well: when a lesson sets a budget, it states the reasoning, and it does not present the budget as a platform threshold. **TE1.** The model does not run the test framework itself; there is no framework in this environment. In the model, what a test measures is the number of steps a function takes for one input. The test class, on the other hand, is a platform piece and prints nothing. ## What a Unit Test Pins Down A unit test runs a function in the pure Kotlin layer of the app and compares its result with an expectation. The function that runs is the test target (system under test), and the call that states the expectation is the assertion. The pure Kotlin layer knows nothing about the screen, the file system, the network, or the clock; the function always returns the same result for the same input. For that reason the test runs on the development machine in very little time, without a device or an emulator. An assertion pins a single number for a single input. The behavior of the function, however, changes with the input: a function that takes two steps on one input can take ten steps on another. Once a set of inputs is chosen, the unit test turns into a collection of assertions that pin the number for every input in that set. On the platform it looks like this. ```kotlin // Platform code: ValidatorTest import org.junit.Assert.assertEquals import org.junit.Test class ValidatorTest { @Test fun acceptsShortCode() { assertEquals(4, validateSteps("AB12")) } @Test fun stopsAtFirstBrokenCharacter() { assertEquals(2, validateSteps("A-1234")) } } ``` Each `@Test` method pins one input. The model sets the rule of this measurement; the model is not the tool itself. In the model, instead of assertions we look at the rule: how many of the samples exceed the budget. It is easy to confuse this question with two questions from earlier courses. In M30/K06 we compared a test with a release pair and asked whether this test separates the correct release from the faulty one. What was measured there was the correctness of the test, and every measurement rested on a pair made of one test and one release. There is no pair here: there is a single measurement, a single budget, and a list of samples. The question is not the correctness of the test but how many times the budget is exceeded. ## Eight Samples, Four Searches Suppose the pure Kotlin layer of an app has a product code search. The codes are in order from 1 to 16, and eight codes are searched. Four search methods are measured against the same budget: a search may take at most four steps. We do not present four steps as a platform threshold; we set it as the model's counterpart of the share a product screen reserves for one search. `scanSteps` scans from the start and takes as many steps as the position of the searched code. `halveSteps` halves the range at every step. `filterSteps` walks the whole list even after it finds the searched code. `bucketSteps` places a code into a bucket by its remainder modulo eight and searches only in that bucket. Each function returns the number of steps it took to find the searched code. ```kotlin // Main.kt fun scanSteps(items: List, target: Int): Int { var steps = 0 for (item in items) { steps++ if (item == target) return steps } return steps } fun halveSteps(items: List, target: Int): Int { var low = 0 var high = items.size - 1 var steps = 0 while (low <= high) { steps++ val mid = (low + high) / 2 when { items[mid] == target -> return steps items[mid] < target -> low = mid + 1 else -> high = mid - 1 } } return steps } fun filterSteps(items: List, target: Int): Int { var steps = 0 items.filter { steps++; it == target } return steps } fun bucketSteps(items: List, target: Int): Int = items.filter { it % 8 == target % 8 }.indexOf(target) + 1 fun main() { val codes = (1..16).toList() val targets = listOf(2, 5, 7, 9, 12, 14, 16, 3) println("searched codes: $targets") val step = Budget("step", 4) val overruns = listOf( Overrun("linear-search", step, targets.map { scanSteps(codes, it) }), Overrun("binary-search", step, targets.map { halveSteps(codes, it) }), Overrun("full-filter", step, targets.map { filterSteps(codes, it) }), Overrun("bucket-lookup", step, targets.map { bucketSteps(codes, it) }), ) overruns.forEach { it.line() } countOverruns(overruns) } ``` ```text searched codes: [2, 5, 7, 9, 12, 14, 16, 3] linear-search: budget=step<=4 | over=6/8 -> majority binary-search: budget=step<=4 | over=1/8 -> minority full-filter: budget=step<=4 | over=8/8 -> all bucket-lookup: budget=step<=4 | over=0/8 -> within within: 1 minority: 1 majority: 1 all: 1 ``` Each of the four measurements gave a different overrun: within 1, minority 1, majority 1, all 1. This distribution is the binding number of the course; the number of samples is 8 and the budget is 4 steps. The samples of each measurement are the list that the corresponding function returned for the eight inputs. `linear-search` is majority. The positions of codes 5, 7, 9, 12, 14, and 16 are greater than four, while 2 and 3 stay under the limit. Six samples are more than half of eight. `binary-search` is minority: the only place where sixteen codes need five steps is the last code, so only 16 exceeds. `full-filter` takes 16 steps in every sample and is all; even when the searched code is among the first, the filter walks the list to the end. `bucketSteps` puts two codes in each bucket, so no sample goes beyond two steps and the measurement is within. The difference between majority and all lies not in the number but in the inference. If it is majority, the budget fails only for some inputs, and the choice of inputs matters. If it is all, no input can meet the budget, and the function itself is outside the budget. ## Input Validation: A Budget From the Worst Case A unit test is set up not only for search but for every pure function. Take a function that validates a campaign code. The function walks the characters of the code in order and stops as soon as it sees an invalid character. A code has at most six characters, so the worst case of the function is six steps. Setting the budget at this worst case is the way to pin that the function never exceeds its own limit on any input. In this lesson the function stays in a helper file and the later measurements call it. ```kotlin // Validation.kt — walks the characters of a campaign code in order, stops at an invalid character fun validateSteps(code: String): Int { var steps = 0 for (ch in code) { steps++ if (!ch.isLetterOrDigit()) return steps } return steps } ``` We chose eight codes. One of the codes is short, two are broken right at the start, and three are long enough to walk all six characters. The list of codes will stay the same; only the budget will change. ```kotlin // Main.kt — the input validation budget is set from the worst case fun main() { val codes = listOf("AB12", "A-1234", "SUM24", "123456", "K 9", "TR2026", "ZZ", "P7Q8R9") val steps = codes.map { validateSteps(it) } println("tried codes: $codes") println("steps: $steps") val worst = Budget("step", 6) val overrun = Overrun("code-validation", worst, steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text tried codes: [AB12, A-1234, SUM24, 123456, K 9, TR2026, ZZ, P7Q8R9] steps: [4, 2, 5, 6, 2, 6, 2, 6] code-validation: budget=step<=6 | over=0/8 -> within within: 1 minority: 0 majority: 0 all: 0 ``` The steps of the eight codes came out as 4, 2, 5, 6, 2, 6, 2, and 6. None went past six, so the measurement is within. The three codes that take six steps sit exactly at the limit of the budget, and a sample equal to the limit does not count as exceeding. The value of this measurement is not a number but a contract: the function will change in the future, and if one code takes seven steps, a test that said within up to that moment will break. This is why a unit test is not written once and forgotten; with every change, the same eight inputs are measured again with the same budget. **TE2.** The budget of 6 steps derives from the character count of the longest code for this function. On the platform, the product team can choose a different worst case; the model sets this choice as a budget and does not turn it into a threshold assertion. ## Common Misconceptions Three misconceptions about unit tests keep coming back, and the measurement gives each one a concrete answer. The first is the thought that "if the test is green, the function is correct". A green test says only that the assertions pinned for the chosen inputs hold. If the set of inputs is chosen narrowly, the worst case can stay outside the set. This is why we set the budget from the step count of the longest code in this lesson: if the budget does not rest on the hardest member of the set, the answer within says nothing. The second is the thought that "the tighter the budget, the better". A tight budget raises the overrun, but the function does not get worse because the overrun count went up; only the limit moved. The same function came out within at a budget of 6 steps and minority at a budget of 5 steps. The function was the same. What was measured was not the function itself but its relation to the limit. This is why a budget value is always written together with a reason, and when the reason changes, the budget is reviewed too. The third is the thought that "a unit test looks only for bugs". Here the test pins not a bug but a number. If the number moves in the next change, the test breaks and the team notices the change. Even in a bug-free function, a silent rise in the step count is the first sign that the budget is being approached; the test exists to catch that sign. What the three misconceptions share is reading more than a single result can give. The rule says only this: in how many samples the limit was strictly exceeded. Anything beyond that, that is, why the overrun happened, is the team's work. ## The Bounding Measurement Tighten the budget. The function did not change and the list of codes did not change; only the budget went down from 6 steps to 5 steps. The same eight samples will be compared with the new limit. ```kotlin // Main.kt — same function, same codes, tighter budget fun main() { val codes = listOf("AB12", "A-1234", "SUM24", "123456", "K 9", "TR2026", "ZZ", "P7Q8R9") val steps = codes.map { validateSteps(it) } val tight = Budget("step", 5) val overrun = Overrun("code-validation-tight", tight, steps) overrun.line() countOverruns(listOf(overrun)) } ``` ```text code-validation-tight: budget=step<=5 | over=3/8 -> minority within: 0 minority: 1 majority: 0 all: 0 ``` The same function, the same eight inputs, and the same steps: 4, 2, 5, 6, 2, 6, 2, 6. When the budget drops to 5, the three codes that take six steps exceed it. Three of eight samples exceed, and because three is under half, the measurement is minority. The count was zero in the previous measurement. Not a single line changed in the function. This is not a contradiction; it is the measurement itself. The overrun is a property not of the function alone but of the function and the budget together. The same function can be within at one budget, minority at another, and majority or all at an even tighter one. This is why, when a measurement's overrun is stated, the budget is stated too: "code validation is within" alone is not information; "code validation is within at a budget of 6 steps" is information. The two things the rule misses apply here as well. It misses magnitude: three codes that exceed a five-step budget by one step are, in the eyes of the rule, the same as three codes that exceed it by ten steps. It misses cause as well: in this measurement the cause of the overrun is not the function, because the function stayed the same; the cause is that the budget was set tighter. Someone writing a unit test should, when reading a measurement's overrun, look first not at the function but at what the budget was set from. Two lessons come out of the four searches. The first is that the same budget can give four overruns on four functions: keeping the budget fixed tells the functions apart. The second is that the same function can give two overruns at two budgets: keeping the function fixed tells the budgets apart. A unit test pins something in both directions. By fixing the set of inputs it locks the function, and by fixing the budget it locks the expectation. ## Summary - This lesson's overrun: within. Input validation with a budget of six steps did not exceed the budget in any of the eight codes. - The lesson's criterion, in how many samples a measurement exceeds its budget: the four searches came out within 1, minority 1, majority 1, all 1; the number of samples was 8 and the budget was 4 steps. - A unit test pins a number for a set of inputs, not for a single input; a function in the pure Kotlin layer gives the same result for the same input. - The same function and the same eight codes moved from within to minority when the budget dropped from 6 steps to 5 steps: the overrun is a property of the function and the budget together, not of the function alone. - The rule sees how many samples overrun, not by how much or why; the prose states both. ## Next Step The functions measured in this lesson touched no outside resource. A real repository, however, calls a network resource, and a unit test does not want that call. The next lesson, Mocking, measures in how many samples the call count exceeds the budget when a dependency is replaced with its test double.