Testing Tracks

The previous lesson measured how many jobs a change runs in the pipeline. At the end of the pipeline a build ready for distribution waits, but being ready does not mean reaching users. The build first lands on the team’s own devices, then on a selected group of testers, then on a wider test audience; only at the end does it reach production. The track at each stop is a test of that build with a different audience.

The course’s question is the same here: in how many samples does this measurement exceed its budget? The measurement is the number of promotion steps a build takes before it reaches production, the budget is the number of steps the team accepts, and each build is a sample.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

Overrun counts the samples that strictly exceed the limit and reports how widespread they are with four names: within, minority, majority, all. The rule sees how many builds exceed the budget; whether a build took one step too many or three is the same for every sample that overruns.

Four Tracks

The release path is divided into four stops. The internal testing track belongs to the team itself: the build is uploaded and can be tried at once on the team’s devices. The closed testing track is opened to a group of testers the team has chosen. The open testing track is opened to a test that is visible in the app store and that anyone can join. The production track is the users themselves.

The order of the four stops gives the release path: internal, closed, open, production. A build is either promoted to the next track or rejected on a track. A rejection is the discovery of a bug: the build is fixed and tried again.

The model builds the path as a list. Entering a track counts as one step. A build rejected on a track uses up the rejection and enters again, which means one more step.

// ReleaseTrackModel.kt — models the track path and the number of promotion steps
class Build(val label: String, val rejectedAt: List<String>)

fun trackPath(): List<String> = listOf("internal", "closed", "open", "production")

fun promotionSteps(path: List<String>, build: Build, restartOnReject: Boolean): Int {
    var steps = 0
    var index = 0
    val pending = build.rejectedAt.toMutableList()
    while (index < path.size) {
        steps++
        val track = path[index]
        if (track in pending) {
            pending.remove(track)
            index = if (restartOnReject) 0 else index
        } else {
            index++
        }
    }
    return steps
}

fun sampleBuilds(): List<Build> = listOf(
    Build("build-a", emptyList()),
    Build("build-b", emptyList()),
    Build("build-c", emptyList()),
    Build("build-d", emptyList()),
    Build("build-e", listOf("closed")),
    Build("build-f", listOf("open")),
    Build("build-g", listOf("internal")),
    Build("build-h", emptyList()),
)

Four of the eight builds were never rejected and three were rejected once: one on the internal track, one on the closed track, one on the open track. The last build also passed clean.

Measuring the Path

The team says a build accepts four tracks plus one retry: the budget is five steps. We first measure the case where a rejected build retries only that track.

// Main.kt — a rejected build is retried on the same track, budget five steps
fun main() {
    val path = trackPath()
    val builds = sampleBuilds()
    val samples = builds.map { promotionSteps(path, it, false) }
    for (build in builds) {
        println("${build.label}: rejected=${build.rejectedAt}")
    }
    println("steps: $samples")
    Overrun("promotion-steps", Budget("step", 5), samples).line()
}

build-a: rejected=[]
build-b: rejected=[]
build-c: rejected=[]
build-d: rejected=[]
build-e: rejected=[closed]
build-f: rejected=[open]
build-g: rejected=[internal]
build-h: rejected=[]
steps: [4, 4, 4, 4, 5, 5, 5, 4]
promotion-steps: budget=step<=5 | over=0/8 -> within

No build exceeded five steps: the three rejected builds finished in exactly five steps, the rest in four. The answer is within. This is the lesson’s overrun. Five steps equals the limit, and equality does not count as an overrun.

This result hides something: the budget was set wide enough to cover the share of a build rejected once. Lowering the budget to four makes even a single rejection an overrun.

DI3. The model counts each track as one step and does not set up the time spent on a track. In reality a group of testers using the build takes waiting; the model sees neither the length of the wait nor that one track can take longer than another. The rejections are also an input: which build was rejected on which track was written by hand.

A Budget Is a Decision

We measure the same eight builds with three budgets: five steps, four steps, three steps.

// Main.kt — the same eight builds give three different overruns with three budgets
fun main() {
    val path = trackPath()
    val samples = sampleBuilds().map { promotionSteps(path, it, false) }
    val roomy = Overrun("five-steps", Budget("step", 5), samples)
    val normal = Overrun("four-steps", Budget("step", 4), samples)
    val tight = Overrun("three-steps", Budget("step", 3), samples)
    listOf(roomy, normal, tight).forEach { it.line() }
    countOverruns(listOf(roomy, normal, tight))
}

five-steps: budget=step<=5 | over=0/8 -> within
four-steps: budget=step<=4 | over=3/8 -> minority
three-steps: budget=step<=3 | over=8/8 -> all
within: 1
minority: 1
majority: 0
all: 1

With the four-step budget only the three rejected builds exceeded it, and the answer is minority. With the three-step budget the four tracks themselves exceed the budget, so all eight builds overran: even a build that passes clean takes four steps. Note the floor here: as long as the path consists of four tracks, no build can fall below four steps. If a three-step budget were written down as a team decision, it would require skipping one of the tracks.

Rebuilding

A rejected build is fixed and rebuilt. A rebuild is a new package, and its version code, as we saw in an earlier lesson, must be greater than the previous one.

// Platform code: rebuild
android {
    defaultConfig {
        versionCode = 13
    }
}

This fragment is platform code; we do not claim output for it. The model sets up the rule of this measurement; the model is not the tool itself. What matters is the consequence: because the rebuilt package is not the same as the previous build, the result obtained on the earlier tracks may not carry over to the new package automatically. In this case the team either retries only the track it was rejected on, or walks the path from the beginning. This choice is a policy; the model sets up both with a flag.

The versionCode = 13 line in the script says two things at once. The first is that a rebuild carries a new version code; per the previous lesson’s rule, the code can only increase. The second is that the result obtained on the tracks is tied not to the package itself but to the identity that the package’s code carries. When the identity changes, the earlier track’s approval does not carry over to the new package unless the team’s policy says otherwise. In the model the restartOnReject flag sets up this policy: if false, only the rejected track is repeated; if true, the path returns to the start.

The Bounding Measurement

The lesson’s overrun is within; the bounding measurement asks what this comfortable result rests on. If a rebuild starts from the beginning, every rejection also recounts the earlier tracks. We measure the same eight builds with two separate rules and add a ninth that is rejected twice on the same track.

// Main.kt — a rebuild walks the path from the start, the steps are counted again
fun main() {
    val path = trackPath()
    val builds = sampleBuilds()
    val budget = Budget("step", 5)

    val retry = builds.map { promotionSteps(path, it, false) }
    val restart = builds.map { promotionSteps(path, it, true) }
    println("retry on same track: $retry")
    println("restart from start: $restart")
    Overrun("same-track", budget, retry).line()
    Overrun("restart", budget, restart).line()

    val twice = Build("build-i", listOf("closed", "closed"))
    val withTwice = builds + twice
    Overrun("same-track-two-rejections", budget, withTwice.map { promotionSteps(path, it, false) }).line()
    Overrun("restart-two-rejections", budget, withTwice.map { promotionSteps(path, it, true) }).line()
}

retry on same track: [4, 4, 4, 4, 5, 5, 5, 4]
restart from start: [4, 4, 4, 4, 6, 7, 5, 4]
same-track: budget=step<=5 | over=0/8 -> within
restart: budget=step<=5 | over=2/8 -> minority
same-track-two-rejections: budget=step<=5 | over=1/9 -> minority
restart-two-rejections: budget=step<=5 | over=3/9 -> minority

On the path retried on the same track the overrun came out within; under the rule where the path is walked from the beginning it came out minority. On the path that restarts, the build rejected on the closed track took six steps and the one rejected on the open track took seven; the build rejected on the internal track stayed at five, because at the moment it was rejected the path was already at its beginning. The further along the rejection happens, the more steps are counted from the start.

The ninth build changes the table too. A build rejected twice on the same track takes six steps with the retry and eight steps under the rule that restarts. The category stayed minority in both cases, but the number of overrunning samples went from one to three. The rule counts the number and does not see the magnitude: one build exceeding by one step and another by three give the same category.

DI4. The model takes a rejection as an input and assumes the fix will succeed. In reality a fix can bring a new bug; the model does not set this up. The number of rejections is also a hand-written list, not a rule.

Skipping a Track

The shortest way to reduce the number of steps is to remove a track from the path. A path that skips the closed track consists of three stops, and a clean build finishes in three steps. But a track’s job is not to count steps; it is to catch a bug that would be caught there. The measurement below sets up the path as internal, open and production and lowers the budget to three steps. It also lists which builds would be rejected on the closed track, that is, which would have been caught had that track been on the path.

// Main.kt — a path that skips the closed track reduces steps but misses the rejection that track would catch
fun main() {
    val builds = sampleBuilds()
    val skipped = listOf("internal", "open", "production")
    val samples = builds.map { promotionSteps(skipped, it, false) }
    println("steps: $samples")
    val missed = builds.filter { build -> build.rejectedAt.any { it !in skipped } }
    println("missed rejections: ${missed.map { it.label }}")
    Overrun("closed-skipped", Budget("step", 3), samples).line()
}

steps: [3, 3, 3, 3, 3, 4, 4, 3]
missed rejections: [build-e]
closed-skipped: budget=step<=3 | over=2/8 -> minority

The four-track path had come out all at the three-step budget; the three-track path came out minority at the same budget. But the gain came at a cost: build-e would have been rejected on the closed track, and because the path did not include that track the rejection was never seen, and the build moved on to the open track without being fixed. The number of steps dropped; the stop where the bug is caught disappeared.

DI5. The model does not assume that a rejection on a skipped track will be caught on another track, and it shows this with the missed rejections list. But it does not set up whether the bug will appear later on the open track or in production, or what it would cost if it did.

Common Misconceptions

“The number of tracks is a measure of quality.” The number of tracks is the number of stops a build passes through. A build passing through many tracks does not show it was well tested; the quality of testing depends on the audience on the track and on the feedback.

“A rejected build is a bad build.” A rejection is a track doing its job. Tracks exist to find bugs before they reach production.

“Removing a track is enough to meet the budget.” Removing a track lowers the number of steps and can bring the measurement inside the budget or near it. But the bug that the track would have caught goes uncaught. The measurement above showed this on one build: the overrun decreased, and the missed rejection was one.

“A rebuild carries over the result of the earlier tracks.” It does not, because it is a new package and carries a new version code. Whether it carries over is the team’s policy; in this lesson both policies were measured separately.

“The step count gives the duration.” It does not. A step can take a day or a few hours; the model does not measure duration.

“A build that passes the internal track also passes the open track.” It may not. As the audience grows, the variety of devices and usage grows; a bug that never shows up on the internal track appears only in a wider audience. The order of the tracks follows a growing audience for this reason.

“Skipping tracks only saves time.” It does save time, but the bug the track would have caught also goes to production. In this lesson the path that skips the closed track met the step budget and the number of missed rejections rose to one; the gain and the loss were seen in the same table.

“The order of the tracks is a matter of the team’s taste.” The order determines whose hands a build passes through before it reaches whom. Changing the order means showing a build earlier to a less closely watched audience. This is a design decision, and the measurement itself does not recommend an order.

“The amount of feedback is the success of a track.” A lot of feedback can mean a lot of bugs, or it can mean the audience is engaged. The model does not measure the amount of feedback; it measures the number of steps.

“If the overrun decreased, the process has improved.” The overrun is counted only against a budget. If the budget is loosened or a track is removed, the overrun decreases by itself; whether the process has improved is understood by looking at which bugs are still caught before production. For this reason the overrun count and the number of missed rejections should be read side by side. Deciding by looking at a single number is not hearing what the other number would say.

“Every build must pass through all four tracks.” This is a team policy, not a rule. A small fix and a large feature need not follow the same path; but every decision that shortens the path brings with it a list of bugs that will go uncaught. Writing this list down and knowing who accepted it is the precondition for shortening the path.

Summary

• This lesson’s overrun: within. None of the eight builds exceeded the five-step budget; the three builds rejected once stayed at exactly five.
• The release path consists of the internal, closed, open and production tracks, and entering each track counted as one step; for this reason no build can fall below four steps.
• The same eight builds came out within, minority and all at budgets of five, four and three steps.
• When a rejected build walks the path from the beginning, the steps rose to six and seven, and the overrun went from within to minority.
• Removing the closed track took the three-step budget from all down to minority, but a rejection that would have been caught on the closed track was missed.
• The ninth build, rejected twice, raised the number of overrunning samples from one to three; the category stayed minority because the rule does not see the magnitude.

Next Step

The build has passed through the tracks and is ready to go to production, but going to production does not have to mean going to all users at once. The next lesson, Staged Rollout, widens the release stage by stage and compares the crash count at each stage with a threshold.