Staged Rollout

The previous lesson measured how a build passes through the tracks. A build that reaches the production track lands on users’ own devices; however well tested it is, some bugs show up only in real use. A staged rollout limits the cost of such a bug: the build is first opened to a small group of users, and if no problem is seen it widens to a larger group in the next stage.

The team watches the release at each stage. The monitoring window is the period during which a stage stays open and crashes are counted. If the crash count exceeds the error threshold, the team has two decisions: to halt the release, that is, not to move on to the next stage and to stop offering the version to new users, though devices that already received the version do not go back to the old one; or to roll back, that is, to republish the previous version with a higher version code. The stages are mentioned in order in this lesson; every team has its own stage layout.

The course’s question is the same here: in how many samples does this measurement exceed its budget? The measurement is the crash count (crashCount) in a stage, from a user group of the same size; the budget is the error threshold; each stage is a sample.

// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}

Overrun counts the samples that strictly exceed the limit and reports how widespread they are with four names: within, minority, majority, all. The rule sees how many stages exceed the threshold; whether a stage carries one crash too many or ten is the same for every sample that overruns.

The Code That Collects Crashes

When the app crashes, the system arrives at a default handler; teams route this handler to their own code to record the crash.

// Platform code: CrashRecorder
Thread.setDefaultUncaughtExceptionHandler { thread, error ->
    recordCrash(thread.name, error)
}

This fragment is a platform call; we do not claim output for it. The model sets up the rule of this measurement; the model is not the tool itself. In the model a crash is the outcome of a session: a session either ends sound or crashes.

The handler does not change the system’s crash path; it only ties the recording of the crash to the team’s code. After the handler takes its record, the app still terminates. This distinction matters for the measurement: the crash count is the number of crashes that could be recorded. A crash that could not be recorded at all, or a session that never crashed but ran in a broken way, does not enter the count. Since in the model a session is either crash or sound, both of these gaps stay outside the model.

Modeling the Stages

The helper file builds each stage as a group made of a fixed number of sessions. Because the stages are the same size, the crash counts can be compared directly; had the stages differed in size, the count would mislead.

// RolloutModel.kt — models the staged rollout and the crash count per stage
class Stage(val label: String, val outcomes: List<String>)

fun crashCount(stage: Stage): Int = stage.outcomes.count { it == "crash" }

fun stageNames(): List<String> = listOf(
    "first-stage", "second-stage", "third-stage",
    "fourth-stage", "fifth-stage", "final-stage",
)

fun rolloutStages(crashes: List<Int>): List<Stage> =
    stageNames().zip(crashes).map { (name, crashed) ->
        Stage(name, List(10) { if (it < crashed) "crash" else "sound" })
    }

fun releaseA(): List<Int> = listOf(1, 1, 2, 5, 3, 4)

fun decide(count: Int, limit: Int): String = if (count > limit) "halt" else "continue"

fun firstOverIndex(counts: List<Int>, limit: Int): Int {
    val index = counts.indexOfFirst { it > limit }
    return if (index == -1) counts.lastIndex else index
}

Each stage is built from ten sessions; releaseA gives the crash count per stage. Six stages means six samples.

Measuring the Stages

We set the error threshold at three crashes: the team does not accept more than three crashes in a stage.

// Main.kt — the crash count in each stage is measured, error threshold three crashes
fun main() {
    val stages = rolloutStages(releaseA())
    val samples = stages.map { crashCount(it) }
    for (stage in stages) {
        println("${stage.label}: crashes=${crashCount(stage)}/${stage.outcomes.size}")
    }
    println("crashes: $samples")
    Overrun("stage-crashes", Budget("crash", 3), samples).line()
}

first-stage: crashes=1/10
second-stage: crashes=1/10
third-stage: crashes=2/10
fourth-stage: crashes=5/10
fifth-stage: crashes=3/10
final-stage: crashes=4/10
crashes: [1, 1, 2, 5, 3, 4]
stage-crashes: budget=crash<=3 | over=2/6 -> minority

Two stages exceeded three crashes: the fourth stage carried five crashes, the final stage four. The fifth stage stopped at three and, being equal to the limit, did not count as an overrun. Since fewer than half of the six stages exceeded, the answer is minority. This is the lesson’s overrun.

DI3. The model builds each stage from ten sessions of the same size and counts a crash as one outcome per session. In reality stages differ in size, and one user’s many sessions can repeat a single crash; the model sees neither. The crash counts are hand-written inputs, not a measurement.

The Threshold Is a Decision

We measure the same six stages with four separate thresholds. When the threshold changes, the overrun changes; this is not a contradiction, it is what the measure is.

// Main.kt — the same six stages give four different overruns with four thresholds
fun main() {
    val samples = rolloutStages(releaseA()).map { crashCount(it) }
    val loose = Overrun("five-crashes", Budget("crash", 5), samples)
    val normal = Overrun("three-crashes", Budget("crash", 3), samples)
    val strict = Overrun("one-crash", Budget("crash", 1), samples)
    val zero = Overrun("zero-crashes", Budget("crash", 0), samples)
    listOf(loose, normal, strict, zero).forEach { it.line() }
    countOverruns(listOf(loose, normal, strict, zero))
}

five-crashes: budget=crash<=5 | over=0/6 -> within
three-crashes: budget=crash<=3 | over=2/6 -> minority
one-crash: budget=crash<=1 | over=4/6 -> majority
zero-crashes: budget=crash<=0 | over=6/6 -> all
within: 1
minority: 1
majority: 1
all: 1

The four thresholds gave four categories. The threshold that allows five crashes counted no stage as an overrun. The threshold that allows zero crashes, on the other hand, was exceeded by all six stages, because every stage had at least one crash. What value the threshold should have is a product decision; the model does not decide, it shows how the decision changes the overrun.

An Overrun Is Not a Decision

A distinction is needed here. A halt is not one of the four categories of the overrun; it is a decision made as a result of the overrun. The rule sees only the number and says within, minority, majority or all. Halting the release is the decision the team makes after seeing the overrun: when the threshold is exceeded in a stage, the release does not move on to the next stage.

For this reason the decision lives in a separate function. Overrun never says “halt”; decide makes a separate decision for each stage.

// Main.kt — the category and the decision are separate functions
fun main() {
    val limit = 3
    val stages = rolloutStages(releaseA())
    val samples = stages.map { crashCount(it) }
    val overrun = Overrun("stage-crashes", Budget("crash", limit), samples)
    overrun.line()
    for (stage in stages) {
        println("${stage.label}: decision=${decide(crashCount(stage), limit)}")
    }
    println("category: ${overrun.verdict}")
}

stage-crashes: budget=crash<=3 | over=2/6 -> minority
first-stage: decision=continue
second-stage: decision=continue
third-stage: decision=continue
fourth-stage: decision=halt
fifth-stage: decision=continue
final-stage: decision=halt
category: minority

The category is minority; the decision is to halt at the fourth stage. The two answer different questions: the category answers “how widespread is the overrun?”, the decision answers “what will we do?”. If the release is going to be halted the moment the first overrun is seen, the fifth and final stages are never opened and their counts are never measured.

The Bounding Measurement

This section asks when a category stops being readable. A halt cuts off the sampling of the later stages. The sample count shrinks and the category can change. We take three separate releases: in the first the overrun appears at the fourth stage, in the second at the second stage, in the third at the first stage. Each is measured with all six stages and in its form halted at the first overrun.

// Main.kt — halting at the first overrun reduces the sample count, the category can change
fun main() {
    val limit = 3
    val releases = listOf(
        "release-a" to releaseA(),
        "release-b" to listOf(1, 4, 1, 1, 2, 1),
        "release-c" to listOf(4, 1, 1, 1, 2, 1),
    )
    for ((name, crashes) in releases) {
        val full = rolloutStages(crashes).map { crashCount(it) }
        val halted = full.take(firstOverIndex(full, limit) + 1)
        Overrun("$name-full", Budget("crash", limit), full).line()
        Overrun("$name-halted", Budget("crash", limit), halted).line()
    }
}

release-a-full: budget=crash<=3 | over=2/6 -> minority
release-a-halted: budget=crash<=3 | over=1/4 -> minority
release-b-full: budget=crash<=3 | over=1/6 -> minority
release-b-halted: budget=crash<=3 | over=1/2 -> majority
release-c-full: budget=crash<=3 | over=1/6 -> minority
release-c-halted: budget=crash<=3 | over=1/1 -> all

In the first release four stages were sampled and the category stayed minority. In the second release only two samples remained, and the same release that was minority in the full measurement appeared as majority in the halted one. In the third release a single sample remained and the answer became all. The full measurement of all three releases is minority; what changed is not the behavior of the release but the number of samples measured.

From this comes the sample count rule: at least four samples are needed to draw a category from a measurement. Below four samples the category only tells that sample collection was stopped, not about the release itself. With two samples, one overrun is half of the samples. For this reason the category of a halted release is not read; the decision itself is read.

DI4. The model assumes the halt happens right at the first overrun. In reality teams may wait for a few overruns, or a person may assess the overrun; the model does not set up this delay. The model also does not know what will happen to the later stages of a halted release: if the halted stages are opened later, the measurement starts again.

Stage Size

In the third assumption we said the stages are the same size. A real release does not proceed this way: the first stage is small and the last stage covers most of the users. When the size changes, the raw crash count becomes misleading, because in a large stage the same crash rate means more crashes. The measurement below builds six stages with growing sizes and puts two measurements side by side: the raw crash count and the number of crashes per thousand sessions. The second is computed with integer division, so the rounding is downward. The budgets of the two measurements are in different units and are not derived from each other; we set three crashes for the raw count and fifty crashes per thousand sessions for the rate.

// Main.kt — in growing stages the raw crash count and the rate per thousand sessions give different answers
fun main() {
    val sizes = listOf(10, 20, 50, 100, 200, 500)
    val crashes = listOf(1, 1, 2, 5, 6, 40)
    val stages = stageNames().indices.map { index ->
        Stage(stageNames()[index], List(sizes[index]) { if (it < crashes[index]) "crash" else "sound" })
    }
    val counts = stages.map { crashCount(it) }
    val rates = stages.map { crashCount(it) * 1000 / it.outcomes.size }
    println("crashes: $counts")
    println("crashes per thousand sessions: $rates")
    Overrun("raw-count", Budget("crash", 3), counts).line()
    Overrun("rate", Budget("crash-per-thousand", 50), rates).line()
}

crashes: [1, 1, 2, 5, 6, 40]
crashes per thousand sessions: [100, 50, 40, 50, 30, 80]
raw-count: budget=crash<=3 | over=3/6 -> majority
rate: budget=crash-per-thousand<=50 | over=2/6 -> minority

In the raw count three stages exceeded the budget and the answer was majority; in the rate only two stages exceeded and the answer stayed minority. The difference shows in the first and the last stage. In the first stage a single crash means a hundred crashes per thousand sessions and does not show up in the raw count at all. In the last stage forty crashes look like a lot in the raw count, but divided by five hundred sessions they come to eighty crashes per thousand sessions. The fourth stage’s five crashes correspond to fifty crashes per thousand sessions and are not above the budget in either measurement.

The result is that the choice of measurement moves the category. The rule does not decide which measurement is right; the team sets the budget in a unit and counts in that unit.

DI5. The model counts a session as one crash outcome and computes the rate with integer division. In reality rates are fractional, and on small stages the fact that a single crash makes the rate so volatile carries a statistical uncertainty; the model does not measure this uncertainty. The sizes and the crashes are inputs.

Common Misconceptions

“A staged rollout prevents bugs.” It does not; it limits how many users a bug affects.

“If the crash count is low, the release is healthy.” The count only counts sessions that crashed. A version that does not crash but runs in a broken way does not show up in the count.

“A single crash in a small stage is unimportant.” If the stage is small, a single crash is a high rate; the raw count hides this. The first stage above corresponded to a hundred crashes per thousand sessions.

“Many crashes in a large stage is a bad sign.” Having many can be a result of being large. In the final stage forty crashes corresponded to eighty per thousand sessions, and it was only one of the two rates that exceeded the budget.

“A halt is an outcome.” A halt is not one of the categories, it is a decision. The same category can end in two different decisions on two teams.

“More stages make a release safer.” Each extra stage adds a window in which a crash can show up, but it also delays the completion of the release. The number of stages is a balancing decision; this lesson only measured the given stages and did not recommend an ideal number.

“The rate is always more correct than the raw count.” The rate takes size into account, but on a small stage even a single crash makes the rate jump. This was seen in the first stage above. For the rate to be reliable, enough sessions are needed; how many sessions are enough was not given in this lesson, because it was not measured.

“Crash monitoring ends when the release ends.” Crashes keep coming after the release has reached all users. Stages only raise the chance of catching a bug early; the closing of the monitoring windows does not say the version is free of problems.

“Setting the threshold once is enough.” The threshold is reconsidered according to the size of the release and the number of users. Putting the same number on every stage carries a different meaning as the stages grow; a raw-count threshold is exceeded sooner in a large stage, while a rate threshold is independent of size. For this reason which measure the threshold is tied to matters as much as the threshold itself. When the threshold changes, the results of the earlier stages are interpreted again too.

Summary

• This lesson’s overrun: minority. Two of the six stages exceeded the three-crash threshold; since fewer than half exceeded, the answer is minority.
• A staged rollout limits the cost of a bug; when an overrun is seen, the team halts the release or rolls it back.
• The same six stages came out within, minority, majority and all at thresholds of five, three, one and zero crashes.
• A halt is not a category but a decision made as a result of the overrun; the category is minority, the decision is to halt at the fourth stage.
• When the stage sizes grow, the raw crash count came out majority and the rate per thousand sessions came out minority; the choice of measurement moves the category.
• When fewer than four samples remain after a halt, the category tells that sample collection stopped, not about the release: it came out majority with two samples and all with one.

Next Step

The release widened stage by stage and each stage was compared with a threshold. But before it goes out there is one more obstacle: the app store’s review rules. The next lesson, Store Review and Policies, counts the permissions the app declares but does not use.