---
title: 'Post-Release Monitoring'
source: 'https://academia.sh/en/courses/testing-performance-and-release/post-release-monitoring'
course: 'Testing, Performance and Release'
language: en
updated: '2026-10-21T05:02:42+00:00'
license: 'CC BY-SA 4.0'
objectives:
    - "Measures each day's crash count in the monitoring window as a separate sample"
    - 'Explains how the jump in the first days leaves the overrun at minority'
    - 'Shows that summing the same counts daily or weekly decides the overrun'
    - "Compares the overrun of the course's 19 lessons"
---

# Post-Release Monitoring

Compares each day's crash count in the monitoring window after release with the error threshold. Measures how the jump in the first days leaves the overrun at minority, and how the same counts change category when summed weekly and the sample count drops; then gathers the course's 19 lessons in one table.

The previous lesson took the app through review and delivered it to users; `Post-Release Monitoring` compares the crash counts of the days after release with a budget. A release is not passing through a gate but the opening of a window: while the build is in real users' hands, the team waits for data to arrive for a period called the **monitoring window**. This lesson treats each day of the window as a measurement.

The course's question is the same here: **in how many samples does this measurement exceed its budget?** The measurement is a day's crash count (`crashCount`), the budget is the error threshold the team accepts in a day, and each day of the window is a sample.

```kotlin
// Overrun.kt — classifies in how many samples a measurement exceeds its budget
class Budget(val name: String, val limit: Int)

class Overrun(val label: String, val budget: Budget, samples: List<Int>) {
    val total: Int = samples.size
    val over: Int = samples.count { it > budget.limit }
    val verdict: String = when {
        over == 0 -> "within"
        over == total -> "all"
        over * 2 < total -> "minority"
        else -> "majority"
    }

    fun line() {
        println("$label: budget=${budget.name}<=${budget.limit} | over=$over/$total -> $verdict")
    }
}

fun countOverruns(overruns: List<Overrun>) {
    for (name in listOf("within", "minority", "majority", "all")) {
        println("$name: ${overruns.count { it.verdict == name }}")
    }
}
```

`Overrun` counts the samples that strictly exceed the limit and gives the prevalence with four names: within, minority, majority, all. The rule sees how many days overrun; whether a day carries one or ten crashes more is the same for every day that overruns.

## Distinction: Report or Threshold

In the earlier part of the course, the Analytics and Crash Reporting lesson counted in how many copies a crash report reaches its recipient. That lesson measured the delivery of the report. This lesson does not measure delivery: it assumes the reports arrive and counts how many times over the days the threshold was exceeded. One examines the path of an event, the other the relation of a number to a budget.

## The Platform Piece That Shows a Crash

A crash report is the record produced at the moment the app stops with an unexpected error. The record may also carry an Application Not Responding (ANR) condition as a separate type. During monitoring, teams usually look at the crash-free session rate; because this lesson measures a count and not a rate, we use only the raw count.

```kotlin
// Platform code: CrashReporting
val reporter = crashReportingFramework.instance()
reporter.setCustomKey("release_window_day", dayIndex)
reporter.recordException(error)
```

This snippet is a platform call; we claim no output. The model sets up the rule for this measurement; the model is not the tool itself. In the model a session ends either as `crash` or as `sound`; the extra information the framework collects, the stack, or the ANR type remains outside the model.

## Modeling the Monitoring Window

The helper file builds each day of the window from ten sessions. Because the days are the same size, the crash counts are directly comparable.

```kotlin
// MonitoringModel.kt — models the monitoring window and the crash count per day
class ReleaseDay(val label: String, val outcomes: List<String>)

fun crashCount(day: ReleaseDay): Int = day.outcomes.count { it == "crash" }

fun releaseWindow(crashes: List<Int>): List<ReleaseDay> =
    crashes.mapIndexed { index, crashed ->
        ReleaseDay("day-${index + 1}", List(10) { if (it < crashed) "crash" else "sound" })
    }

fun launchWindow(): List<Int> = listOf(6, 5, 4, 2, 2, 1, 1, 2, 1, 1, 2, 1, 1, 1)

fun weeklyTotals(counts: List<Int>): List<Int> = counts.chunked(7).map { it.sum() }
```

`launchWindow` keeps the first days high and the rest low. Crashes cluster in the first days after release: in the first hours of a release come the most eager users, the sessions that try the most different paths.

## Daily Measurement

Set the error threshold at three crashes a day.

```kotlin
// Main.kt — each day of the monitoring window is measured, the error threshold is three crashes a day
fun main() {
    val days = releaseWindow(launchWindow())
    val samples = days.map { crashCount(it) }
    println("crashes: $samples")
    Overrun("daily-crash", Budget("crash", 3), samples).line()
}
```

```text
crashes: [6, 5, 4, 2, 2, 1, 1, 2, 1, 1, 2, 1, 1, 1]
daily-crash: budget=crash<=3 | over=3/14 -> minority
```

Three of the fourteen days exceeded the threshold: the first three days carried six, five and four crashes respectively. Because fewer than half exceeded, the answer is **minority**. This is the lesson's overrun.

**DI6.** The model builds every day from the same ten sessions and counts a crash as one outcome per session. In reality the number of daily sessions grows as the release widens, and a user can repeat the same crash; the model sees neither. `launchWindow` is a hand-written input; the days' crash counts come out of the model through `days.map { crashCount(it) }`, but the numbers themselves do not come from a real monitoring.

## Separating the Jump

The difference of the first three days from the others does not show in the rule; the rule only says that three days exceeded. If we split the window in two and measure each part separately, the jump becomes visible.

```kotlin
// Main.kt — the window is split into the first three days and the later days, and measured
fun main() {
    val samples = releaseWindow(launchWindow()).map { crashCount(it) }
    val spike = Overrun("first-three-days", Budget("crash", 3), samples.take(3))
    val rest = Overrun("later-days", Budget("crash", 3), samples.drop(3))
    spike.line()
    rest.line()
    countOverruns(listOf(spike, rest))
}
```

```text
first-three-days: budget=crash<=3 | over=3/3 -> all
later-days: budget=crash<=3 | over=0/11 -> within
within: 1
minority: 0
majority: 0
all: 1
```

In the first three days every day exceeded, and in the following eleven days none did. The whole window came out minority because eleven calm days diluted the three jump days. This is not a problem but a property of the measure: as the window grows, the overrun value depends more on the length of the window than on the jump.

**DI7.** The three-day split is the split of a team that knows from the start where the jump ends. In real monitoring the end of the jump is not known; choosing the split point is the team's decision, and the model does not make this decision.

## The Bounding Measurement

What happens if the same fourteen days are summed weekly rather than daily? Teams often report with a single number, and that number is usually weekly.

```kotlin
// Main.kt — the same crash counts are summed weekly and measured with two separate budgets
fun main() {
    val samples = releaseWindow(launchWindow()).map { crashCount(it) }
    val weekly = weeklyTotals(samples)
    println("weekly: $weekly")
    val same = Overrun("weekly-same-budget", Budget("crash", 3), weekly)
    val scaled = Overrun("weekly-sevenfold", Budget("crash", 21), weekly)
    same.line()
    scaled.line()
}
```

```text
weekly: [21, 9]
weekly-same-budget: budget=crash<=3 | over=2/2 -> all
weekly-sevenfold: budget=crash<=21 | over=0/2 -> within
```

The same fourteen days were summed into two weeks: 21 crashes in the first week, 9 in the second. When the daily budget was left as it is, both weeks exceeded and the answer was **all**. When the budget was raised to seven times, that is 21, the first week also did not count as an overrun because 21 equals the limit, and the answer was **within**. The daily cut's answer of **minority** did not appear in either weekly measurement. With two samples, minority cannot be built: the condition `over * 2 < total` holds with 2 samples only for zero overruns, and zero overruns means within.

**DI8.** Taking the weekly total's budget as seven times the daily budget assumes the days are evenly distributed. Crashes after release are not evenly distributed; the sevenfold budget therefore hides the first week's jump. Pulling the budget to another number would have given another overrun.

This measurement tests not a rule but a way of cutting: the lesson says how the sample is cut decides the overrun. In the daily cut there were fourteen samples and the overrun came out minority; in the weekly cut two samples remained and the overrun became tied to the sample count itself. This is why the course has the principle that the sample count should be at least four.

## The Budget Is a Decision

Three crashes a day is a number the team chose. A stricter threshold counts more days as overruns; a looser threshold also takes in the jump of the first days. Choosing the number is deciding which release you will halt. The model does not make this decision; it only says how many days the chosen number makes exceed.

## Common Misconceptions

**"The longer the window, the more accurate the data."** As the window grows, calm days multiply and the jump thins out; the overrun value changes, but this change is not the release improving, it is the way the sample is cut.

**"The weekly total is a better summary than the daily one."** The weekly total keeps the count but divides the sample count by seven. With two samples, one of the four categories, minority, cannot be built at all.

**"A release that is minority is healthy."** Minority says that fewer than half of the days exceeded; it does not say which days exceeded. In this lesson the days that exceeded were the first three, the moments when the release reached the most users.

## Closing Table

The course asked the same question in 19 lessons. Each lesson's overrun is the "This lesson's overrun:" line in that lesson's `## Summary` section; the measured unit of work and the bounding measurement are in the source lesson itself.

| Lesson | Measured unit of work | Overrun | Bounding measurement |
|---|---|---|---|
| Unit Tests | input validation step | within | at a budget of 5 steps, minority |
| Mocking | network call in a transaction | all | with a stub cache, calls drop to zero, within |
| On-Device Tests | work per test | all | with shared setup, minority |
| UI Tests | step to find a node | majority | with a test tag, a single step, within |
| Test Stability | wait for readiness | minority | with condition-based polling, within |
| Static Analysis | violations per module | majority | with a baseline file, minority |
| Startup Time | work until the first frame | minority | by deferring the dependency, within |
| Rendering Smoothness | items recomposed per frame | all | with a stable key, minority |
| Memory Management | retained object | majority | binding the listener to the lifecycle, within |
| Network and Battery Profiling | requests per round | within | with one-by-one sending, minority |
| Benchmarking | work per iteration | minority | with warm-up discarded, the new version is all |
| Code Shrinking and Obfuscation | classes remaining per module | within | with a broad keep rule, all |
| App Package Formats | files that go down to a configuration | all | in a split APK, within |
| Version Management | device's distance from the release | majority | stays the same when only the version name changes |
| Continuous Integration and Delivery | jobs that run per change | minority | with an unfiltered pipeline, all |
| Testing Tracks | steps until production | within | when a rejected build is counted from the start, minority |
| Staged Rollout | crashes per stage | minority | with a halt the sample count shrinks, all |
| Store Review and Policies | unused permissions per module | majority | with the permissions removed, within |
| Post-Release Monitoring | daily crash count | minority | with weekly summing the sample count drops |

Of the 19 lessons, 4 close within, 6 minority, 5 majority and 4 all. The core in the opening lesson gave a different distribution: the four measurements came out 1 within, 1 minority, 1 majority, 1 all. The two distributions do not count the same thing; the core counts four measurements, the table counts nineteen lessons.

## Summary

- This lesson's overrun: minority. Three of fourteen days, the first three days, exceeded the threshold of three crashes a day.
- The jump of the first days thins out when spread over the whole window: the first three days came out all, the following eleven days within.
- When the same counts are summed weekly, the sample count falls from fourteen to two and the category changes. How the sample is cut decides the overrun.
- This lesson counts not the delivery of the crash report but how many times over the days the reports exceeded the threshold.
- The crash reporting framework is platform code and is not measured in this lesson.

## Course Wrap-Up

Across testing, performance and distribution, the course asked a single question: in how many samples does this measurement exceed its budget? The table shows that all four separate answers to the same question were really given. The next curriculum, M32, moves to the subject area of cross-platform mobile development.