Crash Reporting and Analytics

When a released version crashes for a user, the way to find out is a crash report. The report carries the trace of the crashed session back to the team. Analytics, on the other hand, counts what the user does in sessions that do not crash. Both are counters, and counters mislead when they are read without knowing where a fault lives.

The measuring question of the course is always the same: when the same check is applied to the correct version and to the faulty version, do the two answers differ. A check where the correct version passes and the faulty one fails is separating, one where both pass is blind, one where both fail is noisy, and one where the correct version fails and the faulty one passes is inverted. In the output they are written as separating, blind, noisy, and inverted. The core comes in unchanged, as in the earlier lessons.

// Separation.swift — classifies whether a check tells the correct version from the faulty one
struct Contrast<Output: CustomStringConvertible> {
    let label: String
    let correct: Output
    let faulty: Output
    let passesCorrect: Bool
    let passesFaulty: Bool

    var separation: String {
        switch (passesCorrect, passesFaulty) {
        case (true, false): return "separating"
        case (true, true): return "blind"
        case (false, false): return "noisy"
        case (false, true): return "inverted"
        }
    }

    func describe() -> String {
        let onCorrect = passesCorrect ? "pass" : "fail"
        let onFaulty = passesFaulty ? "pass" : "fail"
        print("\(label) | correct=\(correct)/\(onCorrect) faulty=\(faulty)/\(onFaulty) -> \(separation)")
        return separation
    }
}

func contrast<Output>(_ label: String, correct: Output, faulty: Output, check: (Output) -> Bool) -> Contrast<Output> {
    Contrast(label: label, correct: correct, faulty: faulty, passesCorrect: check(correct), passesFaulty: check(faulty))
}

func countSeparations(_ results: [String]) {
    for name in ["separating", "blind", "noisy", "inverted"] {
        print("\(name): \(results.filter { $0 == name }.count)")
    }
}

This lesson’s fault lives in only one language configuration: the application crashes for users whose date format cannot be resolved, and works in the other configurations. The crash itself does not happen in the model; the model keeps the session and crash counts for each configuration and reads the numbers like a report.

The Report Model

ConfigTally is the session and crash count of one configuration. Telemetry carries three configurations and a single analytics event name; it summarizes itself as sessions, crashes, and event. overallPasses looks over all sessions at whether the crash-free ratio stays at the threshold, and configPasses looks at the same thing over a single configuration. dashboardCount returns the record the dashboard counts with a given event name.

// Crash.swift — session and crash counts per configuration, a single event name
struct ConfigTally {
    let config: String
    let sessions: Int
    let crashes: Int
}

struct Telemetry: CustomStringConvertible {
    let tallies: [ConfigTally]
    let eventName: String

    var sessions: Int { tallies.reduce(0) { $0 + $1.sessions } }
    var crashes: Int { tallies.reduce(0) { $0 + $1.crashes } }

    var description: String {
        "sessions=\(sessions) crashes=\(crashes) event=\(eventName)"
    }
}

func crashFree(sessions: Int, crashes: Int, threshold: Int) -> Bool {
    (sessions - crashes) * 100 >= sessions * threshold
}

func overallPasses(_ telemetry: Telemetry, threshold: Int) -> Bool {
    crashFree(sessions: telemetry.sessions, crashes: telemetry.crashes, threshold: threshold)
}

func configPasses(_ telemetry: Telemetry, config: String, threshold: Int) -> Bool {
    telemetry.tallies.filter { $0.config == config }.allSatisfy {
        crashFree(sessions: $0.sessions, crashes: $0.crashes, threshold: threshold)
    }
}

func dashboardCount(_ telemetry: Telemetry, name: String) -> Int {
    telemetry.eventName == name ? telemetry.sessions : 0
}

let crashFreeThreshold = 90
let correctTelemetry = Telemetry(
    tallies: [ConfigTally(config: "en", sessions: 40, crashes: 0),
              ConfigTally(config: "tr", sessions: 30, crashes: 0),
              ConfigTally(config: "ja", sessions: 10, crashes: 0)],
    eventName: "order-completed")
let faultyTelemetry = Telemetry(
    tallies: [ConfigTally(config: "en", sessions: 40, crashes: 0),
              ConfigTally(config: "tr", sessions: 30, crashes: 0),
              ConfigTally(config: "ja", sessions: 10, crashes: 4)],
    eventName: "order-completed")

DS21. The session and crash counts are the model’s own numbers; they do not show the crash rate of a real application. The ninety percent threshold is also the lesson’s rule, not the threshold of a platform or a service.

Three Checks

The first check looks at all sessions. The second looks only at the counter of the ja configuration. The third looks for the record the dashboard still counts under the old name, because the event was renamed from order-placed to order-completed in the version in question. The event is sent under the new name in both versions.

// main.swift — total counter, configuration counter, old event name
let overall = contrast("total-crash-free-sessions-at-threshold", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: crashFreeThreshold) }.describe()
let japanese = contrast("ja-config-crash-free-at-threshold", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "ja", threshold: crashFreeThreshold) }.describe()
let oldName = contrast("dashboard-has-old-event-name", correct: correctTelemetry, faulty: faultyTelemetry) { dashboardCount($0, name: "order-placed") > 0 }.describe()

countSeparations([overall, japanese, oldName])

total-crash-free-sessions-at-threshold | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind
ja-config-crash-free-at-threshold | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating
dashboard-has-old-event-name | correct=sessions=80 crashes=0 event=order-completed/fail faulty=sessions=80 crashes=4 event=order-completed/fail -> noisy
separating: 1
blind: 1
noisy: 1
inverted: 0

The total check came out blind. In the faulty version four sessions crash, but within eighty sessions these four do not have to fall below the threshold. The check looks at something correct, only it does not look at the place where the fault lives. The per-configuration check separates the fault from the same data, because the four crashed sessions are concentrated in a small configuration. Within an average, the crash of a small group of users is lost.

The dashboard check came out noisy. Because the event was renamed, the dashboard that counts under the old name sees zero in both versions and fails. A red dashboard does not show the crash; it shows that the dashboard does not know the new name.

Reporting Configuration

The crash reporting service and symbolication are not measured in this lesson. Both remain SOURCE: the configuration says that reporting is on and that symbols are uploaded.

# telemetry.yaml — reporting configuration (SOURCE, not measured)
crash_reporting:
  enabled: true
  upload_symbols: true
analytics:
  events:
    - order-completed

# symbol upload step — SOURCE
# 1. symbol files are extracted from the archive
# 2. symbol files are uploaded to the reporting service
# 3. the service makes the incoming crash trace readable with these files

A crash trace that arrives without symbols consists of address lines and does not say in which function the crash happened. For this reason the symbol upload step is a step that turns no step of the pipeline red, but that makes the report useless if it is forgotten.

DS22. With what delay and at what rate a crash report reaches the team was not measured. The reporting service, symbolication, and the real behavior of the dashboard were not measured either.

The Bounding Measurement

The blindness of the total check depends on the threshold. Below, the same two versions are checked with changing values of the threshold, with the tr configuration that does not crash, and with the new event name.

// main.swift — when the threshold and the configuration change
let strictOverall = contrast("total-threshold-96", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: 96) }.describe()
let turkish = contrast("tr-config", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "tr", threshold: crashFreeThreshold) }.describe()
let newName = contrast("dashboard-has-new-event-name", correct: correctTelemetry, faulty: faultyTelemetry) { dashboardCount($0, name: "order-completed") > 0 }.describe()
let strictJapanese = contrast("ja-config-threshold-100", correct: correctTelemetry, faulty: faultyTelemetry) { configPasses($0, config: "ja", threshold: 100) }.describe()

countSeparations([strictOverall, turkish, newName, strictJapanese])

total-threshold-96 | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating
tr-config | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind
dashboard-has-new-event-name | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/pass -> blind
ja-config-threshold-100 | correct=sessions=80 crashes=0 event=order-completed/pass faulty=sessions=80 crashes=4 event=order-completed/fail -> separating
separating: 2
blind: 2
noisy: 0
inverted: 0

When the threshold goes up from ninety to ninety-six, the total check becomes separating. The check that looks at the tr configuration, which does not crash, is blind again, because the fault does not live there. The dashboard that counts under the new name passes on both versions and is blind: the event the dashboard counts correctly knows nothing about the crash.

The limits are clear. Separation is not the whole selection criterion of an architecture, a test suite, or a release process. Cost, the size of the team, and the sustainability of a structure over time were not measured in this course and were not reduced to a number; they are discussed in prose.

DS23. The thresholds are the lesson’s rule. Which threshold is right depends on how much crashing the team tolerates for its users, and this lesson does not say it.

Course Wrap-Up

The nineteen lessons of the course stand side by side in the table below, each with its own fault. The separation column was taken from the decision in each lesson’s own ## Summary section.

Lesson	Fault	Separation	Bounding measurement
Architectural Patterns	the discount is applied twice	all four (2/2/1/1)	when the percentage changes, the same check is blind, then separating
The Coordinator Pattern	wrong screen after payment	blind	when the fault shifts to an intermediate step, only the last screen is blind
Unidirectional Data Flow	the reducer produces a wrong state	separating	inverted when the recorded sample is taken from the faulty one
Dependency Injection	the list is deleted on an empty response	noisy	blind when the stub always returns full
Modularization	a feature module is tied to another feature	blind	rebuild budget: one separating, one blind
Unit Tests	the fee is still charged at the threshold	separating	inverted when the expected value is copied from the faulty one
UI Tests	the button does not send the order	noisy	noisy when looking before the screen settles
Test Plans and Coverage	the discount is calculated wrongly	blind	an expectation written in a single language is noisy
Profiling Tools	a cell does more work than needed	separating	by budget: noisy, separating, blind
Memory Leak Detection	the closure holds the object strongly	separating	blind when the counter counts the screen
Launch and Scrolling Performance	extra work at launch and in scrolling	inverted	a renewed baseline is blind
Code Quality Tools	force unwrapping crashes on empty input	blind	a strict rule is noisy
Signing and Provisioning Profiles	the package does not carry the notification entitlement	inverted	the same check gives three separations on three inputs
Continuous Integration	the cart total skips the last item	noisy	when the step order changes, the separation changes
Release Automation	the build number does not increase	separating	noisy or inverted depending on the source of the last number
Beta Distribution	the flag is open outside the audience	blind	inverted for a newly invited tester
The Store Review Process	the permission description is empty	noisy	the length item is blind on a description of spaces
The Store Listing	the screenshot shows the old interface	inverted	separating with the correct reference
Crash Reporting and Analytics	crashes in only one language	blind	separating when the threshold changes

The separation column of the table writes the opening lesson as all four and the other eighteen lessons with a single separation. The measurement below counts these eighteen decisions; the last decision comes from this lesson’s own total check. Under it, the six checks of the core are put separately.

// main.swift — the distribution of the eighteen lessons other than the opening one
let ownLesson = contrast("wrap-up", correct: correctTelemetry, faulty: faultyTelemetry) { overallPasses($0, threshold: crashFreeThreshold) }.separation
let remaining = ["blind", "separating", "noisy", "blind",
                 "separating", "noisy", "blind", "separating", "separating", "inverted", "blind",
                 "inverted", "noisy", "separating", "blind", "noisy", "inverted",
                 ownLesson]
print("opening-excluded-eighteen")
countSeparations(remaining)
print("core-six-checks")
countSeparations(["separating", "separating", "blind", "blind", "noisy", "inverted"])

opening-excluded-eighteen
separating: 5
blind: 6
noisy: 4
inverted: 3
core-six-checks
separating: 2
blind: 2
noisy: 1
inverted: 1

The distribution of the eighteen lessons other than the opening one is five separating, six blind, four noisy, and three inverted. The 2/2/1/1 distribution of the core is a different thing: the core counts the six checks in the first lesson, while this distribution counts the separation of eighteen lessons. What should be read from the table is not the distribution itself, but that the same kind of check can fall into all four separations on different faults. Blind and inverted together come out in nine lessons; separating stays at five.

M30 closes here: six courses were completed, namely the Swift language, the platform and tools, declarative interfaces, imperative interfaces and bridging, data and networking, and architecture and release.

Summary

• If the crash counter is read over the total of all sessions, a fault that crashes in only one language configuration is lost in the total.
• The check that splits the crash counter per configuration separates the fault from the same data.
• A dashboard that counts a renamed event under its old name sees zero in both versions and becomes noisy.
• When the threshold of the total check is tightened, the same check becomes separating; being blind is a shared property of the check and the threshold, not of the check alone.
• Across the course’s nineteen lessons the separation spreads over four categories; no kind of check stays in a single category.
• This lesson’s separation: blind.

Next Step

This is the last lesson of the course. The question to ask when writing a new check does not change: apply the check to the correct version and to the faulty version, and read the two answers side by side.