Analytics and Crash Reporting

In the previous lesson, a version was installed on the device. The way to know what a running app is doing is to send events: a screen opened, an item was added to a cart, the app crashed. These events do not go to the server the moment they are produced; they accumulate in a queue and are sent when the network is suitable. The question is the course’s question: how many copies of this sent event reached the receiver? The event is the reported event, and the receiver is the server.

// Arrival.kt — classifies how many copies of a sent event reached the receiver
class Signal(val key: String, val seq: Int)

class Arrival(val label: String, val sent: Signal, received: List<Signal>) {
    val copies: Int = received.count { it.seq == sent.seq }
    val verdict: String = when {
        copies > 1 -> "duplicated"
        copies == 1 -> "once"
        received.any { it.key == sent.key && it.seq > sent.seq } -> "superseded"
        else -> "lost"
    }

    fun line() {
        println("$label: key=${sent.key} | copies=$copies -> $verdict")
    }
}

fun countArrivals(arrivals: List<Arrival>) {
    for (name in listOf("once", "duplicated", "superseded", "lost")) {
        println("$name: ${arrivals.count { it.verdict == name }}")
    }
}

Event, Report, and Session

An analytics event is the record of a step the user took in the app: a screen opened, a button pressed. A crash report is the record produced when the app stops with an unexpected error; it carries the error type and the stack. A crash-free session is a session that ends without a crash, and it forms the denominator of the crash rate calculation. All three travel the same way: they enter a queue and are sent to the server.

The hook that produces the crash report lives in the platform’s crash reporting framework. When the app crashes, a handler is called before the process closes, and the report is added to the queue.

// Platform code: Thread.UncaughtExceptionHandler
val previous = Thread.getDefaultUncaughtExceptionHandler()
Thread.setDefaultUncaughtExceptionHandler { thread, error ->
    reportQueue.enqueue(CrashReport(thread.name, error))
    previous?.uncaughtException(thread, error)
}

Emptying the queue is done in the background, while the network is connected.

// Platform code: WorkManager
val constraints = Constraints.Builder()
    .setRequiredNetworkType(NetworkType.CONNECTED)
    .build()
val request = OneTimeWorkRequestBuilder<UploadWorker>()
    .setConstraints(constraints)
    .build()
WorkManager.getInstance(context).enqueue(request)

The two pieces are platform code, and we make no claim about output, duration, or counts here. The model builds the platform’s rule; the model is not the platform itself. In most clients, retry is done with exponential backoff: the wait grows after each failed attempt. This wait changes the interval between attempts, not their number; the model has no wait.

If the Response Is Lost

A send has two halves: the request reaches the server, and the server returns an acknowledgement response. If the response is lost on the way, the client cannot know whether the request arrived, and it resends the same event until an acknowledgement comes. In the model, each event carries how many times its acknowledgement will be lost; every send reaches the server, and only the acknowledgement is lost.

// Report.kt — an event whose acknowledgement is lost is resent until an acknowledgement arrives
class Report(val signal: Signal, val lostAcks: Int = 0)

fun retryUntilAck(reports: List<Report>): List<Signal> =
    reports.flatMap { r -> List(r.lostAcks + 1) { r.signal } }

fun boundedQueue(capacity: Int, queued: List<Signal>): List<Signal> = queued.takeLast(capacity)

fun dedupeById(received: List<Signal>): List<Signal> =
    received.distinctBy { it.key to it.seq }

// Probe.kt — the receiver counts the list that reached the server for the sent events
fun probe(sent: List<Signal>, received: List<Signal>): List<Arrival> =
    sent.mapIndexed { i, s -> Arrival("event-${i + 1}", s, received) }

If the Acknowledgement Comes Every Time

Four events are sent: two analytics events, a crash report, and a crash-free session record. None of their acknowledgements is lost.

// Main.kt — if the acknowledgement arrives, each event is sent once
fun main() {
    val open = Signal("event.open", 1)
    val cart = Signal("event.cart", 2)
    val crash = Signal("crash.session", 3)
    val clean = Signal("session.clean", 4)
    val reports = listOf(Report(open), Report(cart), Report(crash), Report(clean))
    val received = retryUntilAck(reports)
    val arrivals = probe(listOf(open, cart, crash, clean), received)
    arrivals.forEach { it.line() }
    countArrivals(arrivals)
}

event-1: key=event.open | copies=1 -> once
event-2: key=event.cart | copies=1 -> once
event-3: key=crash.session | copies=1 -> once
event-4: key=session.clean | copies=1 -> once
once: 4
duplicated: 0
superseded: 0
lost: 0

All four events arrived as one copy each. There was no need to retry.

If the Same Event Is Retried

Now the cart event’s acknowledgement is lost once, and the crash report’s acknowledgement is lost twice. The client sends until an acknowledgement comes, and the server takes each send as a separate copy.

// Main.kt — an event whose acknowledgement is lost is retried without deduplication
fun main() {
    val open = Signal("event.open", 1)
    val cart = Signal("event.cart", 2)
    val crash = Signal("crash.session", 3)
    val clean = Signal("session.clean", 4)
    val reports = listOf(Report(open), Report(cart, 1), Report(crash, 2), Report(clean))
    val received = retryUntilAck(reports)
    val arrivals = probe(listOf(open, cart, crash, clean), received)
    arrivals.forEach { it.line() }
    countArrivals(arrivals)
}

event-1: key=event.open | copies=1 -> once
event-2: key=event.cart | copies=2 -> duplicated
event-3: key=crash.session | copies=3 -> duplicated
event-4: key=session.clean | copies=1 -> once
once: 2
duplicated: 2
superseded: 0
lost: 0

This lesson’s delivery: duplicated. The event whose response was lost reached the server a second and a third time. The client made no mistake in this behavior: retrying a send for which no acknowledgement came is safer than sending nothing. The cost is extra copies at the server. If a number such as the crash rate is computed at the server from copies, crashed sessions are counted more than they are.

A distinction is needed here. A receiver that processes the same copy twice has not duplicated the event. Duplicated means a copy reaching the receiver a second time; the receiver processing it twice does not change the number of copies. In this lesson, copies are counted as they reach the server, and what happens inside the server is not looked at.

SE1. The model assumes that every send reaches the server and only the acknowledgement is lost. On the platform, the request can also be lost on the way; in that case a retry does not duplicate the event, it delivers it for the first time.

SE2. The model continues the retry until an acknowledgement comes and sets no upper limit. On the platform, the number of attempts and the wait come from the work policy.

Why the Queue Is Bounded

The queue not being unbounded is a design necessity. The device’s memory and storage are limited; if the network is absent for days and the app writes every screen opening to the queue, the queue can become larger than the app itself. That is why clients set an upper limit, and when the limit is reached they choose one of two ways: drop the oldest, or not accept the newest.

The results of the two choices are not the same. A queue that drops the oldest sends the recent past when the network returns and loses the distant past; a queue that rejects the newest keeps the distant past but does not know the current state when the network returns. For crash reports, the second choice is usually less harmful, because the oldest crash is often the first and the most explanatory one. The model builds the first choice: takeLast(capacity) keeps the newest events.

The loss appears as the cost of this design, and the client cannot see it by itself. Because no send was ever attempted for a dropped event, no acknowledgement is awaited, no error is produced, nothing is recorded. The receiver knows nothing either; only those that arrived are visible to it. The shortfall in the count is noticed only if the number of events produced is compared with the number of events that arrived.

Retry was measured above and the queue bound is measured below; in a real client both work on the same event stream. Some of the events entering the queue are dropped, the rest are sent, and some of those sent have their acknowledgements lost. In the measurement below, five events enter a queue of three places; the acknowledgement of the payment event remaining in the queue is lost once.

The Bounding Measurement

Duplication is not the only risk of retry. If the network is absent for a long time, events accumulate in the queue, and the queue’s length is bounded. When the queue is full, the oldest events are dropped to make room for the new one. Five events accumulate, and the queue takes three.

// Main.kt — when the bounded queue is full, the oldest events are dropped
fun main() {
    val sent = listOf(
        Signal("event.open", 1),
        Signal("event.cart", 2),
        Signal("crash.session", 3),
        Signal("event.payment", 4),
        Signal("session.clean", 5),
    )
    val received = boundedQueue(3, sent)
    val arrivals = probe(sent, received)
    arrivals.forEach { it.line() }
    countArrivals(arrivals)
}

event-1: key=event.open | copies=0 -> lost
event-2: key=event.cart | copies=0 -> lost
event-3: key=crash.session | copies=1 -> once
event-4: key=event.payment | copies=1 -> once
event-5: key=session.clean | copies=1 -> once
once: 3
duplicated: 0
superseded: 0
lost: 2

The oldest two events were dropped and never reached the server. Because their keys are different, there is no other copy that displaced them; the rule counts them as lost. Because the queue throws out the oldest, the first opening events are lost and the newest events remain. This choice is a policy: if a crash report is old enough, it can be dropped too.

Now assume the server deduplicates by event identifier. The earlier retry scenario plays out as it did, but the server reduces copies arriving with the same key and sequence to a single record.

// Main.kt — a deduplicating server preserves the outcome, but the delivery remains duplicated
fun main() {
    val open = Signal("event.open", 1)
    val cart = Signal("event.cart", 2)
    val crash = Signal("crash.session", 3)
    val clean = Signal("session.clean", 4)
    val reports = listOf(Report(open), Report(cart, 1), Report(crash, 2), Report(clean))
    val received = retryUntilAck(reports)
    val arrivals = probe(listOf(open, cart, crash, clean), received)
    arrivals.forEach { it.line() }
    countArrivals(arrivals)
    println("arrived: ${received.size}")
    println("stored: ${dedupeById(received).size}")
}

event-1: key=event.open | copies=1 -> once
event-2: key=event.cart | copies=2 -> duplicated
event-3: key=crash.session | copies=3 -> duplicated
event-4: key=session.clean | copies=1 -> once
once: 2
duplicated: 2
superseded: 0
lost: 0
arrived: 7
stored: 4

The server reduced seven copies to four records, so the outcome is right: the crash rate is not inflated by copies. But the delivery is still duplicated: seven copies reached the server, and the rule counts the copies that arrive. Deduplication is a job inside the receiver; delivery is a copy reaching the receiver. Neither changes the other.

When retry and queue work together, both effects show in the same stream.

// Main.kt — dropped events are lost, an event whose acknowledgement is lost becomes duplicated
fun main() {
    val sent = listOf(
        Signal("event.open", 1),
        Signal("event.cart", 2),
        Signal("crash.session", 3),
        Signal("event.payment", 4),
        Signal("session.clean", 5),
    )
    val queued = boundedQueue(3, sent)
    val reports = queued.map { Report(it, if (it.seq == 4) 1 else 0) }
    val received = retryUntilAck(reports)
    val arrivals = probe(sent, received)
    arrivals.forEach { it.line() }
    countArrivals(arrivals)
}

event-1: key=event.open | copies=0 -> lost
event-2: key=event.cart | copies=0 -> lost
event-3: key=crash.session | copies=1 -> once
event-4: key=event.payment | copies=2 -> duplicated
event-5: key=session.clean | copies=1 -> once
once: 2
duplicated: 1
superseded: 0
lost: 2

Three different deliveries were seen in the same stream. The loss comes from the queue’s limit, the duplicated from the lost acknowledgement; the two are independent, and neither corrects the other. An event that does not enter the queue is never retried, so it never becomes duplicated; an event that is retried cannot have been dropped from the queue either, because sending is done for the events that remain in the queue.

SE3. The model does deduplication by the key and sequence pair. On the platform, the server may use an event identifier, a timestamp, or both together; the model builds only the first.

What the Server Has to Do

The receiver of this lesson is the server, and no one but it can correct the copies. Three jobs are listed.

• Give every event an identifier and deduplicate by identifier. The client produces the identifier, and the same identifier is sent on retry; when the server sees the same identifier a second time, it does not count the event. If the identifier is produced anew at every attempt, deduplication does not work, because the server sees two copies as two separate events.
• Know the deduplication window. The server cannot keep the identifiers it has seen forever. If the window is kept short, a late-arriving copy is counted as a new event; if it is kept long, memory swells. The window’s length must be greater than the client’s retry duration.
• Count the copies separately. Deduplication preserves the outcome but hides how many copies came. The retry rate is an indicator of a healthy network; if the copy count has risen, acknowledgements are being lost, and this, even if it does not change the outcome, is something that needs to be known.

To compare the copy count with a healthy network, the server needs a baseline: without knowing the proportion of events whose acknowledgements are not lost, an increase in copies cannot be interpreted.

The job on the client side is to produce an event once and send it with the same identifier. The moment the identifier is produced must be before the retry.

Common Misconceptions

Backoff changes the interval, not the count. Exponential backoff lets the client try without straining the network; but for a client that sends until an acknowledgement comes, the copy count is still one more than the number of lost acknowledgements. Backoff helps protect the server from copies arriving at the same time; it does not eliminate copies.

An event written to the queue is not counted as having reached the server. In the interval between production and arrival, the event is in the queue, and the process closing in that interval can destroy the event. The measurement assumes the queue stays in memory continuously; a non-persistent queue loses everything pending when the app closes.

There is no duplicated on a deduplicating server. There is. In the measurement, seven copies were reduced to four records, and the delivery remained duplicated 2. Deduplication corrects the outcome; it does not change the number of copies that arrived.

Retry prevents an event from being lost. It only prevents the loss of an event whose acknowledgement is lost. Because an event dropped from the queue was never sent, retry does not reach it; in the measurement, the two dropped events remained lost.

The bigger the queue, the safer. A big queue drops fewer events, but it increases the device’s memory use and the load to be sent when the network returns. The limit is not a safety; it is a balance.

An event’s identifier depends on the server looking at it. If only the client carries the identifier, the server cannot tell the copy apart. Deduplication is a contract applied by both sides together: the client sends the same identifier, and the server counts the same identifier once.

A crash report is counted like other events. It is counted, and that is why a crash report counted with copies inflates the crash rate. In the measurement, the crash report arrived as three copies; without deduplication, that session would have been counted as three crashes.

Summary

• This lesson’s delivery: duplicated. An event whose acknowledgement is lost, retried without deduplication, reaches the server as multiple copies.
• An analytics event records a step, a crash report records a crash, a crash-free session records a sound ending; all three go through the same queue.
• If the acknowledgement comes every time, all four events arrive once. With two events whose acknowledgements are lost, the result is duplicated 2 and once 2.
• When five events enter a queue of three places, the oldest two are dropped: lost 2, once 3.
• When retry and queue work together, lost 2, once 2, and duplicated 1 are seen; neither corrects the other.
• A server that deduplicates by event identifier reduces seven copies to four records, but the delivery is still duplicated 2. A receiver that processes the same copy twice has not duplicated the event.

Next Step

Events reach the server, and how the app behaves in production is visible. Changing behavior does not require releasing a new version every time. In the next lesson, the app fetches a configuration from the server; when the configuration is fetched twice before activation, the last value of the same key wins.