Crash and Usage Monitoring

The previous lesson covered a pack that did not arrive in the first download. The team could know that situation in advance: which pack is missing where can be read from the install. In a released build, much more is unknown. On which device did it crash, who uses which screen, which feature quietly does not work? This lesson covers collecting that information from the field, and it measures the condition of the collection itself.

The question of the course stays the same: how well does this feature work under this condition? This time the question is asked of the monitoring itself. How well does the mechanism that reports a crash work on a device that has lost the network at the moment of the crash?

One Telemetry from Two Platforms

Two platforms mean two separate crash mechanisms and two separate event sources. On iOS the system delivers diagnostic data to the application at the next launch; on Android the application registers a handler for the uncaught error. The format, the name, and the timing of the data differ. The team’s job is to close this difference at the collection point: both platforms produce a single event format with the same fields, so on the dashboard “crash” becomes one concept.

The unified format has to carry three things: the kind of the event, the identifier of the session, and the version of the build. Without the version, it is not known which release a crash belongs to. The staged rollout and rollback decisions from the earlier lessons use this link: if the crash reports of a version opened to a new bucket rise, the halt and the remote kill flag are decided with that data.

The Crash Report and Symbolication

A crash report carries the trace of the moment of the crash: which function went to which function. In a released build the names in this trace are shortened and the addresses are numeric; to make it readable, the symbol file produced when the build was made is consulted. Turning addresses into function and line names is called symbolication. The symbol file has to be from the same version as the build and has to be stored for every release; the crash of a release whose symbols were not stored cannot be read.

The two fragments below show the platform side of recording the moment of the crash locally. The fragments are not measured.

// Platform code: iOS diagnostics subscriber
final class DiagnosticsReceiver: NSObject, MXMetricManagerSubscriber {
    func didReceive(_ payloads: [MXDiagnosticPayload]) {
        for payload in payloads { crashStore.save(payload.jsonRepresentation()) }
    }
}

// Platform code: uncaught error handler
Thread.setDefaultUncaughtExceptionHandler { _, error ->
    crashStore.save(error.stackTraceToString())
}

In both fragments the first job is the same: writing the report to the local store. Sending it over the network is not reliable in a crashing process; the process can die before the sending finishes. Sending is done at the next launch of the application, in a healthy process.

Analytics Events and User Consent

An analytics event records what the user does in the application: a screen was opened, a button was pressed. A crash report is the trace of a fault; an analytics event is behavior and can be tied to the user. For this reason the two are not subject to the same consent. If the user has turned off the tracking of behavior, the analytics event is never produced and is not stored to be sent later. Storing is also a form of collecting.

One distinction matters here: consent is a capability, and it has no fallback. When the network is absent the event can wait, because the network comes back. When consent is absent the event does not wait, because consent does not come back by itself. The first is deferred, the second is disabled. The crash-free session count is tied to this as well: to say a session did not crash, it is not enough that no report came from it; the report must have been able to arrive. No ratio is written here; the number is a count of sessions.

Measuring the Condition of Monitoring

The measurement takes three features. Settings needs no monitoring at all. CrashReport needs the network, and without the network it works as deferred. UsageEvent needs both the network and the user’s consent; the fallback of the network is deferred, and consent has no fallback. The core is as in the earlier lessons and is given again in the same form in every lesson.

// operability.mjs — classifies how fully a feature works under a condition
export const LEVELS = ["full", "degraded", "deferred", "disabled"];
const FALLBACKS = ["degraded", "deferred"];

export function feature(name, needs, fallback = {}) {
  return { name, needs, fallback };
}

export function condition(name, has) {
  return { name, has };
}

function judge(item, cond) {
  const missing = item.needs.filter((need) => !cond.has.includes(need)).sort();
  let level = "full";
  for (const need of missing) {
    const given = FALLBACKS.includes(item.fallback[need]) ? item.fallback[need] : "disabled";
    if (LEVELS.indexOf(given) > LEVELS.indexOf(level)) level = given;
  }
  return { level, feature: item.name, condition: cond.name, missing };
}

export function assess(features, conditions) {
  const records = [];
  for (const cond of conditions) {
    for (const item of features) records.push(judge(item, cond));
  }
  return records;
}

export function printLevels(records) {
  for (const r of records) {
    const missing = r.missing.length === 0 ? "-" : r.missing.join(",");
    console.log(`${r.level}: ${r.feature} @ ${r.condition} missing=${missing}`);
  }
}

export function countLevels(records) {
  for (const name of LEVELS) {
    console.log(`${name}: ${records.filter((r) => r.level === name).length}`);
  }
  console.log(`affected: ${records.filter((r) => r.level !== "full").length}`);
}

After the core comes the lesson’s helper file. conditionsFor(session) turns a session’s network and consent into capabilities.

// telemetry.mjs — turns the session's network and user consent into capabilities
import { feature, condition } from "./operability.mjs";

export function conditionsFor(session) {
  const has = [];
  if (session.network) has.push("network");
  if (session.consent) has.push("consent");
  return condition(session.name, has);
}

export const telemetryFeatures = [
  feature("Settings", []),
  feature("CrashReport", ["network"], { network: "deferred" }),
  feature("UsageEvent", ["network", "consent"], { network: "deferred" }),
];

RO11. The model ties the crash report to no capability other than the network. In reality the sending of the report may also depend on the user’s diagnostics preference; this lesson measures the report only for the lack of the network, and the preference is a capability that could be added separately.

The first measurement runs three sessions on three features: a session with the network and consent, a session without the network, and a session without consent.

// main.mjs
import { assess, printLevels } from "./operability.mjs";
import { conditionsFor, telemetryFeatures } from "./telemetry.mjs";

const sessions = [
  { name: "online", network: true, consent: true },
  { name: "offline", network: false, consent: true },
  { name: "no-consent", network: true, consent: false },
];
printLevels(assess(telemetryFeatures, sessions.map(conditionsFor)));

full: Settings @ online missing=-
full: CrashReport @ online missing=-
full: UsageEvent @ online missing=-
full: Settings @ offline missing=-
deferred: CrashReport @ offline missing=network
deferred: UsageEvent @ offline missing=network
full: Settings @ no-consent missing=-
full: CrashReport @ no-consent missing=-
disabled: UsageEvent @ no-consent missing=consent

With the network and consent present, all three features are full. Without the network the report and the event become deferred, and the settings stay full. Without consent the report goes out, the event is disabled, and the missing capability is consent. The same UsageEvent fell to two different levels with two different missing capabilities: deferred if the missing one is the network, disabled if it is consent.

Until the Connection Returns

An application that crashes in an offline session writes the report to a local queue. The queue is the lesson-local eventQueue array. When the connection returns, the queue is drained and the report goes out. Sending itself is a job; operability says only whether the report can be sent at that moment.

The second measurement runs one crash at two moments: while there is no connection, and after the connection has returned. In the same measurement, in a list of six sessions, the number of sessions that look crash-free is compared with the number that really did not crash.

// main.mjs
import { assess, printLevels } from "./operability.mjs";
import { conditionsFor, telemetryFeatures } from "./telemetry.mjs";

const crashReport = telemetryFeatures.filter((item) => item.name === "CrashReport");
const eventQueue = [];

printLevels(assess(crashReport, [conditionsFor({ name: "offline", network: false, consent: true })]));
eventQueue.push({ kind: "crash" });
console.log(`queue: ${eventQueue.length}`);
printLevels(assess(crashReport, [conditionsFor({ name: "reconnected", network: true, consent: true })]));
console.log(`queue drained: ${eventQueue.splice(0).length}`);

const sessions = [
  { crashed: false, delivered: false },
  { crashed: false, delivered: false },
  { crashed: true, delivered: true },
  { crashed: false, delivered: false },
  { crashed: true, delivered: false },
  { crashed: false, delivered: false },
];
const seen = sessions.filter((item) => !item.delivered).length;
const actual = sessions.filter((item) => !item.crashed).length;
console.log(`sessions: ${sessions.length} crash-free shown: ${seen} actual: ${actual}`);

deferred: CrashReport @ offline missing=network
queue: 1
full: CrashReport @ reconnected missing=-
queue drained: 1
sessions: 6 crash-free shown: 5 actual: 4

The report is deferred while offline and full after the connection returns. The single report in the queue went out. Two of the six sessions had crashed, but one had not yet been delivered; so the dashboard shows five sessions as crash-free, while in reality four did not crash. A waiting report makes the crash-free session count look better than it is.

RO12. The model assumes the queue holds a single event and loses nothing. In reality the queue can fill up, can be deleted when the application is reinstalled, or the sending can be cut halfway. The rule does not see these; it looks only at whether the network is in the condition.

The Bounding Measurement

The focus record of this lesson is CrashReport @ offline. The bounding measurement shows two things together. First, the focus record is deferred, and three of nine records are affected; in the session without consent UsageEvent is disabled because it has no fallback. Second, the number of reports waiting in the queue does not enter the record: whether the queue holds no reports or three, the record is the same.

// main.mjs
import { assess, printLevels, countLevels } from "./operability.mjs";
import { conditionsFor, telemetryFeatures } from "./telemetry.mjs";

const sessions = [
  { name: "online", network: true, consent: true },
  { name: "offline", network: false, consent: true },
  { name: "no-consent", network: true, consent: false },
];
const records = assess(telemetryFeatures, sessions.map(conditionsFor));
printLevels(records.filter((r) => `${r.feature} @ ${r.condition}` === "CrashReport @ offline"));
printLevels(records.filter((r) => `${r.feature} @ ${r.condition}` === "UsageEvent @ no-consent"));
countLevels(records);

const crashReport = telemetryFeatures.filter((item) => item.name === "CrashReport");
const queued = [0, 3].map((count) => ({
  count,
  level: assess(crashReport, [conditionsFor(sessions[1])])[0].level,
}));
const same = queued[0].level === queued[1].level ? "yes" : "no";
console.log(`queue ${queued[0].count} and ${queued[1].count}: record same: ${same}`);

deferred: CrashReport @ offline missing=network
disabled: UsageEvent @ no-consent missing=consent
full: 6
degraded: 0
deferred: 2
disabled: 1
affected: 3
queue 0 and 3: record same: yes

The first two lines give the two ends of the lesson: deferred if the network is missing, disabled if consent is missing. In the count, of nine records six are full, two are deferred, and one is disabled; affected is three. The last line shows the limit of the model: the rule does not see the queue. Many reports waiting in the queue are a workload and are counted separately on the monitoring dashboard; they do not change the operability level.

Course Wrap-Up

Over fourteen lessons the course measured fourteen features, each in its own condition, with the same four levels. The table below gives each lesson’s focus record, its operability, and the measurement that bounds that lesson. The operability column is the one written in each lesson’s ## Summary section.

Lesson	Focus record	Operability	Bounding measurement
Modular Code Sharing	Scan @ android	disabled	The scanner layer was not written, no fallback
Design System Adaptation	DatePicker @ android	full	Two picker forms, the same picker capability
Configuration and Environment Management	Search @ prod-default	full	prod-default-off disabled in the flag whose default is off
Accessibility	CardSwipe @ screen-reader	degraded	CustomGesture without a fallback disabled
Offline-First Design	FeedLocal @ offline	degraded	Network-first FeedRemote disabled
Synchronization Strategies	EditNote @ offline	deferred	EditNote full while connected
Background Processing Limits	MediaBackup @ background-restricted	deferred	LiveLocation without a fallback disabled
Connectivity State Management	Gallery @ weak-network	degraded	Chat deferred in the same condition
Mobile Continuous Delivery	NewCheckout @ production	disabled	In the model, signing does not enter the record (by definition)
Version Management	OrderHistory @ api-v2	degraded	The client version does not enter the record
Staged Rollout and Rollback	NewFeed @ device-a	full	The record does not change when the rollout widens
Forced Update Strategy	Payments @ client-v4	disabled	The severity of the prompt does not enter the record
App Size Management	Filters @ base-install	deferred	The first-download limit does not enter the record
Crash and Usage Monitoring	CrashReport @ offline	deferred	The number of reports in the queue does not enter the record

Thirteen rows of the table are taken from the earlier lessons; the level of the fourteenth row comes from this lesson’s own measurement. The last measurement counts the table and puts beside it the opening count that the core gave in the first lesson.

// main.mjs
import { assess } from "./operability.mjs";
import { conditionsFor, telemetryFeatures } from "./telemetry.mjs";

const earlier = ["disabled", "full", "full", "degraded", "degraded", "deferred", "deferred",
  "degraded", "disabled", "degraded", "full", "disabled", "deferred"];
const crashReport = telemetryFeatures.filter((item) => item.name === "CrashReport");
const focus = assess(crashReport, [conditionsFor({ name: "offline", network: false, consent: true })])[0];
const lessons = [...earlier, focus.level];

const order = ["full", "degraded", "deferred", "disabled"];
const spread = order.map((name) => `${name}=${lessons.filter((level) => level === name).length}`);
const opening = { full: 5, degraded: 1, deferred: 2, disabled: 1 };
const openingTotal = Object.values(opening).reduce((sum, count) => sum + count, 0);

console.log(`lessons: ${lessons.length}`);
console.log(`focus: ${focus.feature} @ ${focus.condition} ${focus.level}`);
console.log(`spread: ${spread.join(" ")}`);
console.log(`core opening: records=${openingTotal} affected=${openingTotal - opening.full}`);

lessons: 14
focus: CrashReport @ offline deferred
spread: full=3 degraded=4 deferred=4 disabled=3
core opening: records=9 affected=4

The distribution of the fourteen lessons is three full, four degraded, four deferred, and three disabled. The opening count of the core is a different thing: of nine records, five were full, one was degraded, two were deferred, and one was disabled, and affected was four. The core counts the nine records of a single application; the table counts fourteen lessons. The two counts are not the same thing and cannot be derived from each other.

The distribution says one more thing: a feature being disabled is not a common outcome, and working fully is not the rule. Each row of the table is a separate measurement showing what one condition does to one feature.

Summary

• This lesson’s operability: deferred. CrashReport @ offline came out deferred because network is missing and a deferred fallback is defined; in the same condition UsageEvent came out deferred and Settings stayed full.
• A crash report is stored in the local store, not in the crashing process, and goes out from a healthy process when the connection returns; the one report in the queue was drained when the connection returned.
• User consent is a capability without a fallback: UsageEvent @ no-consent came out disabled, while the same event was only deferred when the network was absent.
• An undelivered report makes the crash-free session count look better than it is: in six sessions five looked crash-free, while four had really not crashed.
• The distribution of the fourteen lessons is three full, four degraded, four deferred, and three disabled.

Next Step

This lesson is the last lesson of the course and of the catalog; there is no lesson after it. What remains in your hands is the question of the course: write a feature, a condition, and a missing capability, define its fallback, and measure in your own application which level the record falls to.