Incident Classification

The on-call rotation answered the “who” and “when” questions, but it assumed that every call was worth waking a person for. The wake-up decision is made before the call, and that decision is made with a label.

Severity is the label that determines how fast and how hard an incident is met; escalation is calling the next step when the response the label demands does not arrive. What this lesson measures is not the label itself, but the cost of a wrong label in both directions: over-escalation breaks up the night, under-escalation delays the response.

The Incident Set, the Actual Severity, and Two Rules

The example is once more the course’s fictional regional measurement network: the water-meter reading collector, verifier, billing, and work-order services, across three environments. Both the network and the incident set are fictional.

IM7: Sixty incidents are produced; each incident’s actual severity follows only from the subscriber count affected and any data loss, both known only after it closes — d1 is heaviest, d4 is lightest. IM8: At call time, only the visible signature exists: environment, service, error rate, delay factor, nightly batch job window. IM9: d1 and d2 page and wake at night too; d3 waits till morning, d4 drops onto the work list. IM10: The cost of an over-escalation that falls at night is a 4-hour recovery window (the same unit as the on-call lesson); for an under-escalation, if the incident is at night it waits for 06:00, and on top of that a d4 label adds 240 minutes, a d3 label adds 60. IM11: The cost of the delay is impact-minutes: delay minutes × the actual severity’s weight (d1 8, d2 3, d3 1, d4 0). IM12: Escalation is sequential; a step responds with probability 0.70 by day and 0.40 at night, and if no response arrives, the next step is called once the wait time ends.

// incident/rule.mjs — the fictional regional measurement network's incident set and
// classification model (model): there is no real incident record, call order, or person. The generator is our own, the seed is visible.

export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"];
export const SEVERITY = ["d1", "d2", "d3", "d4"];        // d1 is the heaviest
export const WEIGHT = { d1: 8, d2: 3, d3: 1, d4: 0 };  // impact-minute multiplier
export const WAKES = ["d1", "d2"];                // these wake at night too
export const WINDOW = 4;                             // recovery window for an unnecessary wake-up
export const MORNING = 6 * 60;                        // an incident left till morning is handled at 06:00
export const RESPONSE = { day: 0.70, night: 0.40 };    // the probability a step responds

export function generator(seed) {
  let x = seed >>> 0 || 1;
  return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0;
    return x / 4294967296; };
}

// Hidden fields (known only after the incident closes): subscribers affected, data loss -> actual severity.
// Visible signature (at call time): environment, service, error rate, delay factor, batch job window.
export function incidents(seed, count = 60) {
  const r = generator(seed), list = [];
  for (let i = 0; i < count; i++) {
    const service = SERVICE[Math.floor(r() * SERVICE.length)];
    const u = r();
    const environment = u < 0.72 ? "production" : u < 0.88 ? "staging" : "development";
    const minute = Math.floor(r() * 1440);
    const magnitude = r();                            // the size of the impact, a long tail
    const subscribers = environment === "production" ? Math.floor(magnitude ** 3 * 60000) : 0;
    const dataLoss = environment === "production" && service === "billing" && r() < 0.45;
    const silent = dataLoss || (service === "verifier" && environment === "production" && r() < 0.35);
    const errorRate = +(silent ? 0.01 + r() * 0.05
      : environment !== "production" ? 0.35 + r() * 0.6 : 0.04 + magnitude * 0.8).toFixed(2);
    const delayFactor = +(1 + (silent ? r() : environment !== "production" ? 4 + r() * 5
      : magnitude * 8)).toFixed(1);
    const actual = dataLoss || subscribers >= 20000 ? "d1" : subscribers >= 5000 ? "d2"
      : subscribers >= 500 ? "d3" : "d4";
    list.push({ i, service, environment, minute, night: minute < MORNING, batch: minute >= 120 && minute < 240,
      subscribers, dataLoss, errorRate, delayFactor, actual });
  }
  return list;
}

// Two classification rules. Both look only at the visible signature.
export const RULE = {
  // A: threshold-based. Filters out non-production, labels the rest by error rate and delay factor
  A: (o) => o.environment !== "production" ? "d4"
    : o.errorRate >= 0.50 && o.delayFactor >= 4 ? "d1"
    : o.errorRate >= 0.25 || o.delayFactor >= 3 ? "d2"
    : o.batch ? "d3" : "d4",
  // B: cautious. Thresholds are low; because billing degrades silently, its floor is pulled to d2
  B: (o) => o.environment !== "production" ? "d4"
    : o.errorRate >= 0.30 ? "d1"
    : o.errorRate >= 0.10 || o.delayFactor >= 2 ? "d2"
    : o.service === "billing" ? "d2" : "d3",
};

const order = (d) => SEVERITY.indexOf(d);
export const direction = (label, actual) => order(label) < order(actual) ? "over"
  : order(label) > order(actual) ? "under" : "correct";

// The cost of a label. Over-escalation makes an unnecessary night wake-up; under-escalation
// delays the response, and the delay is multiplied by the actual severity's weight into impact-minutes.
export function cost(o, label) {
  const calls = WAKES.includes(label);
  const d = direction(label, o.actual);
  const wakeup = calls && o.night && d === "over" ? 1 : 0;
  const delay = !calls && WAKES.includes(o.actual)
    ? (o.night ? MORNING - o.minute : 0) + (label === "d4" ? 240 : 60) : 0;
  return { wakeup, window: wakeup * WINDOW, delay,
    impactMin: delay * WEIGHT[o.actual], d, calls };
}

// Escalation chain: number of steps and minutes of wait per step. The dice are rolled in advance,
// so the same incident sees the same draws when the chain is lengthened.
export function dice(seed, count, most = 6) {
  const r = generator(seed);
  return Array.from({ length: count }, () => Array.from({ length: most }, () => r()));
}

export function escalate(o, d, steps, wait) {
  const p = o.night ? RESPONSE.night : RESPONSE.day;
  for (let s = 0; s < steps; s++)
    if (d[s] < p) return { reached: true, step: s + 1, minute: s * wait, woken: s + 1 };
  return { reached: false, step: steps, minute: steps * wait, woken: steps };
}

// incident/measure.mjs — the classification's two-way error, its cost, and the escalation chain sweep.
import { incidents, RULE, SEVERITY, cost, direction, dice, escalate, WAKES, WINDOW }
  from "./rule.mjs";

const SEED = 20260921;
const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const f1 = (x) => x.toFixed(1);
const count = (a, f) => a.filter(f).length;

const O = incidents(SEED);
const Z = dice(SEED + 1, O.length);
console.log(`seed ${SEED}: ${O.length} incidents, ${count(O, (o) => o.night)} of them at night, ` +
  `${count(O, (o) => o.environment !== "production")} of them non-production`);
console.log("actual severity distribution: " +
  SEVERITY.map((d) => `${d} ${count(O, (o) => o.actual === d)}`).join(", "));

const A = [-7, 9, 18, 18, 15, 16];
console.log("\n1. two classification rules on the same incident set");
write(A, "rule", "correct", "over-escalation", "under-escalation", "pages", "actually pages");
for (const [name, k] of Object.entries(RULE)) {
  const e = O.map((o) => direction(k(o), o.actual));
  write(A, name, count(e, (x) => x === "correct"), count(e, (x) => x === "over"),
    count(e, (x) => x === "under"), count(O, (o) => WAKES.includes(k(o))),
    count(O, (o) => WAKES.includes(o.actual)));
}

const B = [-7, 17, 13, 17, 14, 14];
console.log("\n2. the two-way cost of misclassification");
write(B, "rule", "unnec. wake-ups", "hours lost", "del. response", "delay min", "impact-min");
const T2 = {};
for (const [name, k] of Object.entries(RULE)) {
  const b = O.map((o) => cost(o, k(o)));
  const late = b.filter((x) => x.delay > 0);
  T2[name] = { hours: b.reduce((a, x) => a + x.window, 0),
    impact: b.reduce((a, x) => a + x.impactMin, 0) };
  write(B, name, b.reduce((a, x) => a + x.wakeup, 0), T2[name].hours,
    late.length, late.length ? f1(late.reduce((a, x) => a + x.delay, 0) / late.length) : "-",
    T2[name].impact);
}
const dHours = T2.B.hours - T2.A.hours, dImpact = T2.A.impact - T2.B.impact;
console.log(`A -> B trade-off: hours lost +${dHours}, impact-minutes -${dImpact} ` +
  `(${f1(dImpact / dHours)} impact-minutes per hour spent awake)`);

const C = [-16, 8, 8, 8, 8];
console.log("\n3. the incidents rule A confuses (label -> actual)");
write(C, "label \\ actual", ...SEVERITY);
for (const e of SEVERITY)
  write(C, e, ...SEVERITY.map((g) => count(O, (o) => RULE.A(o) === e && o.actual === g)));

const T = [-9, -9, 13, 16, 21, 14];
console.log("\n4. escalation chain sweep (incidents that rule B pages)");
write(T, "steps", "wait", "avg step", "avg reach min", "unreached incidents", "people woken");
const paging = O.filter((o) => WAKES.includes(RULE.B(o)));
for (const steps of [2, 3, 4])
  for (const wait of [5, 15]) {
    const R = paging.map((o) => escalate(o, Z[o.i], steps, wait));
    write(T, steps, `${wait} min`, f1(R.reduce((a, x) => a + x.step, 0) / R.length),
      f1(R.reduce((a, x) => a + x.minute, 0) / R.length), count(R, (x) => !x.reached),
      R.reduce((a, x) => a + x.woken, 0));
  }
console.log(`${paging.length} pages, ${count(paging, (o) => o.night)} of them at night; ` +
  `the unnecessary wake-up window is ${WINDOW} hours`);

const Q = [-55, 22, 8];
console.log("\n5. the question set the classification answers");
write(Q, "question", "data", "answer");
const QUESTION = [
  ["should this incident wake someone", "visible signature", true],
  ["at which severity", "visible signature", true],
  ["who is called if there is no response", "escalation chain", true],
  ["when is the next step", "escalation chain", true],
  ["how many subscribers did the incident actually affect", "-", false],
  ["was the label correct", "-", false],
  ["what happened if it reached no one", "-", false]];
for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no");
console.log(`${count(QUESTION, (x) => x[2])} of ${QUESTION.length} questions are answered at call ` +
  `time, ${count(QUESTION, (x) => !x[2])} cannot be answered until the incident closes`);

seed 20260921: 60 incidents, 15 of them at night, 17 of them non-production
actual severity distribution: d1 12, d2 14, d3 13, d4 21

1. two classification rules on the same incident set
rule     correct   over-escalation  under-escalation          pages  actually pages
A             37                16                 7             31              26
B             27                28                 5             40              26

2. the two-way cost of misclassification
rule     unnec. wake-ups   hours lost    del. response     delay min    impact-min
A                      3           12                5         306.0         11040
B                      7           28                1          60.0           180
A -> B trade-off: hours lost +16, impact-minutes -10860 (678.8 impact-minutes per hour spent awake)

3. the incidents rule A confuses (label -> actual)
label \ actual        d1      d2      d3      d4
d1                     8       6       0       0
d2                     0       7      10       0
d3                     0       0       1       0
d4                     4       1       2      21

4. escalation chain sweep (incidents that rule B pages)
steps    wait          avg step   avg reach min  unreached incidents  people woken
2        5 min              1.4             2.8                    5            57
2        15 min             1.4             8.3                    5            57
3        5 min              1.6             3.0                    2            62
3        15 min             1.6             9.0                    2            62
4        5 min              1.6             3.3                    2            64
4        15 min             1.6             9.8                    2            64
40 pages, 12 of them at night; the unnecessary wake-up window is 4 hours

5. the question set the classification answers
question                                                                 data  answer
should this incident wake someone                           visible signature     yes
at which severity                                           visible signature     yes
who is called if there is no response                        escalation chain     yes
when is the next step                                        escalation chain     yes
how many subscribers did the incident actually affect                       -      no
was the label correct                                                       -      no
what happened if it reached no one                                          -      no
4 of 7 questions are answered at call time, 3 cannot be answered until the incident closes

The numbers are of the measurement class; their inputs are the IM7–IM12 assumptions. Of the sixty incidents, 15 fall at night, 17 are in a non-production environment, and the number that genuinely earns a page (d1 or d2) is 26. The threshold rule A produces 31 pages, the cautious rule B produces 40; both sit above the truth.

Accuracy Does Not Measure the Cost

If the first table compares the two rules by correct-label count, A wins by a wide margin: it labels 37 of 60 incidents correctly, B labels 27. Over-escalation is 16 for A, 28 for B. Anyone looking at this measure picks A.

The second table reverses the decision. Of A’s 16 over-escalations, only 3 fall at night, meaning a cost of 12 recovery hours; of B’s 28, 7 land at night and take 28 hours. Against that, A’s 5 delayed responses wait 306 minutes on average and produce 11,040 impact-minutes; B’s single delayed response costs 180 impact-minutes in 60 minutes. The trade-off reads in a single line: moving from A to B buys 10,860 impact-minutes at the cost of 16 hours of sleep, that is, 678.8 impact-minutes per hour spent awake. The correct-label ratio is not a quality measure; the two directions’ costs are in different units, and they are comparable only when counted separately.

The Rule Cannot See Silent Degradation

The third table shows where A goes wrong. In the d4 row there are four true d1 incidents: the rule threw these four onto the work list. All four sit in the same place — data loss in the billing service, error rate between 0.01 and 0.05, delay factor between 1.2 and 1.8. A silent degradation is indistinguishable from a healthy service in the visible signature; 10,320 of the 11,040 impact-minutes come from these four incidents.

B catches these four, but not with a rule — with an exception: it pulls the billing service’s floor up to d2. The exception does not see the silence; it memorizes where the silence is. That is also its price — that service’s light incidents start producing pages too, and B’s unnecessary wake-ups rise from 3 to 7. The one incident B misses is a silent d2 in the verifier service: wherever no exception is written, the same gap still stands.

The ten incidents in the d2 row run the other way: incidents whose actual severity is d3 were escalated because they showed an error rate above 0.25 in production. The cost of these ten over-escalations is close to zero, because most fall during the day. The same count of wrong labels costs nothing or four hours, depending on which hour it falls in.

How Many Steps the Chain Takes, How Many Minutes per Step

The fourth table runs B’s 40 pages through the escalation chain; 12 of them are at night, where the response probability is low. In a two-step chain, 5 incidents reach no one. The third step drops that count to 2, the average step climbs from 1.4 to 1.6, and total people woken rises from 57 to 62: rescuing three of the five incidents costs five extra wake-ups. The fourth step gains nothing — unreached incidents stay at 2, and people woken climbs to 64.

Wait time is the second axis, and it never changes the unreached count: raising it from 5 minutes to 15 pushes the average reach time from 2.8 to 8.3 minutes, without gaining a single incident in return. The chain’s length determines how many incidents get met; the wait time only determines how late they are met.

Summary

• 26 of sixty incidents genuinely earn a page; the threshold rule produces 31 and the cautious rule produces 40, and both sit above the truth.
• The correct-label count is 37 for A and 27 for B; against that, A produces 11,040 impact-minutes and B produces 180 — the accuracy ratio does not measure the cost of either direction.
• The two directions are in different units: A has 3 unnecessary night wake-ups (12 hours) and 5 delayed responses (306 minutes on average); B has 7 wake-ups (28 hours) and 1 delayed response (60 minutes). The price of the switch is 678.8 impact-minutes per hour spent awake.
• All four d1 incidents A misses are silent data loss in the billing service: error rate 0.01–0.05, delay factor 1.2–1.8. The visible signature cannot tell silence apart from health.
• In escalation, two steps fail to reach anyone for 5 incidents; the third step drops that to 2 and costs five extra wake-ups, and the fourth step only wakes 2 more people.
• Raising the wait time from 5 to 15 minutes pushes the average reach from 2.8 to 8.3 minutes, without changing the unreached count at all.

Next Step

Four of seven questions are answered at call time; “how many subscribers did the incident affect”, “was the label correct”, and “what happened if it reached no one” can only be answered once the incident closes. Once the call reaches the right person at the right severity, one question is left: which step will close it. The next lesson measures that step as a document — how many steps, how many decision points, how many can be automated, how far apart the closing time is with and without a runbook, and how many steps go invalid after one period.