Monitoring Categories

The previous topic built ten diagnoses and ran each one through the same three steps: the symptom was written as a number, a second cause producing the same symptom was established, and a measurement separating the two was shown. The topic’s closing left the condition all ten lessons share — every one of them works only if a measurement exists. Where that measurement comes from was never asked in any lesson: which metric is collected, which is not, and which diagnosis the uncollected one takes off the table were all assumed.

This lesson fills that gap and opens with a single question: when does the metric set collected from a system count as complete? Completeness here is not a list length but a coverage measure — which category being empty makes which diagnosis impossible. The measure is therefore defined over the previous topic’s ten lessons.

Monitoring and Observability

Monitoring is continuously collecting answers to questions determined in advance; observability is being able to answer a question that was never asked in advance from data already collected. This lesson builds monitoring, because the precondition for being able to answer an unasked question is knowing what gap the asked questions leave.

A metric corresponds to a question, and its category is that question’s type. The five categories ask:

• Availability: was the request answered? The term should not be confused with interface accessibility and is not abbreviated; the distinction was made in the Introduction to System Design course. Its metrics are request success ratio and minutes of outage; the second directly spends that course’s monthly 43.2-minute outage budget and the 28.2-minute failure share within it.
• Health: which replica can take work? The content of the health endpoint — shallow check, readiness check, deep check — was designed in the Resilience and Reliability course and is not redesigned here; the yes/no it produces is placed here as a monitoring metric, unchanged.
• Performance: how long does the work take? Response-time percentiles and sub-call durations.
• Utilization: how much of the resource was spent? Utilization, bytes carried, calls per request, objects in memory, cache hit rate, records scanned, work share per customer.
• Security: who sent the request, and did they have the right to send it? Rejected unauthorized requests and rate-limit triggers.

The reason the performance category cannot be read on its own was already measured in that same course: a latency number cannot be read without saying what load it was taken under. A percentile measured at the peak edge rate of 513.89 requests/s and the same percentile measured at midnight are not the same number. This is why every performance metric wants a utilization metric alongside it.

Completeness Is a Coverage Measure

The computation below maps ten diagnoses to sixteen metrics, then empties each category one at a time and counts how many diagnoses become impossible. The mapping itself is this lesson’s assumption (IZ1): the previous topic showed a distinguishing measurement for every diagnosis, and the table below picks the metrics that correspond to those measurements and places them into categories. A different mapping gives different numbers. The resulting figures belong to the computation’s class.

// monitoring/coverage.mjs — coverage measure for the five categories: which metric each of the
// ten diagnoses in the Anti-Patterns topic requires, and which diagnoses become impossible if a
// category is never collected.
const METRIC = {
  "request success ratio": "availability",
  "minutes of outage": "availability",
  "replica pool status": "health",
  "readiness status": "health",
  "response-time percentile": "performance",
  "store query duration": "performance",
  "calls per request": "utilization",
  "body bytes carried": "utilization",
  "processor utilization": "utilization",
  "pool utilization": "utilization",
  "objects in memory": "utilization",
  "cache hit rate": "utilization",
  "records scanned": "utilization",
  "work share per customer": "utilization",
  "rejected unauthorized requests": "security",
  "rate-limit trigger": "security",
};

const DIAGNOSIS = {
  "busy database": ["store query duration", "processor utilization", "response-time percentile"],
  "busy front end": ["body bytes carried", "response-time percentile"],
  "chatty I/O": ["calls per request", "response-time percentile"],
  "over-fetching": ["body bytes carried", "cache hit rate"],
  "improper instantiation": ["objects in memory", "processor utilization"],
  "monolithic persistence": ["records scanned", "store query duration", "processor utilization"],
  "no caching": ["cache hit rate", "processor utilization", "response-time percentile"],
  "noisy neighbor": ["work share per customer", "response-time percentile", "rate-limit trigger"],
  "synchronous I/O": ["pool utilization", "readiness status", "response-time percentile"],
  "retry storm": ["request success ratio", "calls per request",
    "replica pool status", "rate-limit trigger"],
};

const CATEGORY = [...new Set(Object.values(METRIC))];
const categoryOf = (o) => METRIC[o];
const diagnoses = Object.keys(DIAGNOSIS);

console.log(`${CATEGORY.length} categories, ${Object.keys(METRIC).length} metrics, ${diagnoses.length} diagnoses`);
console.log(`\n${"category".padEnd(18)}${"metrics".padStart(7)}${"diagnoses fed".padStart(16)}` +
  `${"category empty: diagnoses dropped".padStart(35)}`);
const dropped = {};
for (const k of CATEGORY) {
  const metrics = Object.keys(METRIC).filter((o) => categoryOf(o) === k);
  dropped[k] = diagnoses.filter((t) => DIAGNOSIS[t].some((o) => categoryOf(o) === k));
  console.log(`${k.padEnd(18)}${String(metrics.length).padStart(7)}` +
    `${String(dropped[k].length).padStart(16)}` +
    `${`${dropped[k].length}/${diagnoses.length}`.padStart(35)}`);
}
console.log(`no category can be left empty: the lightest-loaded category is ` +
  `${CATEGORY.reduce((a, b) => (dropped[a].length <= dropped[b].length ? a : b))}, and even it ` +
  `drops ${Math.min(...CATEGORY.map((k) => dropped[k].length))} diagnoses`);

const counts = Object.keys(METRIC)
  .map((o) => [o, diagnoses.filter((t) => DIAGNOSIS[t].includes(o)).length])
  .sort((a, b) => b[1] - a[1]);
const unique = counts.filter(([, n]) => n === 1).map(([o]) => o);
const unused = counts.filter(([, n]) => n === 0).map(([o]) => o);
console.log(`\nmost shared metric = ${counts[0][0]} (${counts[0][1]}/${diagnoses.length} diagnoses), ` +
  `second ${counts[1][0]} (${counts[1][1]}/${diagnoses.length})`);
console.log(`metrics appearing in exactly one diagnosis (${unique.length}): ${unique.join(", ")}`);
console.log(`metrics appearing in no diagnosis (${unused.length}): ${unused.join(", ")}`);

console.log(`\n${"diagnosis".padEnd(26)}${"metrics".padStart(8)}${"categories".padStart(12)}` +
  `${"distinguishing (unique to it)".padStart(31)}`);
for (const t of diagnoses) {
  const categories = new Set(DIAGNOSIS[t].map(categoryOf));
  const distinguishing = DIAGNOSIS[t].filter((o) => unique.includes(o));
  console.log(`${t.padEnd(26)}${String(DIAGNOSIS[t].length).padStart(8)}${String(categories.size).padStart(12)}` +
    `${(distinguishing[0] ?? "-").padStart(31)}`);
}

const pairs = [];
for (let i = 0; i < diagnoses.length; i += 1)
  for (let j = i + 1; j < diagnoses.length; j += 1) {
    const [a, b] = [diagnoses[i], diagnoses[j]];
    const shared = DIAGNOSIS[a].filter((o) => DIAGNOSIS[b].includes(o));
    pairs.push({ a, b, shared });
  }
const maxShared = Math.max(...pairs.map((c) => c.shared.length));
console.log(`\ndiagnosis pairs sharing the most metrics (${maxShared} shared):`);
for (const { a, b, shared } of pairs.filter((c) => c.shared.length === maxShared))
  console.log(`  ${a} / ${b} -> shared ${shared.join(" + ")}; distinguished by ` +
    `${DIAGNOSIS[a].filter((o) => !shared.includes(o)).join(" + ")} / ` +
    `${DIAGNOSIS[b].filter((o) => !shared.includes(o)).join(" + ")}`);

5 categories, 16 metrics, 10 diagnoses

category          metrics   diagnoses fed  category empty: diagnoses dropped
availability            2               1                               1/10
health                  2               2                               2/10
performance             2               7                               7/10
utilization             8              10                              10/10
security                2               2                               2/10
no category can be left empty: the lightest-loaded category is availability, and even it drops 1 diagnoses

most shared metric = response-time percentile (6/10 diagnoses), second processor utilization (4/10)
metrics appearing in exactly one diagnosis (7): request success ratio, replica pool status, readiness status, pool utilization, objects in memory, records scanned, work share per customer
metrics appearing in no diagnosis (2): minutes of outage, rejected unauthorized requests

diagnosis                  metrics  categories  distinguishing (unique to it)
busy database                    3           2                              -
busy front end                   2           2                              -
chatty I/O                       2           2                              -
over-fetching                    2           1                              -
improper instantiation           2           1              objects in memory
monolithic persistence           3           2                records scanned
no caching                       3           2                              -
noisy neighbor                   3           3        work share per customer
synchronous I/O                  3           3               pool utilization
retry storm                      4           4          request success ratio

diagnosis pairs sharing the most metrics (2 shared):
  busy database / monolithic persistence -> shared store query duration + processor utilization; distinguished by response-time percentile / records scanned
  busy database / no caching -> shared processor utilization + response-time percentile; distinguished by store query duration / cache hit rate

If a Category Is Left Empty

The first table gives the coverage measure. The utilization category feeds ten diagnoses out of ten: in a system that collects only availability and response time, none of the ten diagnoses can be made. This is not surprising, but the consequence is severe — a symptom’s cause requires knowing how much of the resource was spent, yet utilization is the category most often neglected, because anyone can see the service is up without ever looking at it.

Performance feeds seven diagnoses. The remaining three categories each feed one or two diagnoses, and that small number does not make them optional: if availability is never collected, retry storm cannot be separated from external demand that genuinely grew; if health is never collected, synchronous I/O cannot be distinguished from a storm; if security is never collected, noisy neighbor cannot be distinguished from a storm. The dropped diagnoses are not compensated for by another category; this is why the measure is a coverage, not a total.

Two metrics appear in no diagnosis: minutes of outage and rejected unauthorized requests. This is not a gap; it is a consequence of how the category is defined. A category is defined by the question it answers, not by the diagnoses it enables. Minutes of outage does not distinguish a single antipattern, but it is the one number that spends the monthly 43.2-minute budget, and no alert threshold can be defended without that budget. A metric set pruned to the diagnosis list loses the questions that fall outside diagnosis.

The mapping’s sensitivity shows up here too: in IZ1, processor utilization feeds four diagnoses and is busy database’s only utilization metric, so losing that one metric drops this diagnosis’s entire utilization leg. A different mapping changes the column’s numbers, but none leaves any of the five categories carrying no load at all.

Diagnosis Is a Metric Composition

The second block gives the lesson’s second result. Response-time percentile appears in six of the ten diagnoses, processor utilization in four. The most collected metric is the least distinguishing metric. A response-time chart climbing is the shared symptom of six separate causes and points to none of them on its own; this is the numeric counterpart of the previous topic’s line that “slow is not a diagnosis.”

Seven metrics appear in exactly one diagnosis, and these are signature metrics: objects in memory, work share per customer, records scanned, pool utilization, readiness status, replica pool status, and request success ratio. Five diagnoses have a signature metric, five do not; the latter are separated only by composition. The last two lines show how.

Busy database and no caching share two metrics — processor utilization and response-time percentile — and a third metric makes the distinction: did store query duration grow, or did cache hit rate fall? In the same way, busy database and monolithic persistence share store query duration and processor utilization; the distinction is made by records scanned, because the symptom of piling onto a single store is a query scanning many times over its own data class.

This also shapes the form of the instrumentation decision: a metric is collected not on its own but for the pair it distinguishes. A metric’s value comes not from “how many diagnoses it appears in” but from “which two causes it separates.”

Summary

• Monitoring collects answers to questions determined in advance; observability is being able to answer a question that was never asked in advance from data already collected.
• The five categories are defined by the question each one asks: availability, health, performance, utilization, and security; the content of the health endpoint was designed in the previous course and is placed here only as a category.
• Completeness is a coverage measure: the utilization category feeds ten of the ten diagnoses, performance seven; availability feeds one, health and security two each, and none can be left empty.
• Minutes of outage and rejected unauthorized requests appear in no diagnosis but are still collected; a category is defined by the question it answers, not by the diagnoses it enables.
• The most shared metric is the least distinguishing metric: response-time percentile appears in six diagnoses, so on its own it points to none of them.
• Seven of the sixteen metrics appear in exactly one diagnosis and are that diagnosis’s signature; five diagnoses have no signature metric and are separated only by composition: busy database and no caching share two metrics, and the distinction is made by store query duration and cache hit rate.

Next Step

This lesson put sixteen metrics into a list and tested the list’s completeness with coverage, but never accounted for the list having a cost. Every metric is produced, carried, stored, and read; each one is added to a code path, and that code path runs on every one of the 513.89 requests at the peak edge. As the metric count grows, so does the number of records collected, and at some point the load the monitoring system produces becomes comparable to the load of the system it monitors. The next lesson counts that cost: the records a metric produces per request, the bytes it stores per day, and the code path it adds are weighed on one side against the diagnosis it enables on the other; it measures what information sampling and aggregation lose while making things cheaper.