Dashboard Design

Once the convention was in place, questions became answerable. How the answer is read is a separate matter: nobody writes a query during an incident, they look at the panels in front of them. The dashboard itself grows over time too — a panel gets added after every incident, no panel gets deleted after any incident, and two years later the result is a page nobody knows the whole of.

This lesson audits a dashboard against a set of questions. Three numbers are measured: how many panels answer at least one question, how many panels answer none, how many questions remain without a panel. A fourth number follows from these: how many panels it takes to browse through to find an answer to a question.

The Question a Panel Answers

A panel answers a question only if it meets three conditions at once. The metric must be the same: a latency panel does not answer an error-rate question. The dimension must include the breakdown the question needs: a total error-rate panel cannot answer “which service is broken”, because the breakdown is not in the panel’s query. The resolution must be as fine as the question needs: a panel drawn with five-minute buckets does not answer a question asking about a one-minute window.

These three conditions sit as fields in the model below. The regional measurement network’s panel and question list is fiction; the question frequencies are put in to weight how the dashboard gets looked at.

// dashboard/dashboard.mjs — MODEL of a dashboard and the question set asked in operations. Panel: [name, metric,
// dims, resolution sec, series scanned by the query]. Question: [name, metric, required dim, coarsest resolution
// required, how many of forty incidents it is asked in]. MD13: the panel catalog, question set, and question
// frequencies are fiction; the dashboard has grown by additions over two years. MD14: a panel answers a question
// only if it carries the same metric, the dimension the question needs, and that resolution. MD15: scanning one
// panel by eye and moving on takes 8 seconds; writing a manual query for a panelless question takes 4 minutes.
export const PANELS = [
  ["cluster CPU usage", "cpu", ["node"], 60, 120], ["cluster memory usage", "memory", ["node"], 60, 120],
  ["node disk usage", "disk", ["node"], 300, 120], ["total requests (daily)", "request", [], 3600, 1],
  ["request count", "request", ["service"], 60, 5], ["error rate (total)", "error_rate", [], 60, 1],
  ["version info", "version", [], 3600, 5], ["record count (today)", "record", [], 3600, 1],
  ["p95 latency (total)", "latency", [], 60, 1], ["p95 latency (service)", "latency", ["service"], 60, 5],
  ["span duration (service)", "span_duration", ["service"], 60, 5], ["client version distribution", "client", ["version"], 3600, 40],
  ["node count", "node_count", [], 300, 1], ["queue depth", "queue", ["service"], 60, 5],
  ["cache hit rate", "cache", ["service"], 60, 5], ["garbage collection duration", "gc", ["node"], 60, 120],
  ["error rate (service)", "error_rate", ["service"], 60, 5], ["retry count", "retry", [], 60, 1],
  ["network traffic", "network", ["node"], 60, 240], ["deployment events", "deploy", ["service"], 60, 5],
  ["nightly batch job duration", "batch", [], 3600, 1], ["hourly read count", "read", [], 3600, 1],
  ["dependency latency", "span_duration", ["dependency"], 60, 12], ["error rate (region)", "error_rate", ["region"], 300, 30],
  ["meter type distribution", "meter_type", ["type"], 3600, 8], ["CPU (single node)", "cpu", ["node"], 60, 1],
];
// Question set: what is asked, in order, during an incident (priority order = array order).
export const QUESTIONS = [
  ["is a member affected", "error_rate", [], 60, 40], ["which service is broken", "error_rate", ["service"], 60, 38],
  ["which span has the latency", "span_duration", ["service"], 60, 30], ["did traffic change", "request", ["service"], 60, 25],
  ["is there a new deployment", "deploy", ["service"], 60, 22], ["is the queue backing up", "queue", ["service"], 60, 16],
  ["did retries increase", "retry", [], 60, 12], ["is the external dependency slow", "span_duration", ["dependency"], 60, 10],
  ["is the error specific to one region", "error_rate", ["region"], 60, 9], ["was the nightly batch job affected", "batch", [], 3600, 6],
  ["did the cache hit rate drop", "cache", ["service"], 60, 5], ["which subscriber group is affected", "error_rate", ["subscriber_group"], 60, 4],
];
export const PANEL_SECONDS = 8, QUERY_MINUTES = 4, INCIDENTS = 40;
export const answers = (p, q) => p[1] === q[1] && q[2].every((b) => p[2].includes(b)) && p[3] <= q[3];
export const audit = (panels) => {
  const found = QUESTIONS.map((q) => panels.findIndex((p) => answers(p, q)) + 1);   // 0: no panel
  const used = panels.filter((p) => QUESTIONS.some((q) => answers(p, q)));
  const weight = QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] : 0), 0);
  const timeToAnswer = QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] * found[i] : 0), 0) / weight;
  const minutes = (QUESTIONS.reduce((a, q, i) => a + (found[i] ? q[4] * found[i] * PANEL_SECONDS
    : q[4] * QUERY_MINUTES * 60), 0)) / INCIDENTS / 60;
  return { panels: panels.length, used: used.length, decorative: panels.length - used.length,
    answered: found.filter((b) => b > 0).length, noPanel: QUESTIONS.filter((_, i) => found[i] === 0).map((q) => q[0]),
    timeToAnswer, minutes, series: panels.reduce((a, p) => a + p[4], 0), found };
};

if (import.meta.filename === process.argv[1]) {
  const d = audit(PANELS);
  console.log(`${d.panels} panels, ${QUESTIONS.length} questions`);
  console.log(`panels answering a question ${d.used}, answering none ${d.decorative}`);
  console.log(`questions answered ${d.answered}, questions with no panel ${d.noPanel.length}: ${d.noPanel.join(", ")}`);
  console.log(`time to answer (panels scanned, weighted by question frequency) ${d.timeToAnswer.toFixed(1)}, ` +
    `worst ${Math.max(...d.found)}`);
  console.log(`dashboard minutes per incident ${d.minutes.toFixed(1)}, series scanned on open ${d.series}\n`);
  console.log(`${"question".padEnd(37)}${"panels scanned".padStart(17)}  panel that answered`);
  QUESTIONS.forEach((q, i) => console.log(`${q[0].padEnd(37)}${(d.found[i] || "-").toString().padStart(17)}  ` +
    `${d.found[i] ? PANELS[d.found[i] - 1][0] : "none"}`));
  const decorative = PANELS.filter((p) => !QUESTIONS.some((q) => answers(p, q)));
  console.log(`\ndecorative panels (${decorative.length}): ${decorative.map((p) => p[0]).join(", ")}`);
  console.log(`series scanned by decorative panels ${decorative.reduce((a, p) => a + p[4], 0)} ` +
    `(${(decorative.reduce((a, p) => a + p[4], 0) / d.series * 100).toFixed(0)}% of the total)`);
}

26 panels, 12 questions
panels answering a question 10, answering none 16
questions answered 10, questions with no panel 2: is the error specific to one region, which subscriber group is affected
time to answer (panels scanned, weighted by question frequency) 13.0, worst 23
dashboard minutes per incident 10.1, series scanned on open 859

question                                panels scanned  panel that answered
is a member affected                                 6  error rate (total)
which service is broken                             17  error rate (service)
which span has the latency                          11  span duration (service)
did traffic change                                   5  request count
is there a new deployment                           20  deployment events
is the queue backing up                             14  queue depth
did retries increase                                18  retry count
is the external dependency slow                     23  dependency latency
is the error specific to one region                  -  none
was the nightly batch job affected                  21  nightly batch job duration
did the cache hit rate drop                         15  cache hit rate
which subscriber group is affected                   -  none

decorative panels (16): cluster CPU usage, cluster memory usage, node disk usage, total requests (daily), version info, record count (today), p95 latency (total), p95 latency (service), client version distribution, node count, garbage collection duration, network traffic, hourly read count, error rate (region), meter type distribution, CPU (single node)
series scanned by decorative panels 814 (95% of the total)

Ten of the twenty-six panels answer a question, sixteen answer none. These sixteen panels are not wrong — cluster CPU, node memory, client version distribution all draw correct numbers. But they have no counterpart in this operations question set, so for the person looking during an incident they are just ground to cover.

Their cost is not just taking up space either: of the 859 series scanned when the dashboard opens, 814, that is 95%, come from panels that answer no question. Nearly the entire query cost buys nothing. The reason the ratio is this high is readable too: node-broken-down panels scan 120 series each, service-broken-down panels scan 5 each. How fine the breakdown is does not show up in the panel’s visual size, it shows up in the query cost.

The label “decorative” is not absolute, it is relative to the question set. The cluster CPU and node memory panels are the exact counterpart of capacity questions; the audit here measures not capacity but what gets asked during an incident. That is why these panels are not deleted, they move to a separate dashboard — and that dashboard’s own audit is run with its own question set. The same distinction explains why two CPU panels stand side by side on the dashboard: one was added for the cluster, the other for a single node, both answer the same question, and neither has a counterpart on the incident dashboard.

Two questions are without a panel. That the region-breakdown question is left without a panel is surprising at first, because there is a region-broken-down error-rate panel on the dashboard — but it is drawn at five-minute resolution and the question asks for a one-minute window. The panel exists, the answer does not. The second panelless question ties to something else entirely: the subscriber-group dimension was never collected, so that question can only be opened by instrumentation, not by dashboard layout.

Time to answer in this layout is 13.0 panels. That is, to find an answer to a question, weighted by the question’s frequency, an average of thirteen panels are scanned by eye; twenty-three for the worst question. At eight seconds per panel, that comes to 10.1 minutes per incident.

Reordering the Dashboard

The reordering is done in four steps, and the same audit runs again after every step.

// dashboard/reorder.mjs — the dashboard is reordered against the question set. Four stages are applied in
// sequence and the same audit runs again after each one. MD16: decorative panels are not deleted, they move
// from the incident dashboard to a capacity dashboard; their cost there is that dashboard's own series scan.
import { PANELS, QUESTIONS, audit, answers, INCIDENTS, QUERY_MINUTES } from "./dashboard.mjs";
const copy = PANELS.map((p) => [...p]);
const resolutionFix = copy.map((p) => (p[0] === "error rate (region)" ? [p[0], p[1], p[2], 60, p[4]] : p));
const kept = resolutionFix.filter((p) => QUESTIONS.some((q) => answers(p, q)));
// Ordering rule: questions are walked in frequency order; for each question, the first
// not-yet-placed panel that answers it is put in the queue; the rest are appended at the end.
const ordered = [];
for (const q of QUESTIONS) { const p = kept.find((x) => answers(x, q) && !ordered.includes(x));
  if (p) ordered.push(p); }
for (const p of kept) if (!ordered.includes(p)) ordered.push(p);
const STAGES = [["before", copy], ["resolution fix", resolutionFix],
  ["decorative panels split off", kept], ["ordered by question", ordered]];

console.log(`${"stage".padEnd(29)}${"panels".padStart(8)}${"decorative".padStart(12)}${"answered".padStart(11)}` +
  `${"no panel".padStart(10)}${"time to answer".padStart(16)}${"min/incident".padStart(14)}${"series".padStart(8)}`);
const results = STAGES.map(([name, p]) => {
  const d = audit(p);
  console.log(`${name.padEnd(29)}${String(d.panels).padStart(8)}${String(d.decorative).padStart(12)}` +
    `${String(d.answered).padStart(11)}${String(d.noPanel.length).padStart(10)}` +
    `${d.timeToAnswer.toFixed(1).padStart(16)}${d.minutes.toFixed(1).padStart(14)}${String(d.series).padStart(8)}`);
  return d;
});
const [first, last] = [results[0], results[3]];
console.log(`\n${"question".padEnd(37)}${"before".padStart(7)}${"after".padStart(7)}`);
QUESTIONS.forEach((q, i) => console.log(`${q[0].padEnd(37)}${(first.found[i] || "-").toString().padStart(7)}` +
  `${(last.found[i] || "-").toString().padStart(7)}`));
console.log(`\ntime to answer went from ${first.timeToAnswer.toFixed(1)} panels to ${last.timeToAnswer.toFixed(1)} panels, ` +
  `dashboard minutes per incident from ${first.minutes.toFixed(1)} min to ${last.minutes.toFixed(1)} min`);
console.log(`gained across forty incidents: ${((first.minutes - last.minutes) * INCIDENTS).toFixed(0)} minutes; ` +
  `series scanned ${first.series} instead of ${last.series}`);
const remaining = QUESTIONS.filter((q) => !ordered.some((p) => answers(p, q)));
console.log(`still without a panel: ${remaining.length} questions: ${remaining.map((q) => q[0]).join(", ")} — ` +
  `${remaining.reduce((a, q) => a + q[4], 0)} incidents, ${remaining.reduce((a, q) => a + q[4] * QUERY_MINUTES, 0)} minutes of manual queries`);

stage                          panels  decorative   answered  no panel  time to answer  min/incident  series
before                             26          16         10         2            13.0          10.1     859
resolution fix                     26          15         11         1            13.5          10.0     859
decorative panels split off        11           0         11         1             4.8           3.8      75
ordered by question                11           0         11         1             4.1           3.3      75

question                              before  after
is a member affected                       6      1
which service is broken                   17      2
which span has the latency                11      3
did traffic change                         5      4
is there a new deployment                 20      5
is the queue backing up                   14      6
did retries increase                      18      7
is the external dependency slow           23      8
is the error specific to one region        -      9
was the nightly batch job affected        21     10
did the cache hit rate drop               15     11
which subscriber group is affected         -      -

time to answer went from 13.0 panels to 4.1 panels, dashboard minutes per incident from 10.1 min to 3.3 min
gained across forty incidents: 273 minutes; series scanned 859 instead of 75
still without a panel: 1 questions: which subscriber group is affected — 4 incidents, 16 minutes of manual queries

The first step fixes the resolution of a single panel, and the number of answered questions rises from 10 to 11. What stands out is that time to answer increases at this step: from 13.0 to 13.5. The newly answered question is found in a panel at the twenty-fourth position on the dashboard, and it pulls the average up. Answering one more question does not mean that answer is findable.

The real drop is at the second step. Once the fifteen panels answering no question are pulled from the incident dashboard and moved to the capacity dashboard, the panel count drops from 26 to 11, time to answer from 13.5 to 4.8, series scanned from 859 to 75. The third step orders the panels by question frequency and brings time to answer down to 4.1; now every question is found at its own position, and the most frequently asked question is at the first panel.

The total gain is dashboard time dropping from 10.1 minutes to 3.3 minutes per incident; 273 minutes across forty incidents. All of that time is time added to the incident’s length — the person browsing panels is not doing anything else at that moment.

Measuring time to answer by panel count is a simplification: in reality a person does not scan the dashboard from start to end, they jump to where they know. What the simplification holds onto is the ratio: the number of panels someone who does not know, or who just woke up, browses through is directly proportional to the dashboard’s layout. It is not the number itself that should be read, but the nearly threefold difference between the two layouts.

The Question Left Over

After the reordering, the one question left without a dashboard is the subscriber-group breakdown, and it is not closed by dashboard layout. It is asked in four of forty incidents, each time a manual query has to be written and it takes four minutes: 16 minutes across forty incidents. That number looks small, but how the question gets answered matters too — the result of a manually written query does not go into the dashboard, and the same query gets written again at the next incident.

The three numbers the audit produces are read together. Data that answers no question is cost: 62% of the panels, 95% of the query cost. A question left unanswered is risk: of the two questions, one was closed off by the wrong resolution, the other by an uncollected dimension. Where the answer sits is also cost: the same data, the same panels, gained more than six minutes per incident just by changing the order.

Summary

• A panel answers a question only if it meets three conditions at once: the same metric, the dimension the question needs, the resolution the question needs.
• 16 of the 26 panels answered no question; these panels accounted for 814 of the 859 series scanned on dashboard open, that is 95% of the query cost.
• Two of the twelve questions were without a panel: one from the wrong resolution (the panel existed, the answer did not), the other from a dimension that was never collected.
• The resolution fix raised answered questions to 11 but raised time to answer from 13.0 to 13.5; an answer existing does not mean it is findable.
• Splitting off the decorative panels and ordering panels by question brought time to answer down to 4.1 panels, dashboard minutes per incident from 10.1 to 3.3; 273 minutes across forty incidents.
• Whether a panel is decorative is not absolute, it is relative to the question set: panels that are split off are not deleted, they move to a separate dashboard with its own question set.

Next Step

A reordered dashboard only works when it is looked at. If nobody is looking at three in the morning, the ordering of eleven panels changes nothing; someone has to be paged. The rule that triggers the page, though, is not written in a single form: it can page when a threshold is crossed, when a rate changes at a certain speed within a certain window, or when a band learned from past behavior is left. Three rule types catch different incidents within the same incident set. The next lesson runs all three over the same known incident sequence and compares them by caught count, missed count, false-alarm count, and detection round.