Troubleshooting Method

The previous lesson counted the trace management jobs leave and gave the measure of a scheme that holds while the machine works as expected. When the machine does not work as expected, only symptoms are left: a service cannot get data, a request comes back with no response, a job is delayed.

The path from symptom to cause is a method, and a method can be measured. This lesson ties it to a single number: how many possibilities are eliminated when a hypothesis is tested.

A Method Is an Elimination Count

Troubleshooting is usually described as something between guessing and trying. Making it measurable requires writing down three things.

The set of possibilities. Once the symptom is observed, which causes are still possible. Without writing this set down, the sentence “I’m looking for the cause” carries no number.

The set of tests. Each test probes a hypothesis and gives a result. A test’s power is how many possibilities its result eliminates — and this number is read in the worst case, because which result comes back is not known in advance.

Sequence. Tests are run in some order. The same test set finishes in a different number of steps with a different order; sequence is the one thing gained without acquiring a new tool.

Sixteen Possibilities, Eight Tests

The example is a symptom in the fictional regional measurement network: the branch server is not sending data to the billing service. Both the symptom and the cause list are fictional; the evaluation below is a model — there is no real incident record.

HO28: when the symptom is observed, 16 causes are possible. HO29: there are 8 tests; each test gives a single result for each cause, and causes that give the same result cannot be told apart by that test. HO30: when a test is run, the candidate set narrows to the causes that give the same result as the actual cause. HO31: three sequences are tried — info picks the test that eliminates the most in the worst case, list runs the tests in the order they were written, weak puts the test that eliminates the least in the worst case first. HO32: the run stops once the candidate set narrows to a single cause, or once no test can separate the set any further.

// it/diagnose.mjs — possibilities eliminated per hypothesis in a symptom set (model): fictional
// measurement network, no real incident record. No randomness.

// Symptom: the branch server is not sending data to the billing service. 16 possibilities at the start.
export const CAUSE = ["network-cable", "switch-port-state", "address-conflict",
  "name-resolution", "firewall-rule", "routing-table", "service-down", "service-wrong-port",
  "certificate-expiry", "credential", "permission-change", "disk-full", "memory-exhausted",
  "clock-skew", "version-mismatch", "configuration-error"];

// Each test probes a hypothesis. Every character in the string is that cause's result on the
// test; causes with the same character cannot be told apart by that test.
export const TEST = [
  ["network-connectivity", "hheeeeeeeeeeeeee"],
  ["address-and-routing", "eeceeyeeeeeeeeee"],
  ["name-resolution", "eeeheeeeeeeeeeee"],
  ["port-access", "eeeehekkeeeeeeee"],
  ["service-status", "eeeeeedyeeeeeeee"],
  ["identity-and-certificate", "eeeeeeeeskieeeee"],
  ["resource-status", "eeeeeeeeeeedbeee"],
  ["clock-version-config", "eeeeeeeeeeeeetsy"],
].map(([name, s]) => ({ name, s }));

// The largest group a test leaves in a candidate set: the worst-case remaining possibilities.
export const largest = (t, candidates) => Math.max(...[...new Set(candidates.map((c) => t.s[c]))]
  .map((v) => candidates.filter((c) => t.s[c] === v).length));

export const SEQUENCE = {
  // info-priority: the test that eliminates the most in the worst case. Stop if none eliminate.
  info: (candidates, used) => {
    const k = TEST.map((t, i) => i).filter((i) => !used.has(i));
    if (!k.length) return null;
    const e = k.reduce((a, i) => (largest(TEST[i], candidates) < largest(TEST[a], candidates) ? i : a));
    return largest(TEST[e], candidates) === candidates.length ? null : e;
  },
  // list: tests in the order they were written.
  list: (candidates, used) => TEST.map((t, i) => i).find((i) => !used.has(i)) ?? null,
  // weak: the sequence that puts the test eliminating the least in the worst case first.
  weak: (candidates, used) => {
    const k = TEST.map((t, i) => i).filter((i) => !used.has(i));
    if (!k.length) return null;
    return k.reduce((a, i) => (largest(TEST[i], candidates) > largest(TEST[a], candidates) ? i : a));
  },
};

export function run(actual, sequence) {
  let candidates = CAUSE.map((c, i) => i);
  const used = new Set();
  let steps = 0, empty = 0;
  while (candidates.length > 1) {
    const i = SEQUENCE[sequence](candidates, used);
    if (i === null) break;
    used.add(i);
    const before = candidates.length;
    candidates = candidates.filter((c) => TEST[i].s[c] === TEST[i].s[actual]);
    steps++;
    if (candidates.length === before) empty++;
  }
  return { steps, empty, remaining: candidates.length };
}

// it/diagnosemeasure.mjs — how many possibilities are eliminated per hypothesis, which sequence finishes in how many steps.
import { CAUSE, TEST, largest, SEQUENCE, run } from "./diagnose.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const ALL = CAUSE.map((c, i) => i);
const S = Object.keys(SEQUENCE);
if (TEST.some((t) => t.s.length !== CAUSE.length)) throw new Error("test length mismatch");

console.log(`symptom: branch server not sending data; ${CAUSE.length} possibilities, ` +
  `${TEST.length} tests`);

const A = [-26, 14, 15, 18];
console.log("\n1. each test's power alone (over 16 possibilities)");
write(A, "test", "result types", "worst case", "least eliminated");
for (const t of TEST) {
  const b = largest(t, ALL);
  write(A, t.name, new Set(t.s).size, b, CAUSE.length - b);
}

const B = [-41, 10, 10, 10];
console.log("\n2. three sequences, with each of the 16 causes tried as the actual cause");
const R = Object.fromEntries(S.map((s) => [s, ALL.map((c) => run(c, s))]));
write(B, "measure", ...S);
for (const [name, f] of [["average steps", (r) => (r.reduce((a, x) => a + x.steps, 0) / r.length).toFixed(2)],
  ["worst-case steps", (r) => Math.max(...r.map((x) => x.steps))],
  ["steps eliminating nothing", (r) => r.reduce((a, x) => a + x.empty, 0)],
  ["scenarios narrowed to one cause (of 16)", (r) => r.filter((x) => x.remaining === 1).length]])
  write(B, name, ...S.map((s) => f(R[s])));

const C = [-38, 10, 10, 10];
console.log("\n3. step count for four causes");
write(C, "actual cause", ...S);
for (const c of [0, 7, 10, 15]) write(C, CAUSE[c], ...S.map((s) => run(c, s).steps));

const D = [-46, 10];
console.log("\n4. the method's floor");
const indistinct = ALL.filter((c) => ALL
  .some((d) => d !== c && TEST.every((t) => t.s[c] === t.s[d])));
write(D, "measure", "value");
write(D, "causes no test can tell apart", indistinct.length);
write(D, "smallest set reached with the info sequence", Math.min(...ALL.map((c) => run(c, "info").remaining)));
write(D, "largest set remaining with the info sequence", Math.max(...ALL.map((c) => run(c, "info").remaining)));
console.log(`indistinguishable causes: ${indistinct.map((c) => CAUSE[c]).join(", ")}`);

symptom: branch server not sending data; 16 possibilities, 8 tests

1. each test's power alone (over 16 possibilities)
test                        result types     worst case  least eliminated
network-connectivity                   2             14                 2
address-and-routing                    3             14                 2
name-resolution                        2             15                 1
port-access                            3             13                 3
service-status                         3             14                 2
identity-and-certificate               4             13                 3
resource-status                        3             14                 2
clock-version-config                   4             13                 3

2. three sequences, with each of the 16 causes tried as the actual cause
measure                                        info      list      weak
average steps                                  3.50      5.81      5.81
worst-case steps                                  6         8         8
steps eliminating nothing                         0        22        12
scenarios narrowed to one cause (of 16)          14        14        14

3. step count for four causes
actual cause                                info      list      weak
network-cable                                  4         8         8
service-wrong-port                             2         5         4
permission-change                              2         6         7
configuration-error                            3         8         8

4. the method's floor
measure                                            value
causes no test can tell apart                          2
smallest set reached with the info sequence            1
largest set remaining with the info sequence           2
indistinguishable causes: network-cable, switch-port-state

The numbers belong to the measurement class; their inputs are HO28–HO32.

Tests Are Uneven, the Split Is Not Half-and-Half

The first table breaks an expectation. None of the eight tests splits the possibility set down the middle: the worst-case remaining possibilities fall between 13 and 15, meaning even the strongest test eliminates only 3 of 16 possibilities. The name-resolution test eliminates just 1 possibility in the worst case.

The reason is in the number of result types a test has. If a test gives two results, it splits the possibilities into at most two; if it gives four results, into at most four. But the split does not have to be even, and in this model it is not: each test separates out a few causes and leaves the remaining thirteen, fourteen, or fifteen at the same result. In a system, tests are naturally uneven, because each test probes a specific layer, and causes outside that layer look identical in the test’s eyes.

The practical consequence of this is that picking the test that eliminates the most in the worst case is not finding a half-and-half split — it is finding the least bad split. The expectation of finishing 16 possibilities in four steps with binary search is not met by this test set.

What Sequence Changes

The second table runs the three sequences with each of the sixteen causes taken as the actual cause. The info-priority sequence finishes in an average of 3.50 steps; the list sequence and the weak sequence in 5.81. The worst-case steps are 6 for the info-priority sequence and 8 for the others — eight is the full test count, meaning these two sequences run every single test in the worst case.

The difference does not mean a forty-five-minute gap in a forty-five-minute incident; the cost per step varies from test to test. But the number still gives direction: with the same test set, the same information, and the same tools, changing only the sequence gains more than a third of the average steps.

The third table shows this does not hold for every cause. For service-wrong-port, the info-priority sequence takes 2 steps, the weak sequence 4, the list sequence 5. For permission-change, the weak sequence takes 7 steps and the list sequence 6 — the weak sequence is worse than the list sequence. For network-cable, both the list and weak sequences run 8 steps, while the info-priority sequence runs 4.

A Fixed Sequence Produces More Empty Steps Than an Adapted Bad One

The third row of the second table gives this lesson’s most unexpected number. The count of steps that eliminate nothing is 0 for the info-priority sequence, 22 for the list sequence, and 12 for the weak sequence.

The weak sequence deliberately puts the test that eliminates the least in the worst case first; yet it still produces only half as many empty steps as the list sequence. The reason is that the real difference between the two is not “good choice / bad choice” — it is adapted / not adapted. The weak sequence looks at the current candidate set on every step and does not pick a test that splits the set not at all if one remains available; the list sequence never looks at the candidate set and, in order, also runs tests that probe a layer already eliminated.

An empty step is troubleshooting’s invisible cost. It runs, takes time, produces a result, and that result does not change the candidate set — but the result itself looks like information, because the sentence “that turned out normal too” gives a feeling of progress. What must be counted is not the number of tests run but the number of possibilities eliminated. The list sequence’s 22 empty steps are 22 checks that ran but eliminated nothing.

Elimination Count Is Not the Only Measure

The model treats every cause as equally likely and ranks tests only by their eliminating power. An incident has two more things, and neither is in this table.

The first is the likelihood of the causes. Not all sixteen causes occur at the same frequency; a symptom that appears right after a configuration change is more likely to have configuration-error behind it than the others. The elimination count does not see this: it picks the test that eliminates the most, not the test that probes the most likely cause. When the two do not coincide, the choice is a trade-off between the expected step count and the worst-case step count. Closing the symptom quickly cares about the expected number; bounding the worst case cares about the worst-case number.

The second is the cost of the test. In the model, every step costs the same; in reality, one test finishes with a command in seconds, another with a field team traveling to the branch in hours. Choosing between an expensive test that eliminates three possibilities and a cheap test that eliminates two cannot be made by looking at the elimination count alone. The network-cable and switch-port-state distinction needing a new observation is exactly such an item: the cost of the distinction is a site visit.

Adding these two axes does not change the model — it changes how it is read. The 3.50 in the table holds under the condition that steps cost the same and causes are equally likely; without writing down these two assumptions, the number cannot be turned into a target. The measurement’s value shows up once it is written down which assumption produced it.

The Method’s Floor

The fourth table writes down where the method stops. Two of the sixteen causes cannot be told apart by any test: network-cable and switch-port-state. Both give the same result on every test, because seen from the server, their symptom is identical.

This is why the number of scenarios all three sequences narrow to a single cause is 14/16. In the remaining two scenarios, the method stops at two possibilities, and this is not a shortcoming — it is the test set’s resolution. Making the distinction does not require a new sequence, it requires a new test — in this model, an observation made from outside the server, from the other end of the connection.

This has a decision to match it. Writing a troubleshooting procedure starts with listing out the cause pairs the test set cannot separate; for those pairs, either a new observation is added, or a single fix is defined that can be applied without making the distinction. In the measurement network, progress can be made without distinguishing network-cable from switch-port-state: both causes are addressed in the same field team visit. Two indistinguishable causes are not a problem if their fixes are the same.

Summary

• Troubleshooting is measured by the number of possibilities eliminated per hypothesis; this number is read in the worst case, because which result comes back is not known in advance.
• Tests are uneven: the strongest of the eight tests eliminates only 3 of 16 possibilities in the worst case, the weakest just 1. The expectation of a half-and-half split is not met in real test sets.
• The info-priority sequence finishes in an average of 3.50 steps, the list and weak sequences in 5.81; the worst-case steps are 6 against 8. Sequence is the one thing gained without acquiring a new tool.
• The count of steps that eliminate nothing is 0, 22, and 12. The fixed list sequence produces twice as many empty steps as the adapted sequence that deliberately makes a weak choice; what decides is not the quality of the choice but whether the candidate set is looked at.
• Elimination count is not the only measure: the likelihood of causes and the cost of tests are not in the table. The 3.50 holds under the assumption that steps cost the same and causes are counted as equally likely.
• Two causes cannot be told apart by any test; all three sequences narrow 14 of 16 scenarios to a single cause. The remaining distinction requires not a new sequence but a new observation, and if the fixes are the same, the distinction may not be needed at all.

Next Step

This topic counted a single machine’s boundaries item by item. The eight boundaries hardware draws and two of them opening the entire inventory; operating system families diverging on the same request by 4, 11, and 12 items; the 17 risky items a clean install leaves and the 5 that never close; two permission models hitting their expressive limit on 6 requests; two management forms leaving the same 11 audit trail items; and now the path from symptom to cause dropping to 3.50 steps. The machine’s boundary is no longer an adjective — it is a number.

All of these measurements carried a silent assumption: the machine being counted runs on its own hardware. That is not true of most such machines. There is one more layer beneath the branch server, and that layer’s boundary was never measured: which items are shared between guest and host, how many items it takes to cross that boundary, and how many items become reachable when the boundary falls. The next topic, Virtualization and Redundancy, begins with this question.