The Layered Diagnosis Method

The previous lesson worked out the local service inventory and set two tests’ elimination power side by side: the listening socket list eliminated only one candidate cause, a connection attempt made from outside eliminated seven. There are now eight separate tests in hand, and all of them are runnable. This lesson’s question is not which test to run, but in what order.

The question is not an empty one. Trying whatever comes to mind first when facing a fault, reaching for the most familiar tool, or assuming last time’s cause are all orders, and each carries a cost. The unit of that cost is settled too: how many tests get run. This lesson runs three orders over the same eight faults and counts the difference between them. One of the numbers will come out surprisingly small; why that number cannot be turned into a method is this lesson’s real subject.

The Candidate Cause Set

Diagnosis is the work of narrowing a set. At the start there is one symptom — “cannot connect to the server” — and more than one cause that could produce it. The course’s mock setup fixes this set as eight candidate causes: the interface being down, the address not being assigned, the default gateway being wrong, the route being missing, the resolver not responding, the service not listening, the firewall dropping the packet, and a large packet getting lost along the way.

All eight tests operate on this same set. Each test corresponds to a tool, returns a response, and that response splits the candidate set. The oracle is the real fault: because we chose the mock setup, we know which fault is active. The operator does not know; all the operator has are responses.

The mock setup’s assumptions: there is one client, one server, and a single target service, and the candidate-cause set is limited to these eight (SG1); each fault breaks a single field of the healthy state, and two faults never occur at once (SG2); a test returns one stable response, and the process stops once the candidate count drops to one (SG3); the greedy order picks, at every step, the test that eliminates the most by knowing the real fault (SG4); every test’s cost is equal and counts as one step (SG5).

Where the list itself comes from is a separate question, and it is diagnosis’s least discussed step. The candidate-cause set is derived from the symptom: which components could produce this symptom, which could not. Keeping the set narrow lowers the number of tests but grows the risk of leaving out the real cause; keeping it wide does the opposite. The eight-candidate mock setup is a closed-world assumption: the real fault is taken to be on the list. When the closed world breaks, what breaks is not the measurement itself but its interpretation — every test finishes, one candidate remains, and that candidate is wrong. A diagnosis order finishing does not mean it finished correctly.

Another source that narrows the set is the logs measured in the system administration course. If a service was restarted, a unit failed, or a configuration change was logged, this puts some candidates forward before any test is run. The role of logs here is not evidence but a prior: it affects the ordering, it does not eliminate a candidate. Elimination only happens through a test’s response.

Protocol layers themselves, handshakes, and routing protocols are the subject of the Computer Networks curriculum and are not built here. This course looks at a single machine’s configuration and diagnosis. Namespaces and container networking are not covered either; they are left to the Kernel Interfaces and Isolation course.

The Layered Order and Its Two Rivals

Layered diagnosis orders tests by dependency: first the interface, then the address and route, then local-network reach, then remote reach, then name resolution, and lastly the service itself. The reasoning is intuitive — if the bottom is broken, the result at the top is already meaningless.

The reverse order reads the same list backward: it first tries connecting to the service, then works its way down. This too is a defensible order; an operator who thinks most faults sit on the service side starts right there.

The greedy order picks, at every step, the test that leaves the fewest candidates behind. All three orders use the same tests; the dump below shows which commands these correspond to and it has not been run. The measurement happens in the block that follows, and it runs all three orders over the same eight faults.

# example dump , has not been run
$ ip -brief link show                    # is the interface up
$ ip -brief addr show                    # is an address assigned
$ ip route show                          # default gateway and route
$ ping -c 3 <gateway>                    # echo to the local network
$ getent hosts <server-name>             # name resolution
$ ss -ltn                                # listening ports
$ nc -z -w 3 <target> <port>             # connection attempt

HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True,
           "route_present": True, "resolver_responds": True, "service_listening": True,
           "firewall_passes": True, "path_ok": True}

FAULT = {                                    # SG2: each fault breaks a SINGLE field
    "interface-down":      {"interface_up": False, "address_set": False},
    "no-address":          {"address_set": False},
    "wrong-gateway":       {"gateway_correct": False},
    "missing-route":       {"route_present": False},
    "resolver-silent":     {"resolver_responds": False},
    "service-not-listening": {"service_listening": False},
    "firewall-dropping":   {"firewall_passes": False},
    "large-packet-lost":   {"path_ok": False},
}
CANDIDATES = tuple(FAULT)


def state(fault=None):
    d = dict(HEALTHY)
    if fault:
        d.update(FAULT[fault])
    return d


def path(d):                        # interface + address + gateway + route together
    return (d["interface_up"] and d["address_set"]
            and d["gateway_correct"] and d["route_present"])


TEST = {
    "interface": lambda d: ("open-addressed" if d["interface_up"] and d["address_set"]
                            else "open-unaddressed" if d["interface_up"] else "down"),
    "route": lambda d: ("table-empty" if not d["address_set"]
                        else "complete" if d["route_present"] else "no-default"),
    "gateway-echo": lambda d: ("response" if d["interface_up"] and d["address_set"]
                               else "no-response"),
    "remote-echo": lambda d: "response" if path(d) else "no-response",
    "name-lookup": lambda d: "resolved" if d["resolver_responds"] else "unresolved",
    "listening": lambda d: "listening" if d["service_listening"] else "not-listening",
    "connection": lambda d: ("unreachable" if not path(d)
                             else "timeout" if not d["firewall_passes"]
                             else "refused" if not d["service_listening"]
                             else "established"),
    "large-packet": lambda d: ("passes" if path(d) and d["path_ok"]
                               else "blocked"),
}


def eliminate(candidates, s, response):
    return tuple(a for a in candidates if TEST[s](state(a)) == response)


def eliminates(candidates, s, real):
    remaining = eliminate(candidates, s, TEST[s](state(real)))
    return len(candidates) - len(remaining), len(remaining)


def cost(order, real, candidates=CANDIDATES):     # SG3: stop once one candidate remains
    remaining, step = tuple(candidates), 0
    for s in order:
        if len(remaining) <= 1:
            break
        step += 1
        remaining = eliminate(remaining, s, TEST[s](state(real)))
    return step


def greedy(real, candidates=CANDIDATES):           # SG4: looks at the oracle every step
    remaining, chosen = tuple(candidates), []
    while len(remaining) > 1 and len(chosen) < len(TEST):
        best, best_s = None, None
        for s in TEST:
            if s not in chosen:
                k = len(eliminate(remaining, s, TEST[s](state(real))))
                if best is None or k < best:
                    best, best_s = k, s
        chosen.append(best_s)
        remaining = eliminate(remaining, best_s, TEST[best_s](state(real)))
    return chosen


LAYERED = ["interface", "route", "gateway-echo", "remote-echo",
           "name-lookup", "listening", "connection", "large-packet"]
REVERSE = list(reversed(LAYERED))
ORACLE = "firewall-dropping"

print("candidate faults:", len(CANDIDATES), "| tests:", len(TEST), "| oracle:", ORACLE)
print("test            response        eliminated  left")
for s in LAYERED:
    e, k = eliminates(CANDIDATES, s, ORACLE)
    print(f"  {s:14s}{TEST[s](state(ORACLE)):16s}{e:6d} {k:6d}")
print()
print("fault                       layered  reverse  greedy  greedy order")
for a in CANDIDATES:
    g = greedy(a)
    print(f"  {a:26s}{cost(LAYERED, a):6d}{cost(REVERSE, a):6d}"
          f"{len(g):8d}  {', '.join(g)}")
print("total steps: layered", sum(cost(LAYERED, a) for a in CANDIDATES),
      "| reverse", sum(cost(REVERSE, a) for a in CANDIDATES),
      "| greedy", sum(len(greedy(a)) for a in CANDIDATES))
print()
print("candidate set sweep -- eliminated by connection test as the set grows")
group = [ORACLE]
for a in CANDIDATES:
    if a not in group:
        group.append(a)
        e, k = eliminates(tuple(group), "connection", ORACLE)
        print(f"  candidate {len(group)}  added {a:26s} eliminated {e}  left {k}")
print("layered total when one candidate is removed (33 at 8 candidates)")
for out in CANDIDATES:
    rest = tuple(x for x in CANDIDATES if x != out)
    print(f"  removed {out:26s} {sum(cost(LAYERED, a, rest) for a in rest):3d}")

candidate faults: 8 | tests: 8 | oracle: firewall-dropping
test            response        eliminated  left
  interface     open-addressed       2      6
  route         complete             3      5
  gateway-echo  response             2      6
  remote-echo   response             4      4
  name-lookup   resolved             1      7
  listening     listening            1      7
  connection    timeout              7      1
  large-packet  passes               5      3

fault                       layered  reverse  greedy  greedy order
  interface-down                 1     8       1  interface
  no-address                     1     8       1  interface
  wrong-gateway                  4     7       2  remote-echo, route
  missing-route                  2     7       1  route
  resolver-silent                5     2       1  name-lookup
  service-not-listening          6     2       1  listening
  firewall-dropping              7     2       1  connection
  large-packet-lost              7     2       2  connection, name-lookup
total steps: layered 33 | reverse 38 | greedy 10

candidate set sweep -- eliminated by connection test as the set grows
  candidate 2  added interface-down             eliminated 1  left 1
  candidate 3  added no-address                 eliminated 2  left 1
  candidate 4  added wrong-gateway              eliminated 3  left 1
  candidate 5  added missing-route              eliminated 4  left 1
  candidate 6  added resolver-silent            eliminated 5  left 1
  candidate 7  added service-not-listening      eliminated 6  left 1
  candidate 8  added large-packet-lost          eliminated 7  left 1
layered total when one candidate is removed (33 at 8 candidates)
  removed interface-down              32
  removed no-address                  32
  removed wrong-gateway               29
  removed missing-route               31
  removed resolver-silent             28
  removed service-not-listening       27
  removed firewall-dropping           25
  removed large-packet-lost           25

Oracle, Test, Eliminated Candidate

Three numbers sit side by side. Oracle: the real fault is firewall-dropping, that is, the firewall is silently dropping the packet. Test: the connection attempt returns timeout. Eliminated candidate: this single response eliminates 7 of the eight candidates and leaves 1 behind — the fault is resolved with a single test.

Under the same oracle, the name-lookup test returns resolved, the listening test returns listening; each eliminates only 1 candidate and leaves 7 behind. The resulting ratio is the lesson’s first reading: of two tests done with equal effort, one does seven times the work of the other. A test’s value is not in the response it gives, it is in how many candidates that response drops. A test whose response is correct but eliminates no candidate counts as unmeasured.

The total-step table is the second reading. When each of the eight faults is made the real fault in turn, the layered order spends a total of 33 steps, the reverse order 38, the greedy order 10. The layered order runs 3.3 times as many tests as the optimum. The reverse order’s loss is in the distribution: it finishes the four service-side faults in 2 steps, but spends the full 8 steps on the two interface-side faults.

Why the Greedy Order Is Not a Procedure

The number ten can pull the reader toward a wrong conclusion: the rule “pick the test that eliminates the most” looks like a usable method. It is not. For the rule to work, at every step, it has to compute what response every test will give under the real fault, which is not yet known. The line in the block that does this is the TEST[s](state(real)) call, and real is exactly what the diagnosis is trying to find.

In other words, the greedy order takes the oracle as input. Someone who knows the answer can pick the shortest path; someone who does not know it cannot make that choice. So the number 10 is not a target, it is an unreachable lower bound. It has one job in the measurement — it gives a scale showing how much extra work the layered order does — but it is never presented anywhere as a usable order.

This distinction is often confused in diagnosis writing. The sentence “an experienced operator looks straight at the right place” is the human form of the greedy order, and it carries the same flaw: experience is knowing the distribution of past faults, not the current one. When the past distribution matches today’s fault, the shortcut works; when it does not, the operator lives through the reverse order’s worst case — running all eight tests while the interface is down, for instance.

The Layered Order’s Defense

The layered method’s defense is not speed; 33 steps cost three times as much as 10, and that gap is real. The defense is this: the layered order works without an oracle. The order is fixed before the fault occurs and runs in the same order whichever fault is active. Its only input is the tests’ responses.

The second defense is that the intermediate information the order gives is cumulative. When the interface test returns open-addressed, it does not just eliminate two candidates; it also fixes how subsequent tests get interpreted. The reverse order has no such accumulation: once it is seen that the service cannot be connected to, the remaining four candidates are still unsplit, and every test going downward is interpreted from scratch.

The third defense is protection against a wrong diagnosis. While the reverse order runs, the urge arises to fix something on the service side — restarting the service, loosening a rule — and such fixes made while the layer beneath is broken change the system’s state. The layered order proves the bottom is sound before moving on to a fix.

A fourth point is a practical warning: the most expensive faults sit at the bottom of the table. firewall-dropping and large-packet-lost each cost 7 steps in the layered order. For an operator looking for a shortcut, the right move is not to break the order but to move the connection attempt earlier in it: because it returns four separate responses, it is the strongest single splitter on its own and can be moved up without breaking the order’s logic.

Candidate Set Sweep

There is no randomness in the mock setup; every number is a full count. How much the results depend on the mock setup is therefore tested not with a second seed but by changing the candidate set.

As the set grows, the number of candidates the connection test eliminates climbs from 1 to 7, and what is left is always 1. What follows from this matters: the elimination count is not a property of the tool alone, it is also a measure of the candidate set. The sentence “this tool eliminates seven candidates” is meaningful only once an eight-candidate set has been defined. Comparing tools without writing down the candidate list is writing a fraction with no denominator.

When one candidate is removed, the layered total drops from 33 to somewhere between 25 and 32. Where the drop lands depends on where the removed candidate sits in the order: removing interface-down, which sits at the front, drops the total by only 1 step (32); removing firewall-dropping or large-packet-lost, at the back, drops it by 8 steps (25). What determines the layered order’s cost is not the candidate count but where in the order the candidates sit. The reverse also holds: a newly added deep candidate can grow the total by as much as eight steps.

Summary

• Diagnosis is the work of narrowing a set; a test’s value is measured not by its response but by the number of candidate causes it eliminates, and a test whose elimination count is not written down counts as unmeasured.
• In the firewall-drop scenario, the connection attempt eliminates 7 candidates and resolves it alone; name lookup and the listening list each eliminate only 1.
• Total step cost across all eight faults: layered order 33, reverse order 38, oracle-assisted greedy order 10.
• The greedy order is not a procedure: picking the most eliminating test at every step requires knowing the real fault, so 10 is an unreachable lower bound.
• The layered method’s defense is not speed but that it works without an oracle; its order is fixed before the fault and takes only test responses as input.
• Elimination count depends on the candidate set: as the set grows from 2 to 8, what the connection test eliminates climbs from 1 to 7.

Next Step

The layered order’s first four steps are reachability tests, and the table’s weakest-looking entries sit right there: echo to the gateway eliminates only 2 candidates, echo to the remote address 4. These tests’ real problem is not their weakness, it is that they are misread: no response fits not one cause but several, and a response arriving does not say reach is sound either. The next lesson opens up the silence returned by echo and path-test tools and counts how many separate candidate causes a “no response” state corresponds to.