Common Fault Patterns

The previous lesson’s whole measurement rested on a single assumption: capture was up and running before the fault began. This lesson lifts that assumption and pays the course’s third claim: evidence does not wait. A recording not taken while a fault is running cannot be taken afterward; an event with no recording never happened, as far as diagnosis is concerned.

The lesson’s first half finishes the practical question left over. The previous three lessons measured tests and their orders; a bundle of responses is now in hand. Now the view turns around: a given fault produces which bundle of responses, that is, what is its signature. Three patterns are taken up separately — name resolution, firewall, and path — because all three are common and all three produce symptoms that look alike.

Three Patterns and Their Signatures

A fault’s signature is the tuple of responses the tests give for it (SG17). The signature is what diagnosis looks for: instead of narrowing the candidate set step by step, comparing the observed tuple against known tuples.

The name-resolution pattern has the most distinct signature. The name query gets no response, but everything done by address works: echo returns, connection is established, the large packet gets through. The picture on the user’s side is odd — the service works “sometimes,” because when a cached record is used the query is not made at all. Name resolution’s protocol itself is the Computer Networks curriculum’s subject and is not opened here; the subject here is only the signature.

The firewall pattern’s signature is silence. When a packet is dropped, no response comes from the other side at all, and the connection attempt times out. This is exactly where it splits from being refused: refusal produces a response, dropping does not. The source of the silence this time is not the path but a rule in front of the target, and the path tests say sound.

The path pattern is the sneakiest of the three. Small packets get through, large ones get lost. Echo says sound, connection is established, the first exchange works; it stops once it comes to a large data transfer. The user reports this as “slow” or “drops sometimes,” and diagnosis starts being looked for somewhere related to speed.

# example dump , has not been run
$ getent hosts <server-name>             # name-resolution signature
$ nc -z -w 3 <target> <port>             # refused , or timeout
$ ping -c 3 -M do -s 1400 <target>       # path signature
$ nft list ruleset                       # rules and counters read

HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True,
           "route_present": True, "resolver_responds": True, "service_listening": True,
           "firewall_passes": True, "path_ok": True}
FAULT = {
    "interface-down":      {"interface_up": False, "address_set": False},
    "no-address":          {"address_set": False},
    "wrong-gateway":       {"gateway_correct": False},
    "missing-route":       {"route_present": False},
    "resolver-silent":     {"resolver_responds": False},
    "service-not-listening": {"service_listening": False},
    "firewall-dropping":   {"firewall_passes": False},
    "large-packet-lost":   {"path_ok": False},
}
CANDIDATES = tuple(FAULT)


def state(a=None):
    d = dict(HEALTHY)
    if a:
        d.update(FAULT[a])
    return d


def path(d):
    return (d["interface_up"] and d["address_set"]
            and d["gateway_correct"] and d["route_present"])


TEST = {
    "name-lookup": lambda d: "resolved" if d["resolver_responds"] else "unresolved",
    "listening": lambda d: "listening" if d["service_listening"] else "not-listening",
    "connection": lambda d: ("unreachable" if not path(d)
                             else "timeout" if not d["firewall_passes"]
                             else "refused" if not d["service_listening"]
                             else "established"),
    "large-packet": lambda d: ("passes" if path(d) and d["path_ok"]
                               else "blocked"),
}
UPPER = list(TEST)                       # SG17: signature = tuple of test responses


def signature(a, kit=UPPER):
    return tuple(TEST[s](state(a)) for s in kit)


def groups(kit, candidates=CANDIDATES):
    o = {}
    for a in candidates:
        o.setdefault(signature(a, kit), []).append(a)
    return o


print("fault                       name-lookup   listening      connection   large-packet")
for a in CANDIDATES:
    i = signature(a)
    print(f"  {a:26s}{i[0]:14s}{i[1]:15s}{i[2]:13s}{i[3]}")
o = groups(UPPER)
print("distinct signatures:", len(o), "| unsplit groups:",
      len([v for v in o.values() if len(v) > 1]),
      "| largest group:", max(len(v) for v in o.values()))
print()
print("a test's elimination power changes with its response")
for s in TEST:
    for response, fits in sorted(groups([s]).items()):
        print(f"  {s:13s} {response[0]:12s} eliminated {len(CANDIDATES) - len(fits)}  left {len(fits)}")
print()


def intermittent(window=600, period=45, duration=6):    # SG18
    return [t for t in range(window) if t % period < duration]


def recording_catches(events, start):                   # SG19
    y = [t for t in events if t >= start]
    return len(y), round(len(y) / len(events), 4)


def dropped_packet_trace(recording_on):                 # SG20
    return 128 if recording_on else 0


EVENTS = intermittent()
print("intermittent fault: window 600 s , event", len(EVENTS), "seconds ,",
      len(EVENTS) // 6, "attacks x 6 seconds")
for start in (0, 100, 300, 500):
    y, ratio = recording_catches(EVENTS, start)
    print(f"  recording starts at second {start:3d} -> caught {y:3d} / {len(EVENTS)} = {ratio}")
print("dropped packet trace: recording off ->", dropped_packet_trace(False),
      "lines , recording on ->", dropped_packet_trace(True), "lines")
print()
print("candidate set sweep: name-lookup test's two responses")
kit = ["resolver-silent"]
for a in CANDIDATES:
    if a not in kit:
        kit.append(a)
        o = groups(["name-lookup"], tuple(kit))
        negative = len(o[("unresolved",)])
        positive = len(o.get(("resolved",), []))
        print(f"  candidate {len(kit)}  'unresolved' eliminated {len(kit) - negative} left {negative}"
              f"   'resolved' eliminated {len(kit) - positive} left {positive}")

fault                       name-lookup   listening      connection   large-packet
  interface-down            resolved      listening      unreachable  blocked
  no-address                resolved      listening      unreachable  blocked
  wrong-gateway             resolved      listening      unreachable  blocked
  missing-route             resolved      listening      unreachable  blocked
  resolver-silent           unresolved    listening      established  passes
  service-not-listening     resolved      not-listening  refused      passes
  firewall-dropping         resolved      listening      timeout      passes
  large-packet-lost         resolved      listening      established  blocked
distinct signatures: 5 | unsplit groups: 1 | largest group: 4

a test's elimination power changes with its response
  name-lookup   resolved     eliminated 1  left 7
  name-lookup   unresolved   eliminated 7  left 1
  listening     listening    eliminated 1  left 7
  listening     not-listening eliminated 7  left 1
  connection    established  eliminated 6  left 2
  connection    refused      eliminated 7  left 1
  connection    timeout      eliminated 7  left 1
  connection    unreachable  eliminated 4  left 4
  large-packet  blocked      eliminated 3  left 5
  large-packet  passes       eliminated 5  left 3

intermittent fault: window 600 s , event 84 seconds , 14 attacks x 6 seconds
  recording starts at second   0 -> caught  84 / 84 = 1.0
  recording starts at second 100 -> caught  66 / 84 = 0.7857
  recording starts at second 300 -> caught  42 / 84 = 0.5
  recording starts at second 500 -> caught  13 / 84 = 0.1548
dropped packet trace: recording off -> 0 lines , recording on -> 128 lines

candidate set sweep: name-lookup test's two responses
  candidate 2  'unresolved' eliminated 1 left 1   'resolved' eliminated 1 left 1
  candidate 3  'unresolved' eliminated 2 left 1   'resolved' eliminated 1 left 2
  candidate 4  'unresolved' eliminated 3 left 1   'resolved' eliminated 1 left 3
  candidate 5  'unresolved' eliminated 4 left 1   'resolved' eliminated 1 left 4
  candidate 6  'unresolved' eliminated 5 left 1   'resolved' eliminated 1 left 5
  candidate 7  'unresolved' eliminated 6 left 1   'resolved' eliminated 1 left 6
  candidate 8  'unresolved' eliminated 7 left 1   'resolved' eliminated 1 left 7

Oracle, Test, Eliminated Candidate

Three numbers sit side by side. Oracle: the real fault is resolver-silent. Test: the name query returns unresolved. Eliminated candidate: this response eliminates 7 candidates and leaves 1 behind; the signature is unique and the diagnosis finishes.

The same test, when the oracle is firewall-dropping, says resolved and eliminates only 1 candidate. That is, the name-resolution test was the weakest tool in one lesson and here it is the strongest. What changed is not the tool, it is the response.

The signature table splits eight faults into 5 distinct signatures using four upper-layer tests. The one unsplit group is the lower layer’s four: interface, address, gateway, and route faults all produce the same tuple across these four tests. This is the reverse of the result measured in the second lesson, where reachability tests left the upper four in a single group. The result is symmetric: every test kit blinds itself to what sits outside its own layer, and the split completes only once tests are drawn from both layers.

A Negative Response Eliminates, a Positive One Does Not

The second table is this lesson’s most generalizable finding. A test’s elimination power is not a fixed number; it changes with its response, and the gap is large. Name lookup eliminates 7 candidates when it says unresolved, 1 when it says resolved. The listening list eliminates 7 when it says not-listening, 1 when it says listening. The connection attempt’s four responses eliminate 6, 7, 4, and 7 candidates respectively.

The reason for the pattern sits in the mock setup’s structure: because each fault breaks a single field of the healthy state, that field coming up broken points to that one fault; coming up healthy is consistent with the remaining seven. The practical takeaway is direct: good news is little information. The line “name resolution is working” advances the diagnosis almost not at all; the line “name resolution is not working” finishes it.

This also changes test selection. A test’s expected elimination power depends on how its responses are distributed over the candidate set. A test with a balanced response distribution does similar work in every case; a test with a heavily skewed distribution rarely does much and mostly does little. The connection attempt’s four responses are the most balanced in this respect, eliminating at least 4 candidates even in the worst case.

Intermittent Fault: Evidence Does Not Wait

Up to here, every test was run while the fault was active. There is no such guarantee for an intermittent fault. In the mock setup, the fault lasts 6 seconds every 45 seconds; a 600-second window holds 14 attacks and a total of 84 seconds of fault (SG18). The recording starts at one moment and runs to the window’s end (SG19).

The numbers are plain. If the recording starts at the zeroth second, 84 of 84 events are seen. If it starts at the hundredth second, 66 (0.7857). At the three-hundredth second, 42 (0.5000). At the five-hundredth second, 13 (0.1548). Starting halfway through the recording window loses half of the events; starting five-sixths through loses five-sixths.

Two conclusions follow. First, a late-starting recording measures the wrong frequency. An operator looking at a recording that started at the five-hundredth second sees thirteen seconds of fault and calls the event “occasional”; the real duration is eighty-four seconds. What is wrong is not the observation, it is the assumption about how much of a window that observation represents.

The damage an intermittent fault does to the signature method is even heavier. A signature is correct only when taken while the fault is running. In the mock setup, the fault occupies 84 of the window’s 600 seconds, that is, 0.1400 of it; that is the share of a single test done at a random moment landing on the fault. In the remaining eighty-six percent, the test returns healthy, and by the rule measured in the previous section, that response eliminates the real cause. The candidate eliminated from the set is the correct one, and this shows no sign anywhere along the way: every test is consistent, the signature is a healthy system’s signature, and the service still does not work.

For this reason, the right move against an intermittent fault is to repeat the test, but the warning written in the previous lesson loses its force here: there, repeating the same test eliminated no new candidate, because the system’s state was fixed. If the state changes over time, repetition produces new information, and the information produced is no longer a single response, it is a response distribution. A test that comes back healthy in two of three tries and broken in one says something no single response could say.

Second, the recording’s start moment is a diagnostic decision, and it is made before the fault is reported. Opening a recording once a fault is heard makes everything from that moment on visible; none of the fourteen prior attacks come back. Log rotation was measured, in the system administration course, as erasing evidence; here evidence is not erased, it is never written in the first place. The two share the same conclusion, and the same remedy: the decision has to be made before the fault occurs.

The Trace a Dropped Packet Leaves

The last number is the shortest one. When the firewall drops a packet, the default behavior is to leave no trace: with the recording off, the dropped packet’s trace is 0 lines; with it on, 128 lines (SG20). Both cases are the same fault; the only thing that changed is a setting.

This is the third claim in its harshest form. Silent dropping is diagnosis’s most expensive fault, because no test sees it directly; it is only inferred from the timeout signature. With tracing on, the same fault is read in a single line. That is, the 128-line gap stands in for what would otherwise be a six-test inference.

The trace’s cost has to be written up too. If every dropped packet produces a line, a heavy drop rate turns the recording itself into a load, speeding up log rotation and erasing other evidence. The right setup is not continuous tracing but bounded tracing: for a specific rule, for a set duration, with a capped count. This is a decision made before the fault; made during the fault, it comes too late.

The same arithmetic combined with an intermittent fault produces the worst case: a silently dropped packet, if the fault is also intermittent, shows up in neither the trace nor the signature. Such a fault is read only from a repeated test’s response distribution, and that distribution never forms unless someone repeats the test.

Candidate Set Sweep

The sweep shows how the gap between a negative and a positive response grows with the set. In a two-candidate set, both responses eliminate 1 candidate; there is no difference. Once the set grows to eight, unresolved eliminates 7 candidates, resolved still eliminates 1. The negative response’s power grows linearly with the set, the positive one’s stays fixed.

This is a result that does not depend on the mock setup, and its reason is structural: because only a single fault breaks a given field, that field coming up broken points to the one fault regardless of candidate count; coming up healthy is consistent with the remaining seven. Adding a new fault to the candidate list lowers the positive response’s value and raises the negative response’s. Widening the list is therefore not “being more careful” — it changes the measurement itself, and it changes which test will do the job.

Summary

• A fault’s signature is the tuple of responses the tests give for it; four upper-layer tests split eight faults into 5 signatures, and the lower layer’s four faults stay in one group.
• A test’s elimination power changes with its response: name lookup eliminates 7 when it says unresolved, 1 when it says resolved. Good news is little information.
• An intermittent fault lasts 84 seconds in a 600-second window; a recording starting at second 300 catches 42/84 (0.5000), one starting at second 500 catches 13/84 (0.1548).
• A late-starting recording does not just see less, it measures the wrong frequency; a thirteen-second observation produces the conclusion “occasional.”
• A dropped packet’s trace is 0 lines with recording off, 128 with it on; the fault is the same fault, visibility is a setting, and that setting has to be turned on before the fault.
• A negative response’s elimination power grows with the candidate set, a positive response’s stays fixed at 1.

Next Step

This topic showed how much of diagnosis depends on decisions made beforehand: the recording being on, tracing being kept bounded, the candidate list being written down. The same question comes back harder on the access side. Connecting to a server remotely is itself a configuration decision, and if that decision is wrong, the channel that would do the diagnosing closes too. The next topic covers secure shell access, key-based authentication, and local hardening; then it moves to the packet-filtering framework and measures the growth of the externally exposed surface by counting it.