The Routing Table

When the interface list gave the expected answer — up and addressed — only two of eight candidates were eliminated, and six remained. Two of these six concern where the packet is sent: the default gateway can be wrong, or the route to the destination may not exist at all. Both sit behind an up interface that has an address, and both are invisible from the interface list.

This lesson reads the routing table and applies the same measure to it. The result is instructive in two respects. Reading the table leaves a narrower group than the interface list did — a stronger test. Against this, the first approach that comes to mind for telling these two apart — sending an echo — cannot separate them despite two separate tests.

What the Table Says

The routing table is made of rows that determine, for a destination address, which interface and which next hop a packet is sent through. Rows are not tried in order; every row has a destination prefix, and the kernel picks the longest matching prefix. The default gateway is the row with the shortest prefix under this rule, and for that reason it is used only when no other row matches. When an address is assigned to an interface, that subnet’s row enters the table on its own; this is called a connected route, and it is what gives access to the local network.

# example dump — not executed
$ ip route show
default via 203.0.113.1 dev eth0
203.0.113.0/24 dev eth0 proto kernel scope link src 203.0.113.24
198.51.100.0/24 via 203.0.113.9 dev eth0

It is also possible to ask for the decision for a single destination instead of reading the whole table. The decision query states three things at once: the chosen row, the outgoing interface, and which address will be written as the packet’s source address. Source address selection does not show up in the table, and when it is chosen wrongly the fault becomes one-directional: the packet leaves, the reply cannot come back.

# example dump — not executed
$ ip route get 198.51.100.42
198.51.100.42 via 203.0.113.9 dev eth0 src 203.0.113.24

These two blocks are not executed, and no numeric claim is drawn from them; what they teach is the shape of the row. The addresses are fictional. How routing protocols work, how tables are distributed among neighbors, and how metrics are computed is the subject of the Computer Networks curriculum; in this lesson the table is taken as given and is only read.

• AY11 — The mock server’s table consists of three rows: the default gateway, the local subnet’s connected route, and a single static route written to a remote subnet. More than one routing table at a time is not assumed.
• AY12 — The wrong-gateway candidate is the case where the default row exists but the next hop it points to does not forward the packet. The missing-route candidate is the case where no row matches the destination at all. In the table, one is present and wrong, the other is absent.

The Route Test’s Measure

"""Routing table test: how many of 8 candidate causes are eliminated.
Same eight faults from the shared reference; this lesson reads the route and two echo tests."""
HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True,
           "route_present": True, "resolver_responds": True, "service_listening": True,
           "firewall_allows": True, "path_intact": True}
FAULT = {
    "interface-down":       {"interface_up": False, "address_present": False},
    "no-address":      {"address_present": False},
    "wrong-gateway":        {"gateway_correct": False},
    "missing-route":        {"route_present": False},
    "resolver-silent":      {"resolver_responds": False},
    "service-not-listening": {"service_listening": False},
    "firewall-dropping":    {"firewall_allows": False},
    "large-packet-lost": {"path_intact": False},
}


def state(fault, table):
    d = dict(HEALTHY)
    d.update(table[fault])
    return d


def test_route(d):
    if not d["address_present"]:
        return "table-empty"
    return "complete" if d["route_present"] else "default-missing"


def test_gateway_echo(d):
    return "reply" if d["interface_up"] and d["address_present"] else "no-reply"


def test_remote_echo(d):
    return "reply" if (d["interface_up"] and d["address_present"]
                       and d["gateway_correct"] and d["route_present"]) else "no-reply"


TEST = {"route": test_route, "gateway-echo": test_gateway_echo,
        "remote-echo": test_remote_echo}


def group_by(tests, table=FAULT):
    o = {}
    for a in table:
        key = tuple(TEST[s](state(a, table)) for s in tests)
        o.setdefault(key, []).append(a)
    return o


print("route test: candidates per answer")
for answer, group in group_by(["route"]).items():
    print(f"  {answer[0]:14s} remaining {len(group):2d}  eliminated {len(FAULT) - len(group):2d}"
          f"  {', '.join(group)}")
print()
print("indistinguishable groups as the tool set grows")
for tools in (["gateway-echo"], ["gateway-echo", "remote-echo"],
             ["gateway-echo", "remote-echo", "route"]):
    indistinguishable = [sorted(v) for v in group_by(tools).values() if len(v) > 1]
    print(f"  {len(tools)} test {'+'.join(tools):38s} group {len(indistinguishable)}")
    for o in indistinguishable:
        print(f"      {', '.join(o)}")
print()
print("candidate set sweep (route test)")
WITH_EXTRA = dict(FAULT, **{"address-wrong-subnet": {"gateway_correct": False,
                                            "route_present": False}})
WITHOUT = {k: v for k, v in FAULT.items() if k != "wrong-gateway"}
for ad, table in (("base             8", FAULT), ("+wrong-subnet    9", WITH_EXTRA),
                  ("-wrong-gateway 7", WITHOUT)):
    o = group_by(["route"], table)
    missing = len(o.get(("default-missing",), []))
    print(f"  {ad:18s} distinct answers {len(o)}  worst-case remaining "
          f"{max(len(v) for v in o.values())}"
          f"  remaining in 'default-missing' answer {missing}  eliminated {len(table) - missing}")

route test: candidates per answer
  table-empty    remaining  2  eliminated  6  interface-down, no-address
  complete       remaining  5  eliminated  3  wrong-gateway, resolver-silent, service-not-listening, firewall-dropping, large-packet-lost
  default-missing remaining  1  eliminated  7  missing-route

indistinguishable groups as the tool set grows
  1 test gateway-echo                           group 2
      interface-down, no-address
      firewall-dropping, large-packet-lost, missing-route, resolver-silent, service-not-listening, wrong-gateway
  2 test gateway-echo+remote-echo               group 3
      interface-down, no-address
      missing-route, wrong-gateway
      firewall-dropping, large-packet-lost, resolver-silent, service-not-listening
  3 test gateway-echo+remote-echo+route         group 2
      interface-down, no-address
      firewall-dropping, large-packet-lost, resolver-silent, service-not-listening

candidate set sweep (route test)
  base             8 distinct answers 3  worst-case remaining 5  remaining in 'default-missing' answer 1  eliminated 7
  +wrong-subnet    9 distinct answers 3  worst-case remaining 5  remaining in 'default-missing' answer 2  eliminated 7
  -wrong-gateway 7   distinct answers 3  worst-case remaining 4  remaining in 'default-missing' answer 1  eliminated 6

Three numbers side by side. Oracle: in this round the real fault is wrong-gateway; the default row sits in place and the node it points to does not forward the packet. Test: the routing table is read, and the answer is complete — one of three distinct answers. Candidates eliminated: only three; five candidates remain.

The same test gives a completely different result with a different oracle. Had the real fault been missing-route, the answer would have been default-missing, seven candidates would be eliminated, and one candidate would remain. Had the answer been table-empty, six candidates would be eliminated. The asymmetry from the interface list exists here too, and in the same direction: the test is decisive at seeing an absence in the table, and almost blind at seeing a wrongness in the table. The reason is that the table is not a record of correctness but a record of intent. A row states where the packet is sent; it does not state whether it arrives.

This distinction is directly useful when reading the table. Whether the default row exists is a verifiable fact and is answered definitively just by looking at the table. Whether the address in the row is correct is not a verifiable fact: whether that address is actually a router, whether it forwards the packet, and whether it has a path to the destination while forwarding, is outside the table’s knowledge. In this respect the table resembles a file system entry: an entry existing does not show that where it points is correct. In the storage course, the persistent mount entry was correct on paper while mounting the wrong device; here too, the default row can be correct on paper while pointing to an address that forwards the packet nowhere.

Why the Two Candidates Cannot Be Separated

The second table shows why the wrong-gateway and missing-route pair is a hard pair. Sending an echo looks like the most natural way to separate these two candidates, and two separate echo tests can be done: one to the local gateway, the other to a remote address.

• AY13 — The gateway echo depends only on the interface being up and addressed; the local network is reached through the connected route, and the default row plays no part in this test.
• AY14 — The remote echo requires all four of the interface, the address, the gateway, and the route to be correct at once. This test is done directly by address, with name resolution not involved.
• AY15 — The echo itself is assumed not to be blocked; the case where the echo reply is filtered is not this topic’s problem but the firewall lessons’.

Looked at with a single echo test, eight candidates drop into two groups, and the large group carries six candidates. When the second echo is added, the group count rises to three: the test genuinely does work, splitting the six-candidate group in two. But it does not split where expected. wrong-gateway and missing-route stay together in the new group, because both produce the same pattern: the gateway echo replies, the remote echo does not. Two tests, two waiting periods, and two separate outputs were spent; this pair is still not separated.

The reason for this result is not the tests’ weakness but the shape of the dependency chain. For the remote echo to succeed, all four fields must be correct at once; when any one of the fields breaks, the answer is the same answer. The more fields a test depends on, the more candidates its negative answer fits at once. Tests do not get stronger as they approach the far end of the connection chain — quite the opposite, their power to discriminate drops; what they gain is coverage. The remote echo tests four fields at once in a single step, and for that reason it is a good tool for saying “everything is fine” and a poor one for saying “the problem is here.”

What does the separating is the third test, and that test sends no packet at all. When the table is read, the missing-route candidate gives the default-missing answer, the wrong-gateway candidate gives the complete answer; the group splits apart and the indistinguishable-group count drops from three to two. The course’s second claim is paid off here for the first time: more output does not mean better diagnosis. Reading a local file does in a single step what two remote tests could not do.

Source Address Selection and the One-Directional Fault

The table has one more output that does not show. When a packet is produced, its source address is also chosen, and the choice is made not by the application but by routing: once the row to use is determined, one of the addresses bound to that row’s outgoing interface is written as the source. The decision query’s last field shows this choice; in the table dump it can only be read indirectly, from the connected route’s source field.

When this choice is wrong, the resulting fault pattern is distinctive. The packet reaches the destination, the destination produces a reply, and sends the reply to the address written as the source; if that address is not known on the return path, the reply is lost. What is seen on the local machine is “no reply,” and this is the same picture produced by a wrong gateway. This pattern commonly arises on a server with two interfaces: outbound traffic leaves through one interface, return traffic is routed to the other, and no table row looks wrong. The fault’s distinguishing signature is that two attempts to the same destination with two different source addresses give different results; a single attempt does not show this signature, because the absence of a reply is the same absence of a reply in both cases.

• AY16 — Source address selection is not kept as a separate field in the shared reference; on the single-interface mock server this choice carries no ambiguity. The multi-interface case is outside the model, and the pattern above is a caution about the reading, not about the model.

Adding and Removing a Route

Adding a route also has two forms, and the difference is the same as with address assignment: a route written to runtime is lost on reboot, a route written to configuration comes back at boot. When a route added by hand during diagnosis fixes the problem, the job is not considered done; the same row must also be carried into the persistent configuration.

When two rows are written to the same destination, prefix length decides which one is used; if the prefixes are also equal, the metric given to the rows decides, and the smaller one is chosen. This is a frequent source of ambiguity in diagnosis: two default rows to the same destination can enter from two separate sources — a row given by hand and a row the address configuration added on its own — and the table dump shows both. Whoever looks at the dump counts both rows as correct; only one is actually used. The decision query removes this ambiguity, because it prints not the whole table but the decision actually made.

Operations that change or delete the default row are a separate class. Changing the default gateway on a remotely managed machine can cut off the very session that wrote the command, and because the command does not return, no correction can be made either. This lesson does not give that command sequence in a fully runnable form. The preventive path has three steps, and all three will recur through this course: writing the change out first with a dry run, placing the change behind a timer-based rollback task, and providing management access through a separate channel not affected by the change.

Candidate Set Sweep

The third table shakes the candidate list in two directions.

• AY17 — The added candidate, address-wrong-subnet, is the case where the address is given from the wrong subnet; in that case the gateway becomes unreachable and the default row cannot be established, meaning two fields break at once.
• AY18 — The removed candidate, wrong-gateway, is the candidate the table reading stays weakest on.

When the candidate is added, the distinct-answer count stays fixed at three, the worst-case remaining at five. What changes, again, is the answer thought to be decisive: the default-missing answer eliminates seven of nine candidates but leaves not one but two behind. In other words, the reading “there’s no default row” is no longer a diagnosis but a narrowing down to two candidates. When the candidate is removed, the worst-case remaining drops from five to four, and the candidates the default-missing answer eliminates go from seven to six. The test is the same test; only the list changed.

These two rows show why treating elimination counts as an absolute measure is wrong. The route reading being a test that “leaves five candidates in the worst case” is a number tied to the eight-candidate list. As the list grows, the test’s ceiling rises; as it narrows, the ceiling falls; the one thing that stays fixed is which field the test sees. Ranking tests by this invariant property is a more durable habit than ranking them by elimination counts: the route reading always sees the route field, no matter what the candidate list is.

Summary

• The routing table is a record of intent: it states where the packet is sent, not whether it arrives there. Row selection follows the longest matching prefix rule, and the default gateway is the row with the shortest prefix.
• The route test gives three distinct answers and leaves five candidates in the worst case: the complete answer eliminates only three candidates, the default-missing answer seven, the table-empty answer six. The test sees an absence in the table; it does not see a wrongness in the table.
• The wrong-gateway and missing-route pair cannot be separated by two echo tests: both reply to the local echo and do not reply to the remote echo. The indistinguishable-group count is three after two tests, and two once the table reading is added.
• The separation is done by a local reading that sends no packet at all; the two remote tests, which produce more output, cannot do it.
• The sweep shows the elimination count depends on the candidate list: when a ninth candidate is added, the default-missing answer leaves two candidates; when wrong-gateway is removed, the worst-case remaining drops from five to four.

Next Step

Once the table reading gives the complete answer, four of the remaining five candidates concern not the packet’s path anymore but what is being asked for: the destination’s name may not resolve, the service may not be listening, the firewall may be dropping, or a large packet may be getting lost on the way. The next lesson looks at the first of these four and reads the resolver configuration. The number measured there is one of the course’s smallest: the name resolution test, when it gives the expected answer, eliminates only one of eight candidates. Why such a weak test is still done is that lesson’s real question.