Name Resolution

When the routing table gave the expected answer — complete — five candidates remained. Four of these no longer concern the packet’s path but what is being asked for: the destination’s name may not resolve, the service may not be listening, the firewall may be dropping, or a large packet may be getting lost on the way. This lesson looks at the first of the four.

The number to be measured is one of the course’s smallest, and the lesson’s real question is what that smallness means: if a test, when it gives the expected answer, eliminates only one of eight candidates, why is that test still done at all.

What the Resolver Configures

Name resolution is the conversion of a name into an address, and this job is not done in a single place. The query first hits the name service order: the system reads from a configuration file which source is tried in which order. The first source in the order is usually a local hosts file; after it comes a local cache or a stub resolver; last is the query sent to the resolver on the network. When a name is found in the local file, the network is never reached, and the answer the network resolver would have given is never seen.

# example dump — not executed
$ cat /etc/resolv.conf
nameserver 192.0.2.53
search example
options timeout:2 attempts:2

Three fields of the configuration determine three separate behaviors. The resolver address states where the query goes; when more than one is written, the first is tried, and the next is tried only when no reply comes. The search domain gives the suffix added to a name with no dot in it; a query made with a short name and the same query made with a fully qualified name can therefore produce different results. The timeout and retry count determine how long a fault takes to show: a two-second timeout with two attempts makes every query on a silent resolver wait four seconds, and at the application layer this shows up not as a name resolution fault but as slowness.

# example dump — not executed
$ getent ahosts record-store.example
198.51.100.42  STREAM record-store.example
198.51.100.42  DGRAM

The choice of query tool also changes the result. Tools that ask the resolver directly skip the name service order and test only the network resolver; testing the path the system actually uses requires a tool that goes through the whole order. This distinction produces a concrete result in diagnosis: one query tool can resolve a name that the application cannot, or the other way around.

How names are resolved over the network, which record types the query and the answer carry, and how the authoritative server hierarchy works is the subject of the Computer Networks curriculum; no protocol is built here, only this machine’s resolver configuration is read.

• AY19 — On the mock server, the name service order consists of two sources: the local hosts file and a single network resolver. The name being looked up is assumed not to be present in the hosts file.
• AY20 — The resolver-silent candidate is the resolver never replying to the query at all. A resolver giving a wrong answer is a separate candidate and is not in the base list; it is added in the sweep. The path to the resolver itself being sound is accepted as a requirement of the single-fault assumption.

The Order Itself Is a Source of Faults

The name service order produces two separate traps in diagnosis, and both are common.

The first is the precedence of the local entry. A line written into the hosts file wins regardless of what the network resolver says. When a line added temporarily during a migration is forgotten in place, the system keeps connecting to the old address for months, and because the record on the network has been corrected, no query tool shows the fault — a query tool that skips the order returns the correct address, while the application goes to the wrong one. The only reading that makes the distinction is a query that goes through the order.

The second is the silent expansion of the search domain. When a name with no dot in it is queried, the suffix defined for the system is appended, and the query turns into a different name. The same short name can expand to two different fully qualified names on two machines, because their search domains differ. For this reason, carrying a test result from one machine to another is not reliable: the name tested is not the name written. Writing the fully qualified name in configuration files closes this ambiguity at the source.

A third behavior is not a fault but hides one: when a source in the order does not reply, the query falls through to the next. The fall-through is silent and shows up only as duration. In a configuration with a two-second timeout and two attempts, a silent resolver adds four seconds to every query; in the application log, the counterpart of this is not an error but a delay. The pattern from the System Administration course applies here too: the visible symptom and the real fault sit at different layers.

The Test’s Measure: One Candidate

"""Name resolution test: how many of the same eight candidate causes are eliminated,
and how many candidates it SEPARATES when added to the previous lessons' test set."""
HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True,
           "route_present": True, "resolver_responds": True, "service_listening": True,
           "firewall_allows": True, "path_intact": True}
FAULT = {
    "interface-down":       {"interface_up": False, "address_present": False},
    "no-address":      {"address_present": False},
    "wrong-gateway":        {"gateway_correct": False},
    "missing-route":        {"route_present": False},
    "resolver-silent":      {"resolver_responds": False},
    "service-not-listening": {"service_listening": False},
    "firewall-dropping":    {"firewall_allows": False},
    "large-packet-lost": {"path_intact": False},
}


def state(fault, table):
    d = dict(HEALTHY)
    d.update(table[fault])
    return d


def test_route(d):
    if not d["address_present"]:
        return "table-empty"
    return "complete" if d["route_present"] else "default-missing"


def test_gateway_echo(d):
    return "reply" if d["interface_up"] and d["address_present"] else "no-reply"


def test_remote_echo(d):
    return "reply" if (d["interface_up"] and d["address_present"]
                       and d["gateway_correct"] and d["route_present"]) else "no-reply"


def test_name_resolution(d):
    return "resolved" if d["resolver_responds"] else "unresolved"


TEST = {"route": test_route, "gateway-echo": test_gateway_echo,
        "remote-echo": test_remote_echo, "name-resolution": test_name_resolution}


def group_by(tests, table=FAULT):
    o = {}
    for a in table:
        key = tuple(TEST[s](state(a, table)) for s in tests)
        o.setdefault(key, []).append(a)
    return o


print("name resolution test: candidates per answer")
for answer, group in group_by(["name-resolution"]).items():
    print(f"  {answer[0]:11s} remaining {len(group):2d}  eliminated {len(FAULT) - len(group):2d}"
          f"  {', '.join(group)}")
print()
PREVIOUS = ["gateway-echo", "remote-echo", "route"]
print("when added to the previous lessons' test set")
for tests in (PREVIOUS, PREVIOUS + ["name-resolution"]):
    indistinguishable = [sorted(v) for v in group_by(tests).values() if len(v) > 1]
    covered = sum(len(o) for o in indistinguishable)
    print(f"  {len(tests)} tests  group {len(indistinguishable)}  indistinguishable candidates {covered}")
    for o in indistinguishable:
        print(f"      {', '.join(o)}")
print()
print("candidate set sweep (name resolution test)")
WITH_EXTRA = dict(FAULT, **{"resolver-wrong-record": {"resolver_responds": False}})
WITHOUT = {k: v for k, v in FAULT.items() if k != "resolver-silent"}
for ad, table in (("base                8", FAULT),
                  ("+wrong-record       9", WITH_EXTRA),
                  ("-resolver-silent    7", WITHOUT)):
    o = group_by(["name-resolution"], table)
    row = "  ".join(f"{y[0]}: eliminated {len(table) - len(k):2d} remaining {len(k):2d}"
                    for y, k in sorted(o.items()))
    print(f"  {ad:21s} distinct answers {len(o)}   {row}")

name resolution test: candidates per answer
  resolved    remaining  7  eliminated  1  interface-down, no-address, wrong-gateway, missing-route, service-not-listening, firewall-dropping, large-packet-lost
  unresolved  remaining  1  eliminated  7  resolver-silent

when added to the previous lessons' test set
  3 tests  group 2  indistinguishable candidates 6
      interface-down, no-address
      firewall-dropping, large-packet-lost, resolver-silent, service-not-listening
  4 tests  group 2  indistinguishable candidates 5
      interface-down, no-address
      firewall-dropping, large-packet-lost, service-not-listening

candidate set sweep (name resolution test)
  base                8 distinct answers 2   resolved: eliminated  1 remaining  7  unresolved: eliminated  7 remaining  1
  +wrong-record       9 distinct answers 2   resolved: eliminated  2 remaining  7  unresolved: eliminated  7 remaining  2
  -resolver-silent    7 distinct answers 1   resolved: eliminated  0 remaining  7

Three numbers side by side. Oracle: in this round the real fault is not resolver-silent; some other field is broken on the mock server, and the resolver is working. Test: the name query is made, and its answer is resolved. Candidates eliminated: one; seven candidates remain. This is one of the lowest values among the eight tests the course measures; only the listening-socket list gives the same number.

The Value of a Weak Test

The name resolution test produces two answers, and the gap between the two answers is the largest in the course: resolved eliminates one candidate, unresolved eliminates seven. Same command, same duration, same effort; the information it carries differs by a factor of seven. The test’s value is not in the test, it is in which answer it gives.

The practical reading of this asymmetry has two parts. First: when name resolution fails, the diagnosis is over. Seven candidates are eliminated, one remains, and what needs fixing is clear. Second: when name resolution succeeds, it looks as if nothing was learned, yet something was — a confounder has been removed. Every test done by name asks two questions at once: does the name resolve, and is the target reachable. A connection attempt’s failure, made without resolution being verified separately, does not show which of these two questions it belongs to. The name resolution test fixes the meaning of every test that follows it.

The second table turns this contribution into a number. The previous two lessons’ test set — the local echo, the remote echo, and the table reading — left eight candidates unseparated across two groups, and those groups held six candidates in total. When name resolution is added, the group count stays at two, and the indistinguishable candidate count drops to five. So the test’s contribution is exactly one candidate, and that one candidate is the one no other test can separate on its own. None of the four tests can separate the resolver-silent candidate without touching it, because when the resolver is silent, packets keep flowing along the path without any problem.

This shows why ranking tests by elimination power alone is not enough. How many candidates a test eliminates on its own and how many candidates it separates when added to a set are separate measures, and the two do not move in the same direction. A strong test produces no contribution when it eliminates candidates the tests before it have already separated; a weak test becomes irreplaceable when it looks at the single field no other tool can see. Name resolution is of the second kind. When building a diagnostic order, the question to ask is not “how many candidates does this test eliminate” but “how many candidates invisible to anything else does this test separate.”

A Name Resolving Is Not Reachability

When resolution succeeds, what comes back is an address, and not the information that the address is correct. The record may be stale, the target may have moved to another address, the answer may be coming from a cache. The cache plays a double role here. A positive answer is kept for the lifetime written into the record, and no change is seen until that time elapses; a negative answer can also be kept, and a corrected record appears later than expected. In both cases, what the test sees is not the network’s current state but the state of a moment in the past.

A name can also resolve to more than one address, and the returned list can carry more than one address family. In this case, which address to connect to is chosen not by the resolver but by whoever establishes the connection, and the selection order depends on the system’s configuration. The resulting fault pattern is misleading in diagnosis: the name resolves correctly, the path to the first address in the list is broken, the path to the second works, and the result varies from attempt to attempt. If the same command run twice in a row gives two different results, the first place to look is the list resolution returns; a tool that shows a single-line answer hides this list.

• AY21 — There is no cache in the shared reference; the resolver either replies or does not. Lifetime and negative caching are outside the model, and the caution here is part of the reading, not of the measurement.
• AY22 — The name resolution test is assumed independent of the result of tests done by address; the combined case where the path to the resolver itself is broken is outside the single-fault assumption.

The working habit that follows from this is one sentence: test by address first, then by name. If an attempt by address succeeds and the same attempt by name fails, the fault is in resolution and no other test is needed. If both fail, resolution stops being a candidate and the problem is in the layers below. When the order is reversed — trying by name first and seeing failure — the answer obtained covers two candidates at once and a second test is needed to separate them. The same two commands, in a different order, eliminate a different number of candidates.

Candidate Set Sweep

The third table shakes the candidate list in two directions again, and in this lesson one of the results is the sharpest in the course.

• AY23 — The added candidate, resolver-wrong-record, is the case where the resolver replies but returns the wrong address. In the model this is represented as resolution not giving the expected result, and it produces the same answer as resolver-silent.
• AY24 — The removed candidate, resolver-silent, is the candidate that alone fills the test’s negative answer.

When the candidate is added, the number of candidates the resolved answer eliminates rises from one to two; the test does twice the work, and it does so with nothing changed in its own definition. The unresolved answer still eliminates seven candidates, but it now leaves not one but two behind: a silent resolver and a resolver giving a wrong answer cannot be separated by this test. What does the separating is looking at the answer’s content, not its presence.

When the candidate is removed, the result is harsher. In the seven-candidate list, the number of answers the test can give drops to one, and the candidates it eliminates become zero. The test still runs, still produces an answer, still spends time — and eliminates no candidate. This is the sharpest demonstration of the course’s rule: the output a test produces and the information it carries are separate things. A test with no counterpart in the candidate list, however rich its output, is not unmeasured — it is worthless. The diagnostic counterpart of this is a regular review of tests done out of habit: whichever candidate a test entered the list to separate, once that candidate drops from the list, the test should drop too.

Summary

• Name resolution is not done in a single place: the query goes through the name service order, and the local hosts file and cache can produce an answer without reaching the network; tools that ask the resolver directly skip this order and do not test the path the system actually uses.
• The name resolution test gives two answers, and the gap between them is the largest in the course: resolved eliminates only one candidate and leaves seven remaining, unresolved eliminates seven and finishes the diagnosis.
• The test’s contribution when it gives the expected answer is removing a confounding variable: it separates exactly one of the six candidates the previous three tests could not separate, bringing the indistinguishable count from six to five. No other test can separate that candidate.
• A name resolving does not mean the target is reachable; because of caching and lifetime, the returned answer can be the state of a moment in the past. The working rule is to test by address first, then by name.
• The sweep gives two results: when a resolver giving a wrong answer is added, the resolved answer starts eliminating two candidates; when resolver-silent is removed, the test’s answer count drops to one and the candidates it eliminates drop to zero. A test with no counterpart in the candidate list is worthless.

Next Step

Once name resolution is also confirmed, three candidates remain: the service may not be listening, the firewall may be dropping the packet, or a large packet may be getting lost on the way. All three look as if they leave a place to look on the server itself — the list of listening sockets. The next lesson pulls up that list and puts two numbers side by side: the local inventory eliminates one of eight candidates, a single connection attempt made from outside eliminates seven. Same fault, same moment, a sevenfold difference between two tests. That lesson’s question is where this difference comes from: the listening list sees a single field, while the connection attempt separates four distinct outcomes from one another.