Ports and Listening Services

Once name resolution was also confirmed, three candidates remained: the service may not be listening, the firewall may be dropping the packet, or a large packet may be getting lost on the way. All three look as if they leave a place to look on the server itself — the list of listening sockets. The list is local, fast, detailed, and looks directly at where the problem passes through.

This lesson pulls up that list and places a second test next to it: a single connection attempt made from outside. The two tests are measured at the same time, against the same fault, and the gap between them is one of the largest in the course.

What a Listening Port Is

When a service starts listening on a port, a record is created in the kernel: which address-and-port pair will accept incoming requests. The local inventory lists these records. The list’s most important column is not the port number, it is the bind address.

# example dump — not executed
$ ss -ltn
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port
LISTEN  0       128     0.0.0.0:9443         0.0.0.0:*
LISTEN  0       128     127.0.0.1:7070       0.0.0.0:*

The two rows have the same shape and tell two different facts. The first accepts requests coming from every interface; the second accepts requests only from the loopback interface — that is, from the machine itself. Seen from outside, the second row’s counterpart is a port that is not listening at all. The inventory writes “LISTEN” for both rows.

The process column is a separate reading and requires privilege: a listing tool run without privilege shows which ports are open, not which process opened them. This costs a step in diagnosis — which port is open is known, which unit opened it is not.

How port numbers are assigned, which ranges are reserved for ephemeral ports, and how the transport layer’s connection-establishment behavior works is the subject of the Computer Networks curriculum; no protocol is built here, only this machine’s inventory is read.

• AY25 — On the mock server, the data-ingest unit listens on every interface; a second, local management endpoint listens only on loopback and is never exposed outward.
• AY26 — In the shared reference, the listening list and the connection attempt depend on the same field: if the service is listening, the list shows it and the connection is established. A service bound only to loopback — appearing to listen in the list while being refused from outside — is outside the model, and the caution above is part of the reading, not of the measurement.

Three Things That Get Confused When Reading the List

The inventory gathers three separate concepts into a single table, and because all three are described with the word “listening,” they get confused with one another.

The first is ports that are not connection-oriented. In the connection-establishing transport form, a port explicitly enters a listening state and appears as such in the list; in the connectionless form there is no such state, only a port being bound is seen. The two kinds sit side by side in the same list and are read with the same word, yet for the second kind, “listening” means only “packets arriving at this port are delivered to a process.”

The second is the separation of socket state from service health. The existence of a listening record does not show that the process behind it is actually processing requests. A process that does not accept the request but leaves the socket open looks healthy in the list; the listen backlog fills up, and new connections start being dropped past a certain point. The measurement from the System Administration course is not repeated here; its result there was this: a unit looking “running” and a unit doing its job are two separate facts. Its network-side counterpart is the difference between a listening record existing and a request being answered.

The third is the difference between persistent configuration and running state, and this is the pattern repeated since this topic’s first lesson. A service started by hand shows up in the list; for it to start at boot, it must also be enabled separately. The inventory shows only the present moment.

The Two Tests’ Measure

"""Elimination power of the local listening list versus an external connection attempt.
Same eight candidate causes, same moment, two separate tests."""
HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True,
           "route_present": True, "resolver_responds": True, "service_listening": True,
           "firewall_allows": True, "path_intact": True}
FAULT = {
    "interface-down":       {"interface_up": False, "address_present": False},
    "no-address":      {"address_present": False},
    "wrong-gateway":        {"gateway_correct": False},
    "missing-route":        {"route_present": False},
    "resolver-silent":      {"resolver_responds": False},
    "service-not-listening": {"service_listening": False},
    "firewall-dropping":    {"firewall_allows": False},
    "large-packet-lost": {"path_intact": False},
}


def state(fault, table):
    d = dict(HEALTHY)
    d.update(table[fault])
    return d


def test_listening(d):
    return "listening" if d["service_listening"] else "not-listening"


def test_connection(d):
    if not (d["interface_up"] and d["address_present"] and d["gateway_correct"]
            and d["route_present"]):
        return "unreachable"
    if not d["firewall_allows"]:
        return "timeout"
    return "refused" if not d["service_listening"] else "established"


TEST = {"listening": test_listening, "connection": test_connection}


def groups(test, table=FAULT):
    o = {}
    for a in table:
        o.setdefault(TEST[test](state(a, table)), []).append(a)
    return o


for s in TEST:
    print(f"{s} test: candidates per answer")
    for answer, group in groups(s).items():
        print(f"  {answer:12s} remaining {len(group):2d}  eliminated {len(FAULT) - len(group):2d}"
              f"  {', '.join(group)}")
    print()
print("eliminated candidates per oracle")
print(f"  {'real fault':27s} {'listening':>20s} {'connection':>20s}")
total = {"listening": 0, "connection": 0}
for real_fault in FAULT:
    row = ""
    for s in TEST:
        answer = TEST[s](state(real_fault, FAULT))
        eliminated = len(FAULT) - len(groups(s)[answer])
        total[s] += eliminated
        row += f"  {answer:12s} {eliminated:2d} "
    print(f"  {real_fault:27s} {row}")
print(f"  {'TOTAL eliminated':27s} {total['listening']:18d} {total['connection']:19d}")
print()
print("candidate set sweep")
WITH_EXTRA = dict(FAULT, **{"service-crashed": {"service_listening": False}})
WITHOUT = {k: v for k, v in FAULT.items() if k != "firewall-dropping"}
for ad, table in (("base             8", FAULT), ("+service-crashed 9", WITH_EXTRA),
                  ("-firewall        7", WITHOUT)):
    d = groups("listening", table)
    b = groups("connection", table)
    dk = len(d.get("not-listening", []))
    bk = len(b.get("refused", []))
    print(f"  {ad:18s} listening: answers {len(d)}"
          f"  'not-listening' eliminated {len(table) - dk} remaining {dk}"
          f"  | connection: answers {len(b)} worst-case remaining "
          f"{max(len(v) for v in b.values())}"
          f"  'refused' remaining {bk}")

listening test: candidates per answer
  listening    remaining  7  eliminated  1  interface-down, no-address, wrong-gateway, missing-route, resolver-silent, firewall-dropping, large-packet-lost
  not-listening remaining  1  eliminated  7  service-not-listening

connection test: candidates per answer
  unreachable  remaining  4  eliminated  4  interface-down, no-address, wrong-gateway, missing-route
  established  remaining  2  eliminated  6  resolver-silent, large-packet-lost
  refused      remaining  1  eliminated  7  service-not-listening
  timeout      remaining  1  eliminated  7  firewall-dropping

eliminated candidates per oracle
  real fault                             listening           connection
  interface-down                listening     1   unreachable   4 
  no-address                    listening     1   unreachable   4 
  wrong-gateway                 listening     1   unreachable   4 
  missing-route                 listening     1   unreachable   4 
  resolver-silent               listening     1   established   6 
  service-not-listening         not-listening  7   refused       7 
  firewall-dropping             listening     1   timeout       7 
  large-packet-lost             listening     1   established   6 
  TOTAL eliminated                            14                  42

candidate set sweep
  base             8 listening: answers 2  'not-listening' eliminated 7 remaining 1  | connection: answers 4 worst-case remaining 4  'refused' remaining 1
  +service-crashed 9 listening: answers 2  'not-listening' eliminated 7 remaining 2  | connection: answers 4 worst-case remaining 4  'refused' remaining 2
  -firewall        7 listening: answers 2  'not-listening' eliminated 6 remaining 1  | connection: answers 3 worst-case remaining 4  'refused' remaining 1

Three numbers side by side. Oracle: in this round the real fault is firewall-dropping; the service is listening, packets are arriving, and the filter silently drops them. Test: the local listening list is pulled up and its answer is listening; at the same time a connection is attempted from outside and its answer is timeout. Candidates eliminated: the listening list, one; the connection attempt, seven. The list leaves seven candidates, the connection attempt leaves one and finishes the diagnosis.

The Power of Four Answers

The third table shows this gap is not specific to a single fault. Summed across all eight faults, the listening list eliminates 14 candidates, the connection attempt 42. The ratio is three, and looked at one by one the distribution is even sharper: the listening list eliminates only one candidate in seven of the eight cases and seven in a single case. The connection attempt never drops below four in any case.

The source of the difference is not the amount of output but the number of answers. The listening list can produce dozens of lines; the distinction it carries is a single binary question — is the service listening or not. The connection attempt produces a single line and carries four distinct answers: unreachable, timeout, refused, established. Each answer corresponds to a different candidate set. A test’s elimination-power ceiling is set not by the length of its output but by the number of states it distinguishes.

Which candidates the answers correspond to is used directly in diagnosis. The unreachable answer points to path problems and leaves four candidates together; this is the test’s weakest answer, and the previous three lessons’ tests do the separating. The refused answer says the packet reached the target and there is no listener at the target; this means everything along the path is working, and it leaves a single candidate. The timeout answer says the packet went out and no reply came back at all; this is the signature of a filter silently dropping it. The established answer says the connection opened, and surprisingly leaves two candidates standing: a silent resolver and a large packet dropped along the way. A connection being established does not mean the connection is usable.

This last line is this lesson’s counterpart of the course’s second claim. The distinction between refused and timeout is a single word and eliminates seven candidates each; yet the most detailed operation that could be done to obtain the same information — recording all the traffic — produces thousands of lines. More output does not mean better diagnosis; what matters is how many distinct states the output corresponds to.

This measure also has a reverse reading. The connection attempt’s four answers are valuable because they correspond to four distinct fault classes; if the sheer number of answers were enough on its own, a tool that split every answer into its own error code would be unboundedly powerful. What is valuable is that the answers split the candidate set evenly: the four answers split eight candidates 4-1-1-2. Three of the answers come down to one or two candidates, one gets stuck at four. The answer that weakens the test is the most frequently encountered answer; unreachable here is exactly that, and the previous three lessons’ tests exist precisely to separate the four candidates that answer leaves.

The Inventory’s Place

The listening list being weak does not make it unnecessary; it changes its place. The list is a confirmation tool when the connection attempt says refused: the refusal says there is no listener at the target, the inventory confirms this locally and shows which bind address it is listening on. The list is also the only option in cases where the connection attempt cannot be made — if there is no outside access, if there is no second endpoint to run the test from, the local inventory is the only reading left.

• AY27 — The connection attempt is made from a separate endpoint that can reach the target; that endpoint’s own configuration is assumed sound. The attempt having two endpoints is what raises this test’s cost.
• AY28 — The distinction between timeout and refusal is assumed visible on the client side; cases where an intermediate layer changes the answer are outside the model.

The inventory’s second function is not diagnosis but surface counting. Every port listening from outside is a door opened outward; a port listening only from loopback is not. This distinction is the starting point of the firewall lessons, and it is handled there by counting exposed surface. This lesson says only this: the inventory’s line count is not a health measure, and counting lines without reading the bind-address column is misleading.

Its third function is comparison. Taking the inventory before and after a configuration change is the cheapest record showing what the change actually did. When a unit is restarted, the port it listens on may have changed, the bind address may have narrowed or widened; the difference between two lists gives this in a single glance. This is a measurement taken not during diagnosis but during the change, and having it in hand after a fault is born is far more valuable than searching for it after the fact.

Candidate Set Sweep

• AY29 — The added candidate, service-crashed, is the case of the unit crashing, and in the model it breaks the same field as service-not-listening.
• AY30 — The removed candidate, firewall-dropping, is the candidate that alone fills the connection attempt’s fourth answer.

The fourth table compares the two tests under the same shake. When the candidate is added, the listening list’s not-listening answer still eliminates seven candidates but leaves two behind; the same is true of the connection attempt’s refused answer. Both tests weaken by the same amount, because both see the same field. This is an expected result and confirms the model’s internal consistency.

When the candidate is removed, the tests behave differently. The listening list loses only one candidate: the candidates the not-listening answer eliminates drop from seven to six, and its structure does not change. The connection attempt’s answer count, on the other hand, drops from four to three; because the timeout answer now has no counterpart in the candidate list, the number of states the test distinguishes falls. Even though the worst-case remaining stays fixed at four, the distinction the test carries has shrunk. A test’s power is the number of its answers that have a counterpart in the candidate list, and this number changes as the list changes.

This is the common result of the sweep done four times across the topic. In the interface test, the answer thought to be decisive lost its decisiveness; the same happened in the route reading; in name resolution, the test became entirely worthless; here, the connection attempt lost one of its answers. In all four cases, the test’s definition was never touched. The number measured is not a property of the test, it is a joint product of the test and the candidate list, and this is why a diagnostic method’s portability depends on the portability of the candidate list.

Summary

• The local inventory’s most important column is not the port number but the bind address; a port listening only from loopback is not listening at all from outside, and it appears the same way in the list.
• The listening list gives two answers and eliminates only one candidate on the expected answer; a connection attempt made from outside gives four answers and eliminates seven on the same fault.
• Summed across all eight faults, the listening list eliminates 14 candidates, the connection attempt 42. The source of the difference is not output length but the number of states the test distinguishes.
• The connection attempt’s four answers correspond to four distinct candidate sets: unreachable leaves four candidates together, refused and timeout each leave one, and established leaves two — a connection being established does not mean it is usable.
• The sweep separates the two tests: when a candidate is added, both weaken by the same amount; when a candidate is removed, the listening list’s structure is preserved while the connection attempt’s answer count drops from four to three.

Next Step

This topic measured four tests one by one and counted how many candidates each eliminates. The resulting table shows a single test is not enough: even the strongest leaves four candidates together in some cases, and the weakest separates a candidate no other test can. So the real question is not the value of a single test, but the order tests should be done in. The next topic asks this question: how many steps does the layered diagnostic method spend to resolve all eight faults, how many steps does the reverse order spend, and how much does an order that picks the most eliminating test at every step shorten this. What will be seen there is that the shortest order is not an applicable procedure: picking the most eliminating test requires knowing in advance which fault is real, and that is exactly the one thing the person doing the diagnosis does not know.